TransformerLens 4.4.0: Whisper bridge, SSM and hybrid interpretation

1. Analyzing Whisper through a bridge

Released on November 11, 2026, this version includes a Whisper adapter, generation, weight processing, and model validation. A bridge connects the original model to analysis tools. Whisper now also supports a compatibility mode that transforms weights before analysis.

Compatibility mode supports folding normalization, which standardizes value scales, into weights and centering weight values. The Whisper adapter specifies connections between the encoder, which converts audio into internal representations, and the decoder, which turns those representations into sentences. This lets the weight processor locate transformation targets.

2. Weight processing that preserves transcriptions

According to the contributor's change description, transcriptions for whisper-tiny and whisper-large-v3-turbo were unchanged when choosing the highest-scoring output at each step. Logits, the scores for candidate next outputs, also matched the Hugging Face implementation except for position-specific constant offsets introduced by weight centering.

Developers can apply this processing by connecting Whisper through a bridge and calling enable_compatibility_mode(). Rather than changing model responses, it reshapes the internal computations producing those responses for easier analysis.

3. Residual decomposition for SSM and hybrid models

Residual decomposition splits accumulated model values into contributions from individual blocks. State space models (SSMs) carry past information forward through state values, while hybrid models combine different computational structures. Some blocks use alternatives to attention, which computes relationships between information, or lack the multilayer neural network called an MLP.

The previous API assumed every block had both attention and an MLP, causing errors when analyzing models such as Qwen3.5 and Mamba-2. The fix follows actual block structures, including outputs from attention alternatives and omitting MLP components where absent.

_, cache = bridge.run_with_cache(tokens)   # e.g. Qwen/Qwen3.5-0.8B
cache.decompose_resid()                    # KeyError: 'blocks.0.hook_attn_out'
cache.stack_head_results()                 # KeyError: 'blocks.0.attn.hook_z'
cache.get_full_resid_decomposition()       # KeyError: 'blocks.0.attn.hook_z'
cache.logit_attrs(cache.decompose_resid(), tokens=...)  # KeyError

The original code example shows where analysis failed before the fix. After the fix, only blocks with attention return per-head results, outputs split by attention computation unit. Outputs from other blocks are grouped as remaining components. The change focuses on ensuring that decomposed values sum back to the original accumulated values across model architectures.

4. Precision and scope

The PR author checked the sums by running the actual Qwen/Qwen3.5-0.8B on CPU in fp32, a single-precision floating-point format. The values below show how closely each decomposition sum matched the original accumulated values in that environment. The author reported errors within these respective bounds.

Table: Absolute error bounds for Qwen3.5-0.8B residual decomposition sums

Conditions: Actual Qwen/Qwen3.5-0.8B, fp32, CPU — measured by the PR author

Unit: Absolute error

These figures show consistency when recombining decomposition results, not generation speed. More models can be analyzed, but component names and categories vary by model. Existing analysis code must therefore be checked for assumptions about specific block configurations.

GraniteMoeHybrid's mixture-of-experts blocks, which select some experts from several, lack an MLP slot for that computational component, so residual decomposition still fails. This structure stores the MLP elsewhere, and its neuron-level expansion, which extends analysis to individual neurons, raises a not-implemented error.

The key to 4.4.0 is combining Whisper weight analysis verified to preserve outputs with model decomposition beyond attention-centric designs in a single release.