CS Paper Radar

Interpreting Language Model Hidden States at Scale

Jordan Pettyjohn, Mansi Sakarvadia, Nathaniel Hudson, Daniel McKenzie, Kyle Chard, Ian Foster

cs.AI ·submitted 2026-08-10 21:49 UTC · v1

Abstract

Lens methods interpret large language models (LLMs) by mapping intermediate activations to the output vocabulary, revealing how next-token predictions develop through the network. Trained lenses remain expensive: affine-translator parameters grow quadratically with model width, while exact, full-vocabulary Kullback--Leibler (KL) training dominates memory. Consequently, prior trained lenses have been applied to models of at most 20B parameters and remain tied to particular component types. We present OmniLens, which applies a single lens family to any model-width activation, whether residual stream, attention, or MLP, and combines two independent scaling techniques. First, low-rank translators make per-lens parameter growth linear in model width and reduce trainable parameters by up to 98.4%. Second, Subset-KL materializes only selected vocabulary logits: its Top-k mode cuts peak training memory by up to 70%, while its importance-sampled variant retains unbiased stochastic gradients for the full KL. These savings enable a dense ensemble of 482 lenses for LLaMA-3.3-70B, providing 6x the coverage of a residual-stream design at the same depth. Model-wide coverage then reveals what single-component lenses cannot: the components where a behavior is most visible need not be those where intervention is most effective, and the most effective interventions lie outside the attention heads examined by prior lens studies. Across three case studies (prompt-injection detection, multi-hop memory injection, and toxicity localization), OmniLens reproduces key published results at substantially lower cost.

Closest papers

Nearest neighbours by meaning across the whole index.

Fitting Vision Adapters at Frontier Scales

cs.CV cs.LG ·2026-10-05 ·Jaehoon Lee, Harry Partridge, Mudith Jayasekara, Charles O'Neill, Max Kirkby +1 more
published similarity 0.67
Abstract · NeurIPS 2026 Workshop: Grounded and Faithful Vision-Language Models for Real-World Deployment

Training a small projector between a frozen vision encoder and language model is an established approach to multimodal learning. As the parameter count of language models scales dramatically, we revisit which vision capabilities this approach can add while keeping their pretrained weights fixed. Here we train a 50M parameter projector from the vision encoder of Kimi K2.6 to GLM 5.2 and 5.3, both models without native vision capabilities, and further present a reproducible recipe for training these adapters at scale. We study the following: (a) how vision capabilities of multimodal models scale as purely the language model side scales, and (b) what specific vision capabilities are able to be imbued into a pure language model at scale, and which ones remain limited. We evaluate on MMMU-Pro and BLINK, examining both overall performance and results on individual visual tasks.

0.19

DV-Lens: Revealing the Functional Organization of Language Model Parameters

cs.CL cs.LG ·2026-10-03 ·Chenhang Cui, Jian Yu, Shuyi Miao, Xiaohao Liu, Rui Huang +3 more
benchmark similarity 0.67
Abstract

Understanding parameter functions helps elucidate the internal mechanisms of large language models (LLMs). However, how to connect parameters from different modules to verifiable output effects and further characterize the relationship between their functional organization and model capability remains to be explored. To this end, we introduce the downstream vocabulary lens (DV-Lens), a parameter-level interpretability framework that links native parameter directions to their downstream vocabulary responses. Specifically, we first estimate module-specific downstream Jacobians over a reference prompt set for attention query, key, value, and output (Q/K/V/O) projections and feed-forward networks (FFNs). Second, we use these mappings to project native parameter columns into the final vocabulary space, obtaining signed readouts that characterize their average local output responses. Third, we group parameter columns by their vocabulary readouts and introduce downstream vocabulary complexity (DV-Complexity), which quantifies within-group structural variation using normalized reconstruction residuals of the original weights. At the parameter level, randomized controls and finite-difference tests show that DV-Lens readouts capture non-random vocabulary structure and predict local logit changes with 98.0% coordinate-orientation agreement across 720 cases from nine models. These readouts further guide parameter ablation, steering, and swapping across 21 models, shifting target-token probabilities in the predicted directions under controlled conditions. At the model level, the joint-parameter score of DV-Complexity achieves a Spearman correlation of 0.904 with benchmark-based capability rankings across 48 language models. Together, these results provide intervention-based evidence for DV-Lens interpretations and reveal an association between DV-Complexity and model capability.

0.27

Verbalizing Multi-Token Concepts in LLMs

cs.CL ·2026-08-31 ·Xijie Gong, Zimeng Huang, Tonghan Wang
code/data similarity 0.66
Abstract

Lens methods inspect model computation by mapping intermediate activations to vocabulary tokens. Yet the concepts humans need to read out often span multiple tokens---entities, phrases, intermediate objects---making token-level readouts incomplete. Reliable multi-token readout with little model-specific preparation remains challenging. We introduce Concept Lens: token-level lens clues guide candidate concept search, then the model derives a representation for each candidate and scores it against the original activation. Across 2,400 multi-hop clozes on five LLMs (8B--70B), Concept Lens instantiated with J-lens and R-lens achieves average Rank@10 scores of 36.6\% and 54.5\%, respectively, compared with 21.7\% for Template Lens. Concept-swap interventions on derived concept representations shift model answers toward those associated with the replacement concepts. Further experiments show that Concept Lens can also reveal what a model recognizes along the way, beyond what appears in its final answer. Our code is available at https://github.com/XijieGo/c-lens

0.44

DimPO: Dimensionality Reduction for Attention using Preference Optimization

cs.LG ·2026-09-26 ·Vojtěch Lanz, Yufei Cui, Prasanna Parthasarathi
similarity 0.64
Abstract

A linear projection can reduce the dimension of query and key vectors without updating the pretrained model, but it remains unclear which training objective best preserves model behavior. We ask whether preferences over keys and attention mass on the highest-weighted keys provide a better signal than matching the full attention distribution with KL divergence, especially in long-context settings. We introduce DimPO, which combines listwise preference optimization with a lightweight top-k cross-entropy term for head-fidelity. DimPO is trained offline from the attention patterns of a frozen language model, with one map per layer, shared by the query and the keys and trained separately from the other layers. Across LLaMA3.2-3B, LLaMA3.1-8B, Qwen2.5-7B, and Qwen3-4B Instruct models, pairwise preference objectives outperform the triplet baseline and retain 98% of the original score on short-context tasks when projecting to half the dimension on the last 40% of the layers. With more projected layers or on long-context RULER, they degrade rapidly. In contrast, KL and DimPO, which use every key during training, retain about 95% of the original RULER 4k score on the 8B model when projecting up to 50% of the layers. KL-based projections remain closer to the original attention distribution and attention output, yet DimPO achieves better downstream performance. Beyond 50% of projected layers, DimPO increasingly outperforms KL on tasks including SQuAD, common-word extraction, frequent-word extraction, and variable tracking. These results suggest that under dimensionality reduction, preserving the ordering and concentration of task-relevant attention can matter more than reproducing the full attention distribution.

0.17

HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models

cs.CL cs.LG ·2026-10-16 ·Oscar Kemme, Janek Haberer, Ali Hojjat, Olaf Landsiedel
similarity 0.64
Abstract · Accepted at the BabyLM Workshop at EMNLP 2026

LLMs and LVLMs are increasingly deployed across heterogeneous hardware, ranging from data-center GPUs to resource-constrained edge devices. Their fixed-size architectures limit the ability to reuse a single model across deployment settings with different compute, latency, and model-size targets. We introduce HydraLLM, an elastic MQA-based architecture. Its number of active attention heads determines the effective widths of the embeddings, attention projections, feed-forward layers, and the output head, forming a nested family of subnetworks within a single parameter set. We introduce a static-to-elastic conversion pipeline that, starting from a pre-trained model checkpoint, includes importance-driven weight reordering and stochastic subnetwork training with knowledge distillation. We extend this principle to multimodal inference through an elastic projection module whose output width follows the active text width. HydraLLM covers a wider range of active parameter configurations than FFN-only elastic architectures. With 75% of heads active, HydraLLM achieves 2.2 percentage points higher TriviaQA accuracy than a state-of-the-art nested baseline configuration which has 17.25% more parameters. With 50% of heads active, our multimodal model achieves a CLIPScore comparable to that of the 7$\times$ larger Gemma 3 4B model. With only 25% of heads active, HydraLLM increases token-generation throughput by 75.3% over the full-width configuration.

0.21

From Patching to Pruning Visual Computation in Vision Language Models

cs.CV cs.LG ·2026-10-02 ·Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Shaoyi Huang, Pu Zhao +1 more
similarity 0.63
Abstract

Vision language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every decoder layer, even when token-specific visual computation is unnecessary at many depths. We introduce Patch-to-Prune (P2P), inspired by Mechanistic Interpretability, a training-free framework that converts activation patching from a diagnostic tool into an inference-time computation bypass. P2P performs validation-guided forward and backward layer sweeps to identify decoder regions whose visual-token projection outputs can be replaced by fixed neutral proxy activation vectors within a user-specified accuracy tolerance. Unlike conventional token-pruning methods, P2P preserves the sequence length, token order, positional information, attention mask, and residual pathways, thereby pruning computation without removing tokens or modifying the pretrained model weights. We evaluate P2P on four VLMs from the Qwen2.5-VL and LLaVA families across seven multi-modal benchmarks using mutually disjoint calibration, validation, and test partitions. P2P at a 3% tolerance retains around 94% of dense accuracy while reducing FLOPs by 55%. Beyond these efficiency gains, our layer-wise analysis suggests that visual processing in VLMs is non-uniformly distributed across decoder depth: early and late layers often require little token-specific visual computation, whereas intermediate layers appear to perform most task-relevant visual integration, enabling later reasoning to rely largely on visual information already embedded in shared residual and textual representations. This makes P2P both an efficient inference framework and a causal lens into visual information processing in VLMs.

0.16

BOS Summary: The First Token Matters

cs.AI cs.CLcs.CV ·2026-11-17 ·Hui Zhu, Saleh Zare Zade, Rafi Ibn Sultan, Deng Pan, Arun Ross +1 more
similarity 0.63
Abstract

Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly access carries no information about the input. We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context. At a single transformer layer during prefill, BOS Summary relaxes the causal constraint for the BOS query row alone, letting BOS aggregate all visual and textual input tokens into a self-derived context representation while every other token retains standard causal attention. The enriched representation then remains accessible to the model throughout decoding, with no additional parameters, auxiliary modules, or decoding-time overhead. Across multiple VLM and LLM families and eight multimodal and text-only benchmarks, BOS Summary improves over direct and Chain-of-Thought prompting and outperforms recent sink-intervention methods on multi-step reasoning tasks. A mechanistic analysis shows that making BOS input-specific also weakens its sink dominance and redistributes attention toward content-bearing prompt tokens, indicating that changing what a sink carries reshapes how later layers allocate attention.

0.20

Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics

cs.CL cs.LG ·2026-10-15 ·Jian Gu, Hongyu Zhang, Chunyang Chen, Aldeida Aleti
positioncode/datacommunity pick similarity 0.63
Abstract · Code: https://github.com/jianguda/lexframe

The vocabulary of a language model is its native semantics, and the embedding and unembedding fix the meaning of a state at the two ends of the residual stream. Between them, existing methods reuse a fixed lens or treat states as a point cloud, and neither reads a state as the model does. We argue that every hidden state has a native reference frame, induced by the layers that remain and adaptive to the context. Lexical reference frames approximate it in a model-native and context-aware way, reading a state through the remaining layers with a given number of rounds of attention between positions and a typical attention output for the rest, fitting nothing to the model's output. No lens with fixed directions per layer can reach a mean cosine with the native frame's lexical readouts above the square root of their coherence, 0.30 and 0.35 at mid-depth of SmolLM3-3B and OLMo3-7B. LexFrame[1], of order 1, reaches 0.97 and 0.97, and its readings are on average within 0.16 and 0.15 nats of the model's output. Its coefficient differences attribute predictions more faithfully than attribution patching, and its lexical basis steers predictions while sparing other tokens. On average, its readings settle on the model's top token over the final 0.82 and 0.81 of the layers, those of lenses over at most 0.11. These frames thus make every hidden state readable, attributable and steerable.

0.44