CS Paper Radar

Align, Unify, Suppress, Route: A Coherentist View of Transformer Computation

Nura Aljaafari, Andre Freitas

cs.CL ·submitted 2026-08-22 16:29 UTC · v1

Abstract

Mechanistic interpretability has identified transformer circuits, but lacks a shared vocabulary for describing how their functions compose across tasks and architectures. We introduce Coherentist Probabilistic Compositionalism (CPC), an interpretive framework that grounds transformer computation in coherentist theories of interpretation and describes it through four operator roles. Alignment identifies candidate relations, unification integrates supporting information, suppression reduces incompatible alternatives, and routing carries selected information to the output. Across 15 models from five architecture families, the suppression, unification, and routing weight-space signatures correlate with held-out activation-level role measures above random baselines. Suppression is more stable across tasks than unification. Ablating alignment heads reduces downstream suppressive activity beyond a random-head control in 10 models, but similar effects on no-conflict prompts indicate a general upstream dependency, not contradiction-specific coupling. Explicit contradictions significantly shift a layerwise coherence proxy in 14 models; after removing shared residual covariance, the gap has the predicted direction in every model. Base and instruction-tuned variants preserve induction-head score structure ($r{\geq}0.98$) without a consistent shift of operator signatures towards later layers. These results support CPC as a shared vocabulary for comparing transformer mechanisms while showing that their depth and geometric expression remain architecture-specific.

Closest papers

Nearest neighbours by meaning across the whole index.

The Communication Map of a Transformer

cs.LG cs.CL ·2026-08-22 ·Richard Zhe Wang
code/data similarity 0.60
Abstract · 33 pages (12 pages of main text). Code and results: https://github.com/richardzhewang/communication-map

The components of a transformer communicate by writing to and reading from a shared residual stream, and the mechanistic interpretability literature has mapped these connections by hand, one circuit at a time. We present the communication map, which charts every potential communication channel from the geometry of the model's weights alone. Its unit of analysis is the whole transformer rather than the individual head or circuit. Every ordered writer-reader pair, across all 18 connection classes from head-to-head to neuron-to-neuron, receives a coupling coefficient that generalizes the composition score of Elhage et al. (2021), with an exact chance level. The census finds that 70-89% of head pairs are oriented far from chance, some coupled strongly and others actively avoiding each other. The map recovers the known induction circuits blind from the strongest head-to-head couplings and groups the heads into communities, and ablating one such community destroys the model's in-context copying. As an application, we pool the coupling coefficients of every head to identify a two-dimensional residual stream subspace, distinct from those found by activation PCA or outlier dimensions, whose deletion abolishes induction in six models up to Pythia-6.9B. We release the map, the statistical machinery, and the intervention suite.

0.55

Implicit Reasoning in Transformers: Bridging the Compositionality Gap in GPT-2-like Models

cs.CL cs.AI ·2026-10-28 ·Davide Venuto, Leonardo Venuto
code/data similarity 0.59
Abstract · Code available at: https://github.com/Davide011/implicit-reasoning-bridging-compositionality-gap

Compositional generalization remains a persistent limitation of transformer language models: even when a model learns to combine known facts correctly in-distribution (ID), the same compositional mechanism may fail when applied to out-of-distribution (OOD) knowledge. We investigate this compositionality gap in a controlled two-hop reasoning setting using a GPT-2-like transformer model trained from scratch. Building on prior work identifying a circuit associated with compositional generalization, we test whether the failure of OOD transfer is related to how atomic knowledge is structurally exposed during training. We modify the training distribution by presenting both ID and OOD atomic facts in a format positionally aligned with the second-hop retrieval step of the learned compositional circuit, without modifying the model architecture or adding OOD compositional supervision. Under this intervention, the model successfully transfers the learned ID compositional behaviour to held-out OOD atomic facts, closing the observed compositionality gap in the controlled setting. Our results provide evidence that the observed compositionality gap is not a fundamental architecture limitation and that it can be overcome through mechanistically informed training distribution design.

0.27

What Does Layer-Importance Reveal About Transformers and State-Space Models?

cs.LG cs.AI ·2026-09-15 ·Istabrak Abbes, Nizar Islah, Irina Rish, Sarath Chandar
top venue similarity 0.59
Abstract

Transformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the analytical knowledge built for transformers transfers to SSMs. We address this through the lens of layer importance which underpins compression, selective fine-tuning, and interpretability across both families. We decompose layer importance into two distinct notions. \emph{Necessity} captures how much the pretrained model depends on a layer's existing contribution, measured by the loss increase from bypassing it. \emph{Plasticity} captures where the model absorbs new information during fine-tuning, measured by the magnitude of task-specific weight updates. Our analysis reveals that the two families behave fundamentally differently: in every evaluated residual transformer up to $14$B parameters, Necessity and Plasticity anti-align across depth, whereas in the evaluated Mamba-style SSMs they point to overlapping regions. The sign of this alignment also predicts downstream adaptation behavior. In the evaluated transformers, concentrating updates in the most plastic layers increases catastrophic forgetting, while this tier-dependent effect disappears in the evaluated Mamba-style SSMs.

0.46

Component and Dimension Sparsity in Transformer Refusal Mechanisms

cs.CL cs.AIcs.LG ·2026-09-30 ·Vincent Siu, Glenn Grant-Richards, Vlad Pavlovich, Yizhou Sun, Dawn Song +1 more
publishedcode/data similarity 0.58
Abstract · Accepted to COLM 2026

Activation steering manipulates large language model behavior by intervening on internal activations, but the mechanistic basis of these interventions remains poorly understood. We decompose refusal steering into component-level interventions across four open-weight models, identifying the sparse subsets of attention and MLP components whose steering suffices to reproduce the full behavioral effect. We find that refusal directions concentrate in sparse component mechanisms comprising 28--48\% of upstream components, retaining 88--101\% of steering effectiveness. Within these mechanisms, effective steering further concentrates in approximately 50\% of residual stream dimensions, retaining 85--98\% of the component-mechanism baseline, consistent with a privileged basis structure. Sparsity thus operates at two levels: which components are steered, and which dimensions within those components carry the signal. Together these findings show that refusal is not diffusely encoded across a transformer but assembled by a structured, identifiable mechanism, providing a foundation for mechanistic understanding of how refusal behaviors are represented and steered. To facilitate reproducibility, we release all code and raw experimental results in https://github.com/wang-research-lab/Refusal_Mechanisms.

0.52

Beyond Structural Unification: A Perspective and Roadmap on Unified Multimodal Models

cs.MM ·2026-09-30 ·Yinyi Luo, Hayes Bai, Yiqiao Jin, Wenyue Hua, Srijan Kumar +3 more
publishedposition similarity 0.57
Abstract · Accepted by NeurIPS 2026 Workshop on Linguistic Principles for Foundation Models

Unified Multimodal Models (UMMs) emerge as a promising way to combine understanding and generation within a single architecture. We argue that current approaches mainly achieve structural unification, which can create a Unification Paradox: shared interfaces, token spaces, and parameters do not guarantee an overall capability gain and may degrade individual capabilities. In this position paper, we identify five critical layers in which unification must be rethought: (1) Asymmetric Input Unification, where models must estimate the utility and reliability of each modality rather than treating all modalities equally; (2) Factorized Representation Unification, where semantic abstraction and fine-grained reconstruction should use separate but coordinated capacity; (3) Stateful Reasoning Unification, where models should maintain and revise a shared multimodal belief state across understanding and generation; (4) Synergistic Optimization Unification, where conflicting understanding and generation objectives require coordinated optimization; and (5) Multi-Centric Systemic Evaluation, where benchmarks must evaluate systemic competence, capability preservation, and cross-task robustness rather than isolated outcomes. We argue that progress requires moving beyond structural unification toward functional unification across the full system.

0.43

The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models

cs.CL cs.LG ·2026-09-23 ·Muhammad Usama, Dong Eui Chang
code/data similarity 0.57
Abstract

System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA) to compare layer-wise representations under 20 system prompts in five functional categories. Effects are layer-selective and instruction-type-dependent: persona and formatting instructions deeply restructure intermediate representations, while safety instructions barely move them, producing changes statistically indistinguishable from a minimal baseline. Restrictive safety instructions and explicitly permissive ones ("you have no restrictions") engage near-identical computational pathways (mean CKA correlation 0.997), and this persists at commercial scale, where safety penetration remains below 10% even at 70B-72B. A linear probing baseline exposes the mechanism: the model encodes prompt category at every layer but restructures its computation only at a small subset, so the prompt is reliably "seen" but, for safety, not deeply "acted upon." Causal activation patching confirms these layers mediate behavioral change, and representational depth predicts behavioral effect size across the full 17-model cohort (Spearman rho = 0.761, p < 0.001). The findings provide a mechanistic explanation for the persistent jailbreak vulnerability of system-prompt-based safety. Code: https://github.com/Usama1002/system-prompt-illusion-cka

0.28

CASS: Contribution-Aware Structured Sparsity for Model Merging

cs.AI ·2026-09-28 ·Yan Li, Guiping Cao, Meng Xu, Tao Jiang, Yaguang Song +3 more
top venue similarity 0.57
Abstract · Accepted to NeurIPS 2026

Model merging integrates task-specific fine-tuned models into a single multi-task model, but often suffers from parameter interference caused by conflicting task-vector updates. Existing methods typically mitigate conflicts by pruning task vectors based on weight magnitude or random heuristics, treating Transformers as unstructured ``bags of parameters'' and overlooking their inherent modularity. In this paper, we propose \textbf{C}ontribution-\textbf{A}ware \textbf{S}tructured \textbf{S}parsity (CASS), a unified framework that reduces parameter interference by identifying and preserving task-specific components. At the core of CASS is a contribution-aware structured mask that identifies task-relevant attention heads and FFN neurons. We instantiate this mask in two settings: CASS-Merging, the primary post-hoc setting where masks serve as a plug-and-play denoising filter for existing merging operators, and CASS-Tuning, an extension for scenarios with fine-tuning access where masks constrain gradients to reduce structural overlap between task vectors. Our analysis shows that task-relevant components are sparse and partially disjoint, supporting structured component-level filtering as an effective way to reduce merging interference. Extensive experiments across vision (ViT, 20 tasks) and language (RoBERTa, 8 tasks; Qwen2.5, 4 tasks) benchmarks demonstrate that CASS improves a range of representative merging baselines.

0.58

Geometric and Behavioral Stratification in Transformer Residual Streams

cs.LG cs.CL ·2026-08-12 ·Nelson Guda
code/data similarity 0.57
Abstract · 63 pages, 10 figures, 15 tables. Code and data: https://github.com/nelsonguda/pdsf-residual-geometry

Trained transformer models develop privileged bases: coordinate axes whose statistics differ from the rest of the residual stream. But what kind of direction does such a basis select? We investigate the prediction direction, the unembedding direction of the token a model currently predicts, and find that it functions as a content-defined privileged anchor. Measured with respect to this anchor, residual-stream variation is geometrically and behaviorally stratified by proximity to the prediction. The stratification holds in all eighteen models tested (dense and mixture-of-experts, 7B-120B, base and instruction-tuned). A narrow, scale-invariant prediction interface concentrates readout-relevant structure, while the vast prediction-distal complement expands with model scale. Because the prediction direction sits nearly orthogonal to the principal variance axes, variance-based analyses recover this organization only partly, and the shortfall grows with prompt heterogeneity. Anchoring reveals a steep geometric gradient: prediction-proximal regions are highly structured and cluster related prompts, while the complement is flatter and anti-discriminates among prompt groups. The interface is a narrow slice but functionally decisive. Disrupting the variance directions closest to the prediction causes immediate divergence and frequent task-frame shifts; disrupting the next level down delays divergence and preserves framing. The complement is weakly readout-aligned per direction yet causally and temporally load-bearing, and behavior is driven by direction rather than magnitude. These results establish the prediction direction as a privileged anchor distinct from previously described coordinate axes, and give a geometric account of how high-dimensional computation coexists with linear readout.

0.60