CS Paper Radar

Feature Superposition in Neural Networks: From Theory to Practice

Dai Shi, Xiaoyu Li, Andi Han, José Miguel Hernández-Lobato

cs.LG ·submitted 2026-09-06 22:24 UTC · v2
code/data

Abstract

Superposition refers to neural networks representing more features than they have dimensions. It was proposed to explain polysemantic neurons, which respond to several features, and also other phenomena, such as neural scaling laws and adversarial vulnerability. A feature stored in superposition is useful to the extent that its value can be read back despite interference from the other features. The theory of superposition mainly asks how wide a layer must be for the features to be read back, and by which decoder. In theoretical models, a single linear map that reads back every combination of nonzero features needs a width nearly quadratic in their number, while sparse optimization can succeed with a width linear in that number. In practice, one may hope to interpret a trained network by recovering its features in the same way. However, its features are usually unknown, and thus they are estimated from its activations or weights by recovery methods, such as sparse autoencoders and transcoders. In this work, we organize the theory of superposition by the decoder class and its practice by the strength of the claims about a feature supported by each method's results. On the theory side, we review the lower bounds on the overlap between feature directions, the conditions under which trained toy models share dimensions among features, and the separation between storing features and computing new features. On the practice side, we categorize recovery methods into four families, from probes for a named feature to decompositions of the trained weights. We then compare their evaluations, failure modes, and applications, and grade the evidence on one scale of claims about a feature. Finally, we assess previously stated open problems and identify remaining theoretical and empirical questions about superposition in trained networks. Project page: https://eethanshi.github.io/awesome-superposition/

Closest papers

Nearest neighbours by meaning across the whole index.

High-probability guarantees for linear accessibility in feature superposition

cs.AI stat.MLcs.IRcs.LG+1 ·2026-09-09 ·Enrico Vompa
similarity 0.72
Abstract · preprint

Neural networks can leverage feature superposition to encode more concepts than dimensions, but cross-feature interference constrains the linear accessibility of simultaneously active features. By framing linear accessibility as a compressed sensing problem, we derive high-probability bounds for fixed supports under subgaussian noise, proving the sufficient dimension scales linearly ($d=O_{\varepsilon}(k \log m)$) rather than prior worst-case quadratic limits. We characterize the asymmetry between active and inactive interference and the trade-off between interference and observation-noise budgets. We then validate these bounds across system parameters through Gaussian-tail approximations. We also introduce IHT-SAE, which uses learned iterative refinement to improve feature recovery beyond the limits of linear availability. These results quantify the geometric constraints of the linear representation hypothesis, providing a framework for evaluating sparse autoencoders, compositional generalization, and neural interpretability.

0.18

A Distribution-Free Separation Between Learned and Fixed Features

cs.LG stat.ML ·2026-10-18 ·Daniel Barzilai
hot direction similarity 0.62
Abstract

The success of neural networks is largely attributed to their ability to learn useful representations from training data. However, known results that prove a computational advantage of learned features over fixed ones are restricted to specific input distributions. We settle the COLT open question of Feldman, Kamath, and Srebro (2026) by showing that this advantage can persist in the distribution-free setting. We construct a family of hypothesis classes that can be efficiently learned by a natural gradient-based algorithm on a two-layer ReLU network, but cannot be represented by linear predictors over any fixed feature map with polynomial dimension (and runtime). Specifically, for any sufficiently small error and any sufficiently large runtime budget R (depending on the error), there exists a hypothesis class that the neural network learns under any input distribution to the desired expected classification error in time at most R, while the dimension complexity (governing the dimension needed by linear predictors) is superpolynomial in R.

0.33

Towards a mathematical theory of superposition

cs.IT stat.MLcs.LGmath.CO ·2026-08-27 ·Michael I. Ivanitskiy, John Jasper, Emily J. King, Dustin G. Mixon
theory similarity 0.62
Abstract

We develop a mathematical theory of superposition in neural networks using tools from frame theory and compressed sensing. In our model, a sparse binary vector \(x\) of active features is encoded through an overcomplete dictionary \(W\), and feature recovery is performed by applying \(\operatorname{ReLU}(W^\top W x+b)\) with an appropriate bias vector \(b\). We prove several recovery theorems for this model. In the random-support setting, we establish high-probability support recovery for nearly tight, low-coherence dictionaries, with guarantees when the expected sparsity is up to order \(d/\log n\). In the worst-case support setting, we give a sharp and computable criterion for which sparsity levels permit support recovery. We apply this criterion to Gaussian random matrices and equiangular tight frames. For real equiangular tight frames with \(n>d+1\), we determine the exact recovery threshold in terms of the coherence. The proof of this result for real equiangular tight frames relies on a novel characterization---which should be of independent interest to frame theorists---of the distribution of signs in the Gram matrix.

0.47

Sparse Transformer: a Controlled Testbed for Mechanistic Interpretability

cs.AI ·2026-10-14 ·Anton Korznikov, Alexey Dontsov, Elena Tutubalina
similarity 0.61
Abstract

Mechanistic interpretability aims to decompose neural network computations into human-understandable features, yet progress is limited by the absence of ground-truth units to evaluate against. We address this with a Sparse Transformer (ST): a 5-layer language model trained on TinyStories whose wide, TopK-sparse MLP forces neuron specialization. We show that ST neurons are highly interpretable: they outperform a loss-matched dense transformer by 43% on AutoInterp, and circuits of a median of 12 neurons suffice to reproduce individual model predictions. This testbed allows us to directly compare feature learning methods: Transcoder, which approximates the MLP input-output mapping, recovers ST's neuron vectors with 52.6% higher cosine similarity than Sparse Autoencoder, which only reconstructs outputs. Furthermore, the ST enables a controlled study of polysemanticity: distilling its sparse MLP into a dense counterpart reveals how compression leads to feature superposition. These results show how a known reference basis sharpens evaluation and understanding, a role we hope the ST can serve for the field.

0.19

Emergent phases of superposition: from partial to full representation

cs.LG ·2026-09-29 ·Lihao Guo, Yizhou Liu, Jeff Gore
similarity 0.61
Abstract · 35 pages, 25 figures

Large language models are thought to represent features by vectors in a hidden space of dimension given by the model's width. Superposition, in which more features are represented than the width by letting representation vectors overlap, is a leading account of how representation vectors are organized. However, how model width and data statistics determine the configuration of representation vectors and the resulting loss when the number of features and the width are large remains less understood. Here we show, in Anthropic's toy model of superposition, that increasing the width drives a continuous phase transition from a partial-representation phase, where only a subset of features receives appreciable representation vectors while the rest vanish, to a full-representation phase, where every feature is represented. Our theory via a partial random projection approximation predicts, and experiments confirm, that the critical width grows linearly with the number of active features up to a logarithmic factor. The loss scaling changes across the transition: below the critical width, the loss grows linearly with the number of active features and depends weakly on the width in a form set by data statistics; above it, the loss grows approximately quadratically with the number of active features and decays inversely with the width. Non-uniform firing probabilities delay the transition and lower the loss, as more frequent features occupy more space. Our results provide an account of how model width and data statistics jointly shape representations and loss, a step toward understanding representation scaling in large models.

0.21

Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise

cs.AI ·2026-10-17 ·Stepan Shabalin, Mateusz Piotrowski
publishedcode/data similarity 0.60
Abstract · 60 pages. NeurIPS InterpScience workshop

MLP neurons in transformer language models are widely understood to be polysemantic, motivating transcoders - sparse, learned approximations of MLP layers - as interpretable replacement models for mechanistic analysis. However, recent work shows that MLP neurons can be used to recover sparse, interpretable circuits (Arora et al. 2026, Ye et al. 2025), raising the question of whether transcoders are necessary for circuit analysis. We define neuron features as transcoder features whose weights are strongly aligned with individual MLP neurons, and show that they account for a substantial fraction of active features in late layers across 13 open-source transcoders. Only neuron features have a stable correspondence with something in the underlying MLP: other features vary more between training runs, do not match the MLP's Jacobian, and change less significantly from their random initialization. Our findings reframe MLP neurons as a stronger basis for interpretability than commonly assumed, and call for better methods of attributing the non-neuron structure that is used by the model.

0.45

Why Does Robustness Reduce Superposition?

cs.LG cs.AI ·2026-08-23 ·Adam Elimadi
similarity 0.58
Abstract · 9 pages, 4 figures. Accepted at the COLM 2026 Workshop on AI Interpretability (AIW)

The study of adversarial examples and their origins remains an open area of research. Mechanistic interpretability, and superposition in particular, offers new avenues for approaching this problem. Gorton & Lewis (2025) demonstrate that adversarial examples arise from superposition and show empirically that adversarial training reduces superposition, yet provide no mechanistic account of why this occurs. We present an empirical explanation inspired by the feature taxonomy of Ilyas et al. (2019), tracing the following chain of causalities: adversarial training abandons non-robust features, leading to fewer total features to represent, resulting in less superposition.

0.54

The Ball and the Box: Two Geometries of Computation in Superposition

cs.LG ·2026-10-08 ·Xiaoyu Li, Lequan Lin, Dai Shi, Jiaojiao Jiang, Junbin Gao +1 more
similarity 0.58
Abstract

Neural representations can encode more features than they have dimensions, a phenomenon known as superposition. We study the dimension needed to compute Boolean gates from such representations. For a single threshold layer with a Gaussian random dictionary and uniformly random sparse Boolean inputs, we derive sharp dimension thresholds under two error criteria. A vanishing expected error count can require more dimensions than correctness of every output with high probability. Shared reads explain the gap: rare realizations can produce many errors at once. The expected-count threshold has ball geometry, while joint reliability has box geometry when a gate is evaluated on every feature tuple. Optimizing shared readout weights and biases gives explicit thresholds for conjunction, disjunction, and majority. For pairwise conjunction, the analysis also describes the transition near the threshold, in agreement with exact simulations.

0.25