CS Paper Radar

Sample-efficient learning with Dense Associative Memories

Andrew Lucas, Dmitry Krotov

cs.LG ·submitted 2026-10-19 23:18 UTC · v1

Abstract

We analytically study the evolution of the energy function of Dense Associative Memories (DenseAMs) as they train on randomly sampled codewords of random expander low-density parity-check codes (a.k.a. solutions of random XORSAT problems). We identify two distinct regimes and a transition between them, reminiscent of grokking in neural networks. A regime of small training set size is where the model memorizes a few training examples but cannot discover the underlying rules of the entire codeword dictionary generative process. The opposite regime, corresponding to a larger training set size, is where the model has uncovered the sparse constraints that generated the entire data set after seeing only polynomially many patterns. We also design a novel class of DenseAMs with a thresholding learning rule, and show that any amount of thresholding enables the model to perform simplified sparse Hamiltonian learning, which exponentially reduces the number of training examples needed to learn the true underlying data set. In this regime, the model learns how to discover $\exp[\mathrm{O}(n)]$ solutions from only $\mathrm{O}(\log n)$ training samples, which is asymptotically optimal for any learning algorithm. When training on codewords of certain random codes, the model can even learn the underlying code when a small O(1) fraction of bits are latent variables. These latent variables are not shown to the model during learning, but will be discovered by sparsifying observed higher-order correlations. For certain random codes, we show that a simple and conservative algorithm learns a minimal latent representation for the data set. This provides a transparent, physics-motivated, and analytically tractable toy model for how deep latent representations emerge in an associative memory training.

Closest papers

Nearest neighbours by meaning across the whole index.

Binary Sparse Codes: Where are we on Energy Efficient Representations?

cs.LG ·2026-10-30 ·Jose Miguel Ramos, Luis SaCouto, Andreas Wichert
similarity 0.62
Abstract · 38 pages, 8 figures

Biological neural systems achieve impressive computational capabilities under tight metabolic constraints, relying on sparse cortical representations that minimise total neural activity. This combination of efficiency and representational power has motivated growing interest in sparse coding as a computational strategy for artificial systems. Artificial models designed to produce binary sparse codes span three broad paradigms, theoretical methods grounded in efficient coding, deep learning methods trained via backpropagation, and biologically plausible methods that learn through local Hebbian rules, each yielding codes whose quality depends not only on achieving low activation density but also on preserving the informational structure of the input. The latter property, however, lacks an agreed-upon quantitative definition, leaving the field without a principled basis for comparing methods. We propose a set of metrics that formalise information preservation as similarity preservation, measuring how faithfully a binary sparse code reproduces the relational geometry of the input space, and apply this framework to compare four representative methods under matched conditions. The results reveal a consistent trade-off in which models that achieve high information preservation do so at the cost of sparsity, while models approaching the uniformity of an optimal random sparse code tend to sacrifice it. Competitive Sparse Codes, a biologically plausible approach, is the only architecture to occupy a region of near-maximal sparsity while retaining meaningful information preservation. These findings frame the simultaneous satisfaction of both factors as an open challenge for the field and provide a reusable evaluation framework for measuring progress toward it.

0.33

Inference and learning in sparse autoencoders as natural gradient flow

cs.LG cs.AI ·2026-10-05 ·Hadi Vafaii, Tejas Rao, David Chanin, Thomas Fel, Jacob L. Yates +4 more
code/data similarity 0.59
Abstract · Code: https://github.com/hadivafaii/BeFOND

Sparse autoencoders are widely used to uncover interpretable features in neural networks, yet reliable recovery remains difficult when features overlap or activate infrequently. These challenges involve both inferring which features explain an input and learning the dictionary that represents them. Here, we unify inference and dictionary learning as natural-gradient flows on a shared variational free energy. We instantiate this framework as BeFOND, an encoder-free sparse coding model with closed-form inference and learning dynamics. We show how recurrent explaining away reduces interference between overlapping features, while Fisher preconditioning can compensate for the slow learning of rare features. On synthetic data, BeFOND improves dictionary recovery and rare-feature detection, with a growing advantage over amortized baselines as superposition increases. On language-model activations, it improves single-feature concept detection and selective intervention, outperforming pretrained reference SAEs with substantially less training data. Its feature quality continues to improve with dictionary width, whereas the evaluated baselines largely plateau. Together, these results show how improving inference and learning within a unified probabilistic framework can make better use of data and dictionary capacity to interpret and intervene on neural representations.

0.28

A Dominant Diffuse Phase in the Sparse Autoencoder Phase Diagram

cs.LG ·2026-09-09 ·Alexis D. Plascencia
similarity 0.58
Abstract · 13 pages, 5 figures, 1 table

Sparse autoencoders (SAEs) are increasingly used to recover interpretable features from neural-network activations, yet systematic feature co-occurrence can cause distinct features to be absorbed or merged. The MAIS-O43 open problem proposes a controlled experiment to characterize when recovery of a true synthetic dictionary gives way to feature merging as the nesting fraction $γ$, sparsity penalty $λ$, and dictionary size $M$ vary. We implement the specified protocol and evaluate 200 independently initialized fits across ten of the 165 grid cells. We observe zero full-dictionary recoveries and zero merges. Instead, every run converges to a reproducible diffuse phase: reconstruction is nearly perfect, but learned atoms typically remain far from the true features (median best cosine 0.5-0.7 against a 0.95 recovery criterion) and learned codes are an order of magnitude denser than the ground truth. This behavior persists under robustness checks and across the full 165-cell grid using standard minibatch Adam (3,300 additional fits). Since the global optimum of the exact sparse-coding objective is known to merge nested features in the two-feature case, these results suggest that trained SAEs need not reach the corresponding minima, and that the phase diagram of trained models may differ fundamentally from that of objective minimizers.

0.34

The Surprising Expressivity of a Single Neuron with Autoregressive Chain-of-Thought

cs.LG stat.ML ·2026-10-12 ·Roey Magen, Nirmit Joshi, Gal Vardi
hot direction similarity 0.58
Abstract

The success of autoregressive language models trained with Chain-of-Thought~(CoT) supervision raises a fundamental question: How simple can a next-token predictor be while still enabling the efficient simulation of general computation through its autoregressively generated CoT? Surprisingly, we show that a single neuron suffices. For a broad family of activations, including threshold and ReLU, a single autoregressive neuron can simulate any feedforward circuit with the same activation. Specifically, a size-$s$ circuit on $n$ inputs can be simulated by a neuron of dimension $O(ns+s^2)$ using $O(s^2)$ generation steps, with both bounds independent of circuit depth and tight under natural assumptions. For threshold activations, this improves upon the simulation of Joshi et al. (COLT 2025), whose bounds scale exponentially with circuit depth. Consequently, autoregressive threshold neurons can simulate polynomial-time Turing machines, while remaining efficiently learnable from CoT supervision. Our main expressivity result relies on a novel connection to Sidon sets from additive combinatorics. As an experimental illustration, we efficiently train a single autoregressive ReLU neuron to mimic a neural network teacher with nontrivial test accuracy on CIFAR-10.

0.54

Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling

cs.LG ·2026-08-24 ·Ha Dinh, Xuan Duy Ta, Khoat Than, Khac-Hoai Nam Bui
top venue similarity 0.57
Abstract · Accepted as an EMNLP 2026 Main Conference paper

Semi-structured $N$:$M$ sparsity has emerged as a practical direction for accelerating large language models (LLMs). However, existing learnable-mask approaches incur substantial parameter and memory overhead, limiting their scalability to large models and aggressive sparsity regimes. In this work, we revisit semi-structured pruning from a perspective that reconciles efficiency with scalability. We propose Reservoir of Importance (RoI), a lightweight semi-structured pruning framework that learns sparsity masks through differentiable subset sampling. Unlike prior methods that model full categorical distributions over all feasible $N$:$M$ patterns, RoI introduces a compact-logit parameterization for sparsity mask learning and performs sampling without replacement to select masks, thereby reducing trainable parameters from combinatorial complexity to $\mathcal{O}({M})$. As a result, RoI requires 1.5-8.75$\times$ fewer learnable parameters and significantly lower memory cost, while remaining fully aligned with hardware-friendly sparsity patterns. Extensive evaluations across multiple scales of the Qwen2.5 LLM family (0.5-7B parameters) demonstrate that RoI achieves competitive performance with strong memory efficiency, stability, and scalability to more aggressive $N$:$M$ sparsity patterns, offering a practical path toward efficient LLM deployment.

0.38

Learning-Theoretic Foundation for General Coded Computing: The Straggler Setting

cs.LG ·2026-08-28 ·Parsa Moradi, Behrooz Tahmasebi, Mohammad Ali Maddah-Ali
theory similarity 0.57
Abstract

Coded computing has emerged as a powerful paradigm for mitigating the impact of straggling workers in distributed computing systems. However, existing coded-computing schemes are predominantly designed for the exact recovery of highly structured computations, such as polynomial evaluation and matrix multiplication, and typically rely on strict recovery thresholds. These assumptions significantly limit their applicability to modern machine-learning workloads, particularly deep neural networks (DNNs), whose computations generally lack rigid algebraic structure and, in many applications, require only accurate approximations rather than exact recovery. To address this gap, we revisit coded computing from a learning-theoretic perspective and introduce General Coded Computing (GCC). Rather than adopting existing algebraic tools, GCC formulates coded computing through a natural end-to-end mean-squared error loss that directly measures the discrepancy between the desired computations and their recovered estimates. By deriving suitable upper bounds and restricting the encoder and decoder to a reproducing kernel Hilbert space (RKHS) with mild smoothness constraints, we show that both the encoder and decoder admit specific representations as linear combinations of RKHS kernel functions. This representation allows the corresponding coefficients to be computed efficiently. Moreover, this framework enables us to establish theoretical performance guarantees for GCC under two complementary straggler regimes. In the worst-case setting with $N$ worker nodes, and at most $S$ stragglers, we show that the end-to-end loss decays at least at rate $O(S^3N^{-3})$ for standard configurations. We then study a probabilistic setting in which each worker independently straggles with probability $p$. We prove that the expected loss can still converge at rate $O(\log_{1/p}^3(N)N^{-3})$.

0.44

Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking

cs.LG ·2026-10-08 ·Mohnish Harwani
similarity 0.56
Abstract

Flat loss landscapes have long been linked to better generalization in neural networks. However, its role as a causal mechanism for generalization is less established. Grokking provides an unique testbed to understand this distinction: models are prone to fit observed data using non-generalizing structure and remain in that regime for prolonged periods, transitioning to generalization only under particular training conditions. In this work, we study whether flat loss landscapes can act as a driving mechanism in this transition. While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions. However, when SAM is paired with mechanisms that drive generalization such as weight decay, an interesting property emerges: SAM can accelerate the transition to generalizing solutions by up to 4x at the epoch-level. We theoretically untangle this relationship between SAM and weight decay using a minimal interpolating two-layer ReLU model with both memorizing and generalizing solutions. We show that even in this simple setup, flatness alone cannot distinguish a memorizing solution from a generalizing one, while weight decay favors generalizing solutions. However, under a local stability analysis, there exists a window where a memorizing interpolant is locally stable under gradient descent but unstable under SAM in the low-norm regime, which can explain SAM's ability to accelerate this transition. Overall, our results provide a more interpretable account of the role of flatness in driving generalization, especially in settings where models are vulnerable to minimizing loss through learning non-generalizing structure.

0.10

Beyond Ground States: Learning Many-Body Hamiltonians for p-Bit Invertible Arithmetic

cs.ET cond-mat.dis-nnphysics.app-ph ·2026-11-10 ·Tian Feng, Xutang Fan, Yiyun Chen
similarity 0.56
Abstract · arXiv v1

Probabilistic-bit (p-bit) invertible logic encodes a Boolean relation as an energy whose ground states are its legal assignments, computing in any direction by sampling. Traditional constructions target ground-state membership and a wide minimum energy gap, yet finite-time samplers traverse excited states. We cast Hamiltonian design as inverse statistical mechanics: fit an energy with interactions up to order four whose Gibbs law approaches uniformity on each clamp case's legal states, under a gap floor keeping every legal state below every illegal one. Constrained maximum likelihood with pruning and refitting fits terms from a fixed pool, and every delivered energy is audited exactly on all cases. Each energy is scored by time to a verified solution under restarts with its best of nine tuned samplers. In underdetermined modes, learned multipliers factor in 0.55, 0.44, 0.52 and 0.46 [0.38, 0.56] of the hand-written encodings' updates from 4x4 to 7x7, faster within every family below 7x7 at no resolved work penalty; learned adders solve the reverse mode in 0.71 of the updates at 8 bits and about a quarter at 24 and 32 bits, at 0.55 of the work at 32 bits. At 5x5 under equal pool, budget and floor, a gap-maximizing fit reaches 0.79 [0.73, 0.88] in updates and 1.26 in work against the hand-written encoding, while likelihood fitting takes 0.55 [0.46, 0.58] and 0.58 [0.48, 0.61] of the gap fit's updates and work. Rounding coefficients to the narrowest exact integer words tested, we run both energies at each pair's common word on one FPGA color-sweep SSA core, bit-exact with software: learned multipliers take 0.41 of the hand-written encoding's cycles at 4x4 and 0.057 [0.045, 0.068] at 7x7, at up to 2.4 times its lookup tables. The method requires only a circuit's legal assignments; in determined modes, hand-written encodings take less work, and from 12 bits on the adder fewer updates.

0.38