CS Paper Radar

Phases in a class of associative memories via hidden neurons

Toshihiro Ota, Masato Taki

cs.LG cond-mat.dis-nncs.NEstat.ML ·submitted 2026-09-10 01:50 UTC · v1

Abstract

Associative memory in the Hopfield network is attractor dynamics in a disordered many-body system, and higher-order and exponential extensions turn its retrieval update into softmax attention. The polynomial and exponential regimes have been analyzed by different methods, with no common architecture in which to ask what fixes the storage scale. In this paper we study the bipartite architecture of Krotov and Hopfield, which we call the class $H$, whose model is fixed by a Lagrangian for each layer, taking the hidden neurons as the order parameter of retrieval. At polynomial load the replica method yields the replica-symmetric phase diagrams and closed-form capacities, and the crosstalk moment is common to Ising and spherical visible neurons, so their differences come from the visible entropy. With a softmax hidden layer the load is exponential, and a copy representation maps the thermodynamics onto random-energy-model counting, with paramagnetic, condensed, and frozen phases. Heating destabilizes retrieval by quantized reassignments of attention, and typical Gaussian patterns remain metastable at every load. The regimes differ in their crosstalk statistics, central-limit at polynomial load and large-deviation at exponential load, and the class $H$ splits retrieval into two roles, the visible Lagrangian fixing stability and the hidden one the storage scale, two axes that may also guide the design of new Lagrangians.

Comment
43 pages, 5 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Memory as an Energy Landscape---Hopfield

cs.NE cond-mat.dis-nnnlin.AOphysics.bio-ph+1 ·2026-09-02 ·Nima Dehghani
similarity 0.72
Abstract · Book chapter from NeuroAI: Theoretical Foundations of Dynamics, Learning and Computation in Brains, Minds, and Machines. Includes reproducib…

This chapter reconstructs the Hopfield network as a physical theory of memory rather than merely an early neural-network algorithm. It begins with the problem as it stood before 1982-threshold logic, Hebbian association, correlation memories, and recurrent binary networks-and isolates what Hopfield's synthesis added: a dynamical definition of content-addressable memory, a symmetric recurrent architecture with a Lyapunov function, a Hebbian embedding of patterns in its couplings, and a physical account of basins, robustness, and graceful degradation. The binary and graded-response energy functions are derived in full, together with the signal-crosstalk decomposition governing pattern stability, the mean-field theory of retrieval at extensive load, and the zero-temperature retrieval spinodal at (alpha 0.138) established by Amit, Gutfreund, and Sompolinsky. The energy-based program is then followed through analog optimization networks, polynomial dense associative memories, exponential interactions, and modern continuous Hopfield updates, including the precise conditions under which the update becomes scaled dot-product attention. Throughout, capacity claims are tied to their disorder ensemble, scaling limit, and success criterion, showing why numerically different storage limits need not conflict. A closing assessment distinguishes established results from surviving principles, assumption-bound limitations, and open problems, treating the Hopfield network as an effective theory whose symmetry, locality, and point-neuron assumptions delimit its biological reach. Fixed-seed numerical experiments expose the mechanisms discussed but do not substitute for analytical results.

0.26

Phase Kernel Lifts Capacity of Dense Associative Memory

cs.LG ·2026-10-15 ·Yifei Zhao, Ying Tang
published similarity 0.64
Abstract · Accepted as NeurlPS spotlight

Dense associative memory underlies modern attention mechanisms; however, Modern Hopfield Networks (MHNs) with linear kernels yield overlapping attractor basins and spurious states under high memory load, limiting their theoretically exponential storage capacity in practice. We construct a class of nonlinear phase kernels that modify the energy landscape and retrieval dynamics, and prove that these kernels preserve exponential storage capacity and local fixed-point convergence under given conditions on kernel smoothness, memory separation, and memory distributions. The framework is instantiated by the Sigmoidal Phase Hopfield Network (S-Hop), which guarantees monotone energy descent and mitigates practical capacity loss. S-Hop has $8.3\times$ and $4\times$ gains in one-step noisy retrieval load over Uniform Hopfield Networks (U-Hop) on CIFAR-10 and TinyImageNet respectively, and reaches a $50\times$ gain in first-failure load for ImageNet. The results demonstrate that phase kernels provide a principled route to robust dense associative memory.

0.26

Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems

cs.LG ·2026-10-11 ·Roberto C. Sotero, Pouya Taghipour, Jose M. Sanchez-Bornot
similarity 0.62
Abstract

Spiking Transformers achieve energy efficiency by discarding softmax normalization in favor of thresholded, binary attention. However, this removal breaks the thermodynamic framework used to analyze attention networks, as the normalized measure and partition function cease to exist. We restore this theoretical foundation by proving that softmax-free spiking attention operates exactly as the zero-temperature limit of a product of independent two-level systems. This correspondence places spiking attention strictly in the frozen ordered phase of the thermodynamic phase diagram for all inputs. Consequently, the entropic cost of organizing attention collapses, replaced by a Landauer-type physical floor governed by action-potential generation. Furthermore, this ordered-phase pinning mathematically explains the optimization difficulty caused by vanishing surrogate gradients. We validate this theory with forward-relaxation experiments, placing both standard and neuromorphic attention mechanisms on a unified thermodynamic axis.

0.25

Attention over noisy examples: generalization and memorization in the exponential Hopfield network

cs.LG cond-mat.dis-nncs.NE ·2026-10-15 ·Francesco Alemanno
code/data similarity 0.60
Abstract · 26 pages, 9 figures. Code, data and Monte Carlo results at https://github.com/francescoalemanno/hebb-attention

In Hebbian learning from examples, a Hopfield network retrieves archetypes seen only through noisy examples. We study it in the exponential Hopfield model: one softmax attention step with the unlabelled examples as keys. A signal-to-noise analysis gives the overlap with an unseen archetype in closed form: a majority-rule law in which an attention entropy replaces the dataset entropy. At high temperature the archetype is lost in the crosstalk; at low temperature the softmax freezes on the closest training example, as in a random energy model, and the layer memorizes. In between, in a window widening with the square root of the network size, the layer pools the examples of the correct class and retrieves the unseen archetype as well as their majority vote allows, at any fixed load. On structured datasets, such as MNIST and fashion-MNIST, the best accuracy lies beyond the freezing.

0.48

Hierarchical Prototype Emergence in Modern Hopfield Models

cs.AI cond-mat.dis-nncs.LG ·2026-09-10 ·Aditya Cowsik, Adithya Sriram
similarity 0.58
Abstract · 5 pages, 19 page supplementary information. Additional citation requests welcome

Hierarchical correlations are a universal feature of any realistic model of data, and the question of how associative memory models may learn these correlations and generalize beyond them to construct new sensible images is an important step towards understanding more complex modern architectures such as diffusion models. We consider a hierarchical model for memories which are sampled and stored in a dense Hopfield network with polynomial activation. We analytically derive conditions for each level of this hierarchy to be locally stable - that is they are local energy minima. We use prototype reconstruction as a minimal model of generalization and we find that it takes only a quasi-polynomial amount of information to generalize beyond particular memories and even particular groups in the hierarchy. We observe a qualitatively analogous phase diagram in the number of memories, sharpness of the activation function (polynomial degree) for data from Fashion-MNIST.

0.55

Phase Transitions in Learning Driven by Extremal Data

cs.LG cond-mat.stat-mechphysics.data-an ·2026-10-21 ·Ethan Levien, Trevor GrandPre
similarity 0.57
Abstract · 17 pages, 8 figures

We study an extremal regression problem, an example of multiple instance learning (MIL), where the predictors are given as multisets, called bags. The bag elements (called instances) possess hidden labels which are pooled to produce an observed bag-level label. Max pooling is common in MIL applications, but gives rise to vanishing gradients. To handle this, modern MIL pipelines approximate the max with a softmax, treating the sharpness as a learnable parameter, denoted $λ$. When the max pooling model is the true data generating process, learning will be successful when $λ$ is increased during gradient descent, thereby driving the dynamics into regions of parameter space with vanishing gradients. To understand the conditions under which gradient descent breaks down, we study a minimal model and derive a low dimensional projection of the gradient flow in the joint limit of large bags $n$ and high dimension $d$ with $α=\ln n/d$ fixed. The flow has two distinct phase transitions: an intrinsic freezing transition related to the random energy model of disordered systems, and a retrieval transition occurring in modern Hopfield networks. Numerical simulation confirms both the phase diagram and the convergence of the finite-size training gradient to its predicted asymptotic form.

0.35

Complexity and precision in quantum associative retrieval

cs.CC quant-phcs.IRcs.NE ·2026-11-16 ·Carlo A. Trugenberger
published similarity 0.57
Abstract

We show that efficient and high-precision quantum associative retrieval is possible with an exponential complexity advantage with respect to the classical brute force approach and with a polynomial capacity advantage with respect to classical neural network implementations for any number of patterns scaling polynomially in the number of qubits. Both classical neural and quantum associative memories store the patterns in a Hamiltonian; in the classical case the dynamics is irreversible relaxation toward attractors, in the quantum case it is unitary transformations generating an effective thermal distribution, followed by a measurement.

0.65

Memory flows: geometry and dynamics of sequential retrieval in input-driven Hopfield networks

cs.AI cond-mat.dis-nnmath.DSq-bio.NC ·2026-11-03 ·Simone Betteti, Matteo Martin, Morten Gram Pedersen, Sandro Zampieri, Giacomo Baggio
similarity 0.56
Abstract

Associative memory models classically describe retrieval as convergence to stored patterns, but the theory of sequential retrieval remains comparatively limited. We develop a two-timescale input-driven Hopfield network in which fast associative retrieval is modulated by a slow feedback variable evolving over a prescribed memory-transition graph. This yields autonomous, graph-constrained transitions while preserving analytical tractability. Using slow-fast and geometric singular perturbation theory, we characterize memory fixation via memory manifolds and identify the geometric regions that govern stability loss and escape from retrieved memories. We further derive reduced slow dynamics and explicit criteria for transition onset, self-sustained retrieval, and collapse, expressed through escape times, gain thresholds, and fixed points. Together, these results provide a tractable framework for analyzing feedback-driven sequential retrieval in Hopfield networks.

0.22