CS Paper Radar

Basin-Preserving Discretizations of Modern Hopfield Retrieval Dynamics: Energy Cells, Dissipation, and the Attention Limit

Francisco R. Villatoro

cs.NE math.NA ·submitted 2026-08-21 17:06 UTC · v1
theory

Abstract

The retrieval dynamics of a modern Hopfield network is the gradient flow of a log-sum-exp energy, while the attention update is its exact difference-of-convex minimization step. We study which time discretizations preserve not only energy decay and equilibria but also basins of attraction. We introduce energy cells, connected components of sublevel sets containing one attractor and no other critical point. Our main theorem shows that every finite energy cell below the escape energy is contained simultaneously in the basin of the continuous flow, every relaxed attention map $Ψ_θ=(1-θ)\,\mathrm{id}+θ\,\mathrm{attention}$ for $0<θ<2$, and implicit Euler throughout its uniqueness regime. A parameter-uniform unit-curvature majorant yields unconditional dissipation and a monotone interpolation of each discrete step. We also derive explicit local contraction bounds near well-separated patterns, with a certified optimal slight overrelaxation; characterize proximal tunneling and overshoot beyond the preservation regimes; compare first-order error constants; establish an order barrier for scalar reparametrizations of the relaxed family; construct a second-order scalar-auxiliary-variable scheme; and extend cell preservation to damped difference-of-convex iterations in Bregman geometry, including a certified overrelaxed window under bounded asymmetry. Nine numerical campaigns test the bounds and failure mechanisms. In two-dimensional basin experiments, all observed disagreements between continuous and discrete retrieval occur above the attractor-specific numerically inferred escape level.

Comment
37 pages, 1 figure, 4 tables. Reproducibility code and numerical data: Zenodo DOI 10.5281/zenodo.22048179

Closest papers

Nearest neighbours by meaning across the whole index.

Phase Kernel Lifts Capacity of Dense Associative Memory

cs.LG ·2026-10-15 ·Yifei Zhao, Ying Tang
published similarity 0.64
Abstract · Accepted as NeurlPS spotlight

Dense associative memory underlies modern attention mechanisms; however, Modern Hopfield Networks (MHNs) with linear kernels yield overlapping attractor basins and spurious states under high memory load, limiting their theoretically exponential storage capacity in practice. We construct a class of nonlinear phase kernels that modify the energy landscape and retrieval dynamics, and prove that these kernels preserve exponential storage capacity and local fixed-point convergence under given conditions on kernel smoothness, memory separation, and memory distributions. The framework is instantiated by the Sigmoidal Phase Hopfield Network (S-Hop), which guarantees monotone energy descent and mitigates practical capacity loss. S-Hop has $8.3\times$ and $4\times$ gains in one-step noisy retrieval load over Uniform Hopfield Networks (U-Hop) on CIFAR-10 and TinyImageNet respectively, and reaches a $50\times$ gain in first-failure load for ImageNet. The results demonstrate that phase kernels provide a principled route to robust dense associative memory.

0.26

Memory flows: geometry and dynamics of sequential retrieval in input-driven Hopfield networks

cs.AI cond-mat.dis-nnmath.DSq-bio.NC ·2026-11-03 ·Simone Betteti, Matteo Martin, Morten Gram Pedersen, Sandro Zampieri, Giacomo Baggio
similarity 0.60
Abstract

Associative memory models classically describe retrieval as convergence to stored patterns, but the theory of sequential retrieval remains comparatively limited. We develop a two-timescale input-driven Hopfield network in which fast associative retrieval is modulated by a slow feedback variable evolving over a prescribed memory-transition graph. This yields autonomous, graph-constrained transitions while preserving analytical tractability. Using slow-fast and geometric singular perturbation theory, we characterize memory fixation via memory manifolds and identify the geometric regions that govern stability loss and escape from retrieved memories. We further derive reduced slow dynamics and explicit criteria for transition onset, self-sustained retrieval, and collapse, expressed through escape times, gain thresholds, and fixed points. Together, these results provide a tractable framework for analyzing feedback-driven sequential retrieval in Hopfield networks.

0.22

Mean-Field Oscillator Ising Machines: Gradient Flows and Classification of Limit Solutions

eess.SY math.OCmath.DS ·2026-08-17 ·Arvind R. Venkatakrishnan, Max Emerick, Bassam Bamieh, Francesco Bullo
similarity 0.58
Abstract · 31 pages, 5 figures

Oscillator Ising Machines (OIMs) have emerged as promising computational architectures for approximating solutions to combinatorial optimization problems. We derive and analyze the mean-field limit of an OIM model and show that it inherits the gradient flow structure of the finite-dimensional dynamics. We identify conditions under which this mean-field evolution admits an Eulerian formulation as a gradient flow on the Wasserstein space of probability measures and contrast this with a Lagrangian formulation which is always available. This gradient flow structure strongly constrains the long-time dynamics and enables a complete classification of limit solutions and their stability in the totally symmetric case. In particular, all limit solutions are fixed points whose phases cluster into at most four groups, and for almost all parameter values, only binarized fixed points -- those with clusters at $0$ and/or $π$ -- can be stable. Since binarized states are exactly those for which a feasible solution to the original problem can be read out, this shows that feasible solutions can almost always be recovered. We provide tight bounds on the parameter thresholds for which fixed points in this binarized family are stable, thereby identifying the threshold for binarization in this model, and present numerical evidence that the mean-field model correctly predicts behavioral regimes in certain large random networks, including Erdős-Rényi networks. Moreover, this problem provides a case study in the application of mean-field models, gradient flows, and optimal transport to the study of large-scale systems, and we highlight a number of features of this problem which are informative in this respect.

0.41

Memory as an Energy Landscape---Hopfield

cs.NE cond-mat.dis-nnnlin.AOphysics.bio-ph+1 ·2026-09-02 ·Nima Dehghani
similarity 0.58
Abstract · Book chapter from NeuroAI: Theoretical Foundations of Dynamics, Learning and Computation in Brains, Minds, and Machines. Includes reproducib…

This chapter reconstructs the Hopfield network as a physical theory of memory rather than merely an early neural-network algorithm. It begins with the problem as it stood before 1982-threshold logic, Hebbian association, correlation memories, and recurrent binary networks-and isolates what Hopfield's synthesis added: a dynamical definition of content-addressable memory, a symmetric recurrent architecture with a Lyapunov function, a Hebbian embedding of patterns in its couplings, and a physical account of basins, robustness, and graceful degradation. The binary and graded-response energy functions are derived in full, together with the signal-crosstalk decomposition governing pattern stability, the mean-field theory of retrieval at extensive load, and the zero-temperature retrieval spinodal at (alpha 0.138) established by Amit, Gutfreund, and Sompolinsky. The energy-based program is then followed through analog optimization networks, polynomial dense associative memories, exponential interactions, and modern continuous Hopfield updates, including the precise conditions under which the update becomes scaled dot-product attention. Throughout, capacity claims are tied to their disorder ensemble, scaling limit, and success criterion, showing why numerically different storage limits need not conflict. A closing assessment distinguishes established results from surviving principles, assumption-bound limitations, and open problems, treating the Hopfield network as an effective theory whose symmetry, locality, and point-neuron assumptions delimit its biological reach. Fixed-seed numerical experiments expose the mechanisms discussed but do not substitute for analytical results.

0.26

Two-Clustering Regime of Token Dynamics in Causal Attention

cs.AI ·2026-11-13 ·Trinh T. Nguyen, Duy-Tung Pham, Hoang-Son Do, Tan Minh Nguyen, Thieu N. Vo
theory similarity 0.58
Abstract

We partially resolve Conjecture 1 of [Karagodin et al., Clustering in Causal Attention Masking (NeurIPS 2024)] and extend it to time-dependent parameter regimes. The conjecture predicts that causal attention dynamics converge to two clusters aligned with the eigenvectors associated with the largest eigenvalue $λ_{\max}$ of the value transformation. We prove that this behavior holds when $λ_{\max}>0$ and is generally false when $λ_{\max}\le0$. The resulting two-cluster regime is common in practice and substantially more challenging than the classical single-cluster setting studied previously. A central contribution of our work is the identification of a rigorous dimension reduction phenomenon: despite evolving in a high-dimensional embedding space, token trajectories collapse onto a two-dimensional structure governed by a small number of dominant directions. Building on this insight, we introduce a new analytical framework--instability within the flow--inspired by instability phenomena in physical systems. This framework enables the analysis of steady-regime behavior in non-monotone, non-autonomous attention dynamics. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.

0.15

The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow

cs.LG math.OCstat.ML ·2026-09-01 ·Raphaël Berthier
similarity 0.57
Abstract

The central flow of Cohen et al. (2025) is an empirically accurate continuous-time model of gradient descent at the edge of stability in deep learning, However, its derivation is heuristic. We propose a perturbative regime in which the central flow is the limit of gradient descent: we assume that the loss decomposes as $f = g + \varepsilon h$; in the limit $\varepsilon \to 0$, the dynamics of gradient descent with learning rate $η$ converge to the gradient flow of $h$ constrained to the minimizers of $g$ of sharpness at most $2/η$. Our approach is formal rather than rigorous; it treats gradient descent as a singularly perturbed dynamical system in $\varepsilon$. Three timescales emerge: a fast timescale of oscillations along the sharpest direction, an intermediate timescale of the self-stabilization mechanism, and a slow timescale of the dynamics along the minimizers of $g$-the central flow. Using the method of multiple scales, a classical formal method from singular perturbation theory, we derive the expansion of the dynamics in $\varepsilon$: the central flow emerges as the leading-order term in the expansion, while the self-stabilization mechanism appears in the next-order term. We study this mechanism beyond previous analyses: with a single eigenvalue at the edge of stability, we compute the slow drift of the energy of the fluctuations; with several eigenvalues at the edge of stability, we derive the self-stabilization system and explain why fluctuations persist.

0.16

Optimal exponential memory for sequential Euclidean connections: edge-power costs and phase transitions

cs.CG math.OC ·2026-08-27 ·Pedro M. M. de Castro
code/data similarity 0.55
Abstract · 60 pages, 8 figures, 2 tables. Substantially expanded version with finite-size phase transitions, high-power adversarial bounds, high-dimens…

We study the edge-power cost of the labelled tree generated by the $γ$-strategy, a constant-gain rule for sequential Euclidean connections. Starting with $x_0=p_0$, each input point $p_i$ is attached to $x_{i-1}$, and the state is updated by $x_i=γx_{i-1}+(1-γ)p_i$. Retaining $x_i$ subdivides the insertion segment into a spine edge and a leaf edge. The memory parameter $γ$ controls how long earlier points influence subsequent attachment points. We minimize the sum of the $α$-powers of the edge lengths under independent uniform input and arbitrary input sequences. For uniform points in the unit ball, the stationary problem has a transition at $α=1$. Its continuous extension is minimized at the boundary for $0<α\leq1$, while every global minimizer is interior for $α>1$. We determine the finite optimizer in the joint window $α_N=1+\varepsilon_N$, $\varepsilon_N\log N\toλ$. Below an explicit threshold it lies on the $N^{-1/2}$ scale, at the threshold its scale is $\sqrt{\log N/(N\log\log N)}$, and above the threshold it approaches an explicit stationary root with two computable corrections. A second threshold identifies the governing correction, and differentiated estimates prove eventual uniqueness. At $α=3d+8$, the linear coefficient at the stationary endpoint changes sign and a branch of strict local maxima enters the parameter interval. For arbitrary input sequences, the optimal parameter and asymptotic worst-case edge-power cost per point are explicit for $0<α\leq3$. At high powers, periodic antipodal block inputs give explicit lower bounds which, with a separation argument, show that the optimized cost is asymptotic to $2\log2/\logα$. Exact results for powers two and four, a rational recursion for every even power, and a high-dimensional expansion complete the analysis.

0.31

Physics of Information Geometry - Part I: Principle of Least Action on the Probability Simplex

cs.IT cond-mat.stat-mech ·2026-09-08 ·C. Emre Koksal, Deniz Sargun
similarity 0.55
Abstract

We develop a least-action framework for describing how a probability distribution can evolve from an equilibrium state to a prescribed nonequilibrium state under constrained incremental changes. Taking a Gibbs distribution as the equilibrium reference, the framework gives a direct physical meaning to the geometry of the probability simplex: distance from equilibrium corresponds to nonequilibrium free energy, while changes between successive distributions carry an informational kinetic cost. The Pythagorean structure of relative entropy then provides the central insight of the work. It shows that intermediate distributions chosen via sequential information projections can reduce the kinetic cost of large transitions and establishes an energy-conservation-like relation between the kinetic expenditure along a path and the free energy accumulated in reaching the target distribution. Motivated by this geometry, we construct a greedy least-action path through successive information projections, obtain a closed-form characterization of each projection through the Lambert W function, and establish a finite-step performance guarantee. We further show that state-dependent costs can be incorporated naturally by reshaping the underlying Gibbs reference, providing a thermodynamic interpretation of path penalties as modifications of the effective energy landscape. Together, these results provide a unified view of distributional evolution through least action, information geometry, and nonequilibrium thermodynamics.

0.40