CS Paper Radar

A Control-Theoretic Approximation to Predictive Coding Dynamics

Ryan Fayyazi, Kyle Daruwalla, Mitra Javadzadeh

cs.LG cs.AIq-bio.NC ·submitted 2026-11-19 16:04 UTC · v1

Abstract

Backpropagation remains the dominant method for training deep neural networks, but its reliance on non-local computations has motivated the search for biologically plausible alternatives. Two prominent frameworks, predictive coding (PC) and deep feedback control (DFC), offer distinct approaches to local learning, yet their relationship remains unclear. Here, we show that DFC arises as a local approximation to predictive coding dynamics near fixed points, with exact equivalence in shallow linear networks and systematic deviations arising with depth, nonlinearity, and distance from equilibrium. We characterize how this approximation, which we call Approx PC, deviates from PC in its activity dynamics, and show that these deviations can be mitigated by relinearizing the feedback controller and approaching the output target through a sequence of intermediate nudges. In networks trained on MNIST and FashionMNIST, Approx PC reaches task performance comparable to PC trained with the same nudges, and weights learned with it transfer directly to PC inference. Our results unify energy-based and control-theoretic perspectives on learning and open the door to applying tools from control theory to the training of predictive coding networks.

Comment
17 pages, 10 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Analog-Friendly Predictive Coding without Activation Derivatives

cs.LG cs.AIcs.NE ·2026-09-26 ·Francesco Innocenti
similarity 0.66
Abstract · 20 pages, 14 figures

Predictive coding (PC) is a local, energy-based alternative to backpropagation (BP) whose iterative inference dynamics make it attractive for implementation on analog hardware. However, standard nonlinear PC requires evaluating the derivative of the activation function during both inference and learning, which can be difficult to realise physically. Here, we introduce \textit{activation-matched Bregman PC}, replacing standard squared-error energies with Bregman divergences matched to the activation function. This formulation eliminates activation derivatives and, when combined with inference via mirror descent, yields local inference and learning rules requiring only weighted sums, local prediction errors, state integration, and the activation function. In digital experiments, Bregman PC performs comparably to standard PC and BP on classification and generative tasks, while preserving characteristic learning dynamics of PC and its convergence to BP under stable large-model parameterisations. These results provide a more analog-friendly formulation of nonlinear PC while retaining its key computational properties.

0.30

Natural Gradient Learning in Predictive Coding through Lateral Circuits

cs.AI ·2026-11-12 ·Lhea Beumer, Sander de Haan, Niclas Pokel, Pau Vilimelis Aceituno, Benjamin F. Grewe
similarity 0.63
Abstract

Predictive coding (PC) trains neural networks through local energy-minimization dynamics without depending on a global error signal that backpropagation (BP) requires. Although PC is a promising alternative algorithm for efficient learning, the computational cost of inference dynamics currently outweighs the resulting learning benefits in practice. In this work, we take inspiration from optimization theory to extract more learning benefit from each inference phase. We introduce Natural PC, which augments standard PC with a lateral circuit, enabling exact and efficient natural gradient learning locally in space and time. Our lateral circuit learns the activity correlations that determine the Fisher information matrix and implicitly computes its inverse, avoiding expensive matrix inversion. Across all tested image reconstruction benchmarks, Natural PC achieves state of the art in terms of sample and compute efficiency compared to existing PC approaches. Most notably, Natural PC scales to ImageNet and matches the performance of BP trained with AdamW and various second-order optimizers at an order of magnitude lower computational cost while also using two orders of magnitude fewer training samples. Our results demonstrate that local natural gradient learning makes PC competitive with BP in both sample efficiency and computational cost in reconstruction tasks.

0.22

Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks

cs.LG ·2026-10-11 ·Garrett Kinman, Zeljko Zilic
similarity 0.63
Abstract

Predictive coding networks (PCNs) promise properties not offered by backpropagation-trained networks: learning from purely local information, a single network that both discriminates and generates, and flexible inference by energy minimization. However, their iterative inference is expensive, making hardware acceleration desirable, and a useful, efficient accelerator needs both depth and aggressive quantization. Canonical state-based PC delivers neither: its inference signal decays exponentially with depth, so deep models fail to train, and low precision makes it worse. We show that pairing two recent methods carries the full PCN property set into the deep, 4-bit regime: bidirectional PC (bPC), which unifies discrimination and generation, and the error-based reparameterization (ePC), which optimizes prediction errors rather than states during inference. Error optimization delivers the output signal to every layer in a single backward pass, removing the signal decay; one additional element, global-norm clipping of the error gradients, stabilizes the error dynamics at depth. The resulting ePC+bPC model trains stably up to at least eight layers and retains its full-precision accuracy and clean class-conditional generation under W4A4E4 quantization-aware training, across FP4 and three posit formats. On six-layer networks, where state-based bPC collapses to chance, ePC+bPC classifies more than 50 accuracy points above it after one epoch of training, with class-conditional generations 5-9x better in RMSE. 4-bit quantization is nearly free: full-precision and 4-bit accuracies differ by at most ~5 points, and by under two in the best-performing formats. We characterize the two failure modes the method overcomes, bounded signal decay in state-based PC and error divergence in unregularized error optimization, and discuss implications for a 4-bit-native PCN accelerator.

0.25

Bias-Only Training Dynamics in ReLU Networks

cs.LG cs.AI ·2026-11-18 ·Shreyas Chaudhari, Jose M. F. Moura
similarity 0.61
Abstract

Neural networks with randomly initialized, fixed weights and trainable biases are universal approximators, yet the dynamics of bias-only training remain largely unexplored. We study these dynamics in one-hidden-layer ReLU networks through activation codes, the binary patterns of active neurons that partition the input space into convex polyhedral regions. We show that the gradients with respect to the biases depend on each input only through its activation code, and hence the training dynamics can be characterized with these codes. In particular, as long as the activation codes remain fixed, training reduces to a linear least-squares problem with the codes as features. Training can then be understood as a sequence of least-squares problems, with transitions occurring when neurons change their activation states. We show that the fraction of switching neurons vanishes as the network widens, and the training dynamics approach those of linear least squares on the initial activation codes in the infinite-width limit. Our experiments support these theoretical findings, showing that while activation switches can noticeably affect training in narrower networks, their relative frequency and the associated changes in the dynamics diminish as the width increases.

0.21

The Accuracy Well: Nonmonotone Accuracy Dynamics at Vanishing Initialization

cs.LG ·2026-10-14 ·Yaroslav Gusev, Eugene Golikov, Dmitry Yarotsky
theory similarity 0.61
Abstract · 124 pages, 11 figures. Submitted to ICLR-2027: https://openreview.net/forum?id=OZ6JYRP75V

We study classification dynamics under small initialization and identify a pronounced non-monotone behavior of accuracy that we call the accuracy well: accuracy rises rapidly early in training, subsequently drops, and only much later recovers. We observe this phenomenon in deep linear networks, fully connected ReLU networks on MNIST, deep convolutional networks on CIFAR-10, and controlled synthetic tasks, across different losses and optimizers and in both training and test accuracy. We propose a three-stage explanation in terms of target-driven organization, a degeneracy bottleneck, and subsequent recruitment and recovery. We develop a rigorous theory of these dynamics for shallow and deep linear networks, showing in particular that high accuracy can emerge while the predictor and the change in loss remain asymptotically vanishing, before continued optimization causes accuracy to deteriorate. For a simple synthetic classification task, we prove an extreme instance in which accuracy asymptotically rises to 100%, collapses to 0, and later recovers permanently to 100%. Our results show that classification accuracy can reveal aspects of learning dynamics that are not captured by the training loss alone.

0.35

PotentialEP: Backprop-Free Training of CIFAR-Scale Generative Models via Equilibrium Propagation

cs.LG cs.ETcs.NE ·2026-10-10 ·Alex Gower
theorycode/data similarity 0.60
Abstract · Code: github.com/alexgower/PotentialEP

Generating an image with a diffusion or flow-matching model typically involves integrating a velocity field for hundreds of sequential steps. That computation suits a GPU poorly and an analog neuromorphic substrate well, where one fast, low-power physical relaxation could replace the whole sequence. Two things are needed: a generative algorithm whose computation is a relaxation, and a way to train it without backpropagation, whose global gradient transport is hard to realise in analog. Energy Matching supplies the first, sampling by gradient-descent relaxation on a time-independent potential - but obtaining the gradient that descent follows is itself a backward pass, once per integration step. We prove that a predictive coding network with a single linear injection term added to its energy relaxes to a state whose visible layer carries that gradient inherently, with an error that vanishes for small injection strengths. An Energy Matching model trained by backpropagation can therefore be loaded into this network unchanged and sampled by relaxation alone, matching the original model's FID to within 0.04 at CIFAR-10 and ImageNet-32 scale. The network can also be trained without backpropagation: Equilibrium Propagation reproduces backpropagation's parameter updates to a cosine similarity of 0.9998, and training end to end with those updates reaches an FID of 3.64 on unconditional CIFAR-10 against 3.54 for backpropagation. To our knowledge this is the first CIFAR-10 generative model trained with Equilibrium Propagation, and the first at competitive quality by any backpropagation-free rule. This quality does not require group normalisation or attention: an architecture of only convolutions and pointwise nonlinearities, both native to an analog crossbar, reaches an FID of 3.51 when trained by backpropagation.

0.43

Approaching Backpropagation Accuracy with Weight Perturbation on MNIST

cs.LG cs.NE ·2026-10-15 ·Patrick Inoue
similarity 0.60
Abstract

Weight perturbation (WP) estimates learning directions from random parameter perturbations and a scalar change in loss, without backpropagating derivatives. We examine how batch size, training duration, and estimator stabilization affect its attainable accuracy in fully connected MNIST networks. In the shallow configuration, large-batch training reaches approximately 97\% test accuracy, compared with an early backpropagation (BP) reference near 97.5--98\%. The same comparison exposes a substantial efficiency gap: under a ten-epoch budget, shallow WP reaches 90.2\%, whereas the reported BP reference reaches 97.4\%. Epoch-indexed curves show diminishing returns from increasing batch size beyond 4,000. Learning-rate reduction and averaging independent perturbation-based updates also stabilize deeper networks, although their recorded results do not establish BP-level accuracy. We distinguish these empirical findings from a finite-horizon approximation argument: under stated regularity conditions, population WP can approximate a fixed gradient-descent trajectory arbitrarily closely at any fixed finite depth. Positive classification margins then suffice to preserve the reference predictions on a finite test set with arbitrarily high probability. Together, the results identify near-BP accuracy as a configuration- and budget-dependent possibility, rather than evidence of equal efficiency or an unconditional accuracy guarantee across depth.

0.23

Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks

cs.LG stat.ML ·2026-10-26 ·Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae
benchmark similarity 0.60
Abstract · 43 pages, 9 figures

Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the exponents quantifying how performance changes with resources. Recent theory has shown how scaling exponents are shaped by the data distribution and the structure of the learning task, but whether network depth can improve training-time scaling has remained unsettled. Through a refined analysis of deep linear networks, we clarify that the task spectrum determines when increasing network depth improves training-time scaling. For spectrally difficult tasks, increasing depth improves the asymptotic training-time scaling exponent, whereas for easy tasks, the exponent remains independent of depth, with depth affecting only a logarithmic correction at the boundary between these regimes. Remarkably, we find that a reparameterization of time governed by the predictor norm provides a unified description of the dynamics across task difficulties. Under this reparameterization, the loss follows a common scaling law whose exponent is independent of depth. Numerical experiments on a task derived from benchmark image data and with nonlinear networks exhibit qualitatively consistent task-dependent effects of depth on training-time scaling. Our findings advance the theoretical understanding of how task structure shapes the effects of network architecture on learning and offer a basis for studying scaling behavior across broader classes of neural networks.

0.22