CS Paper Radar

Analog-Friendly Predictive Coding without Activation Derivatives

Francesco Innocenti

cs.LG cs.AIcs.NE ·submitted 2026-09-26 08:13 UTC · v1

Abstract

Predictive coding (PC) is a local, energy-based alternative to backpropagation (BP) whose iterative inference dynamics make it attractive for implementation on analog hardware. However, standard nonlinear PC requires evaluating the derivative of the activation function during both inference and learning, which can be difficult to realise physically. Here, we introduce \textit{activation-matched Bregman PC}, replacing standard squared-error energies with Bregman divergences matched to the activation function. This formulation eliminates activation derivatives and, when combined with inference via mirror descent, yields local inference and learning rules requiring only weighted sums, local prediction errors, state integration, and the activation function. In digital experiments, Bregman PC performs comparably to standard PC and BP on classification and generative tasks, while preserving characteristic learning dynamics of PC and its convergence to BP under stable large-model parameterisations. These results provide a more analog-friendly formulation of nonlinear PC while retaining its key computational properties.

Comment
20 pages, 14 figures

Closest papers

Nearest neighbours by meaning across the whole index.

A Control-Theoretic Approximation to Predictive Coding Dynamics

cs.LG cs.AIq-bio.NC ·2026-11-19 ·Ryan Fayyazi, Kyle Daruwalla, Mitra Javadzadeh
similarity 0.66
Abstract · 17 pages, 10 figures

Backpropagation remains the dominant method for training deep neural networks, but its reliance on non-local computations has motivated the search for biologically plausible alternatives. Two prominent frameworks, predictive coding (PC) and deep feedback control (DFC), offer distinct approaches to local learning, yet their relationship remains unclear. Here, we show that DFC arises as a local approximation to predictive coding dynamics near fixed points, with exact equivalence in shallow linear networks and systematic deviations arising with depth, nonlinearity, and distance from equilibrium. We characterize how this approximation, which we call Approx PC, deviates from PC in its activity dynamics, and show that these deviations can be mitigated by relinearizing the feedback controller and approaching the output target through a sequence of intermediate nudges. In networks trained on MNIST and FashionMNIST, Approx PC reaches task performance comparable to PC trained with the same nudges, and weights learned with it transfer directly to PC inference. Our results unify energy-based and control-theoretic perspectives on learning and open the door to applying tools from control theory to the training of predictive coding networks.

0.35

Natural Gradient Learning in Predictive Coding through Lateral Circuits

cs.AI ·2026-11-12 ·Lhea Beumer, Sander de Haan, Niclas Pokel, Pau Vilimelis Aceituno, Benjamin F. Grewe
similarity 0.66
Abstract

Predictive coding (PC) trains neural networks through local energy-minimization dynamics without depending on a global error signal that backpropagation (BP) requires. Although PC is a promising alternative algorithm for efficient learning, the computational cost of inference dynamics currently outweighs the resulting learning benefits in practice. In this work, we take inspiration from optimization theory to extract more learning benefit from each inference phase. We introduce Natural PC, which augments standard PC with a lateral circuit, enabling exact and efficient natural gradient learning locally in space and time. Our lateral circuit learns the activity correlations that determine the Fisher information matrix and implicitly computes its inverse, avoiding expensive matrix inversion. Across all tested image reconstruction benchmarks, Natural PC achieves state of the art in terms of sample and compute efficiency compared to existing PC approaches. Most notably, Natural PC scales to ImageNet and matches the performance of BP trained with AdamW and various second-order optimizers at an order of magnitude lower computational cost while also using two orders of magnitude fewer training samples. Our results demonstrate that local natural gradient learning makes PC competitive with BP in both sample efficiency and computational cost in reconstruction tasks.

0.22

PotentialEP: Backprop-Free Training of CIFAR-Scale Generative Models via Equilibrium Propagation

cs.LG cs.ETcs.NE ·2026-10-10 ·Alex Gower
theorycode/data similarity 0.65
Abstract · Code: github.com/alexgower/PotentialEP

Generating an image with a diffusion or flow-matching model typically involves integrating a velocity field for hundreds of sequential steps. That computation suits a GPU poorly and an analog neuromorphic substrate well, where one fast, low-power physical relaxation could replace the whole sequence. Two things are needed: a generative algorithm whose computation is a relaxation, and a way to train it without backpropagation, whose global gradient transport is hard to realise in analog. Energy Matching supplies the first, sampling by gradient-descent relaxation on a time-independent potential - but obtaining the gradient that descent follows is itself a backward pass, once per integration step. We prove that a predictive coding network with a single linear injection term added to its energy relaxes to a state whose visible layer carries that gradient inherently, with an error that vanishes for small injection strengths. An Energy Matching model trained by backpropagation can therefore be loaded into this network unchanged and sampled by relaxation alone, matching the original model's FID to within 0.04 at CIFAR-10 and ImageNet-32 scale. The network can also be trained without backpropagation: Equilibrium Propagation reproduces backpropagation's parameter updates to a cosine similarity of 0.9998, and training end to end with those updates reaches an FID of 3.64 on unconditional CIFAR-10 against 3.54 for backpropagation. To our knowledge this is the first CIFAR-10 generative model trained with Equilibrium Propagation, and the first at competitive quality by any backpropagation-free rule. This quality does not require group normalisation or attention: an architecture of only convolutions and pointwise nonlinearities, both native to an analog crossbar, reaches an FID of 3.51 when trained by backpropagation.

0.43

Contrastive local learning in large nonlinear analog circuits

cs.LG physics.app-ph ·2026-11-12 ·Pragyan Pandey, Peter Wu, Deema Alyahya, Eric Y. Ma
similarity 0.63
Abstract · 19 pages (13 main, 6 supplementary) and 11 figures (5 main, 6 supplementary)

Machine learning in physical hardware promises to reduce training cost by colocating computation, memory and adaptation within device physics. Contrastive local learning networks (CLLNs) offer a compelling route: an electrical circuit relaxes to equilibrium in free and weakly clamped states, and each adaptive element updates from local voltage contrasts rather than backpropagated error signals. Yet whether nonlinear transistor implementations can scale beyond proof-of-concept circuits, and what design principles control that scaling, have remained unknown. Here we show that nonlinear analog circuits trained by contrastive local rules can learn larger machine-learning tasks and remain robust under hardware nonidealities. Using device-level circuit simulations, we systematically explore initialization, operating regime, loss-dependent clamping and training parameters, and scale CLLNs beyond ten thousand adaptive edges. The same local rule trains classifiers that reach the accuracy regime of conventional digital methods on Ionosphere and handwritten-digit benchmarks, and a network that autoregressively generates language over a 122-token vocabulary. These results establish the feasibility of local learning in large nonlinear analog circuits.

0.20

Model Predictive Control on Analogue Hardware: A Dual-flow Approach

eess.SY ·2026-11-11 ·Nikilesh Ramesh, Ross Drummond, Pablo R Baldivieso-Monasterios, Morgan Jones, Yuanbo Nie
similarity 0.61
Abstract

Current Model Predictive Control (MPC) implementations are predominantly digital and, despite offering high precision and flexibility, often face challenges with ultra-low latency and hardware certification for safety-critical applications. We present an approach for solving the Quadratic Program (QP) of MPC using a continuous-time neural ordinary differential equation in the dual space. We obtain the solution to these QPs by mapping the differential equation's equilibrium point back to the primal space via a nonlinear transformation. The underlying dual-flow dynamics operate natively on analogue hardware. We validate this approach by demonstrating the hardware implementation of linear MPC on "The Analog Thing" analogue computer. Finally, we extend the framework to nonlinear MPC with successive linearisation, broadening the applicability beyond linear systems.

0.22

Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks

cs.LG ·2026-10-11 ·Garrett Kinman, Zeljko Zilic
similarity 0.61
Abstract

Predictive coding networks (PCNs) promise properties not offered by backpropagation-trained networks: learning from purely local information, a single network that both discriminates and generates, and flexible inference by energy minimization. However, their iterative inference is expensive, making hardware acceleration desirable, and a useful, efficient accelerator needs both depth and aggressive quantization. Canonical state-based PC delivers neither: its inference signal decays exponentially with depth, so deep models fail to train, and low precision makes it worse. We show that pairing two recent methods carries the full PCN property set into the deep, 4-bit regime: bidirectional PC (bPC), which unifies discrimination and generation, and the error-based reparameterization (ePC), which optimizes prediction errors rather than states during inference. Error optimization delivers the output signal to every layer in a single backward pass, removing the signal decay; one additional element, global-norm clipping of the error gradients, stabilizes the error dynamics at depth. The resulting ePC+bPC model trains stably up to at least eight layers and retains its full-precision accuracy and clean class-conditional generation under W4A4E4 quantization-aware training, across FP4 and three posit formats. On six-layer networks, where state-based bPC collapses to chance, ePC+bPC classifies more than 50 accuracy points above it after one epoch of training, with class-conditional generations 5-9x better in RMSE. 4-bit quantization is nearly free: full-precision and 4-bit accuracies differ by at most ~5 points, and by under two in the best-performing formats. We characterize the two failure modes the method overcomes, bounded signal decay in state-based PC and error divergence in unregularized error optimization, and discuss implications for a 4-bit-native PCN accelerator.

0.25

Nonlinear Dynamical Systems as Scalable and Efficient Physical Neural Networks

cs.LG cond-mat.dis-nncs.ETcs.NE ·2026-10-15 ·Tyler King, Magnus Saebo, Jeremie Laydevant, Haoting Zhen, Maxwell G. Anderson +2 more
similarity 0.59
Abstract · 32 pages, 10 figures

The rapid growth in the size of state-of-the-art machine-learning (ML) models has led to increasing compute demands, spurring the development of energy-efficient hardware accelerators. Physical systems trained to perform ML inference, known as physical neural networks, have been proposed as accelerators in part because of their potential to be highly energy efficient. However, the ML inference accuracies reported in most investigations of physical neural networks have been substantially below the state of the art for conventional digital neural networks. In this work, we show in simulation studies that by applying modern ML training practices, including model compression, various candidate physical neural networks based on nonlinear dynamical systems can achieve accuracies as high as 99.2% on MNIST handwritten-digit recognition. We also show for one dynamical system that using it as the core building block of a parameter-efficient ML architecture inspired by MLP-Mixers enables accuracies of up to 86.4% on CIFAR-10 image classification (92.2% with pretraining and knowledge distillation), compared with 64% for the same system without the mixer architecture. The mixer approach achieved its accuracy while using $>5\times$ fewer trainable parameters than the baseline. Finally, we present back-of-the-envelope estimates for inference energy efficiency and speed when using an example physical dynamical system, suggesting that orders of magnitude improvements may be possible over digital-electronic neural networks.

0.27

Lock-in EP: An In-Situ Training Algorithm for Oscillatory Hardware

cs.LG ·2026-10-05 ·Sowjanya Tammali, Wilkie Olin-Ammentorp
similarity 0.58
Abstract

Analog hardware platforms offer the potential to reduce energy consumption over digital architectures, but in order to succeed, large-scale analog systems must also be able to operate with or recover from the variability of their components. Towards this goal, we derive and demonstrate the lock-in equilibrium propagation (LIEP) training method. LIEP provides local gradient information for each component in an oscillatory network without separate forward and backward sweeps, potentially allowing for in-situ learning capabilities on analog oscillatory hardware platforms. We demonstrate that LIEP can be used both for ab-initio training as well as recovering performance when pre-trained parameters are perturbed. We show that LIEP can be formulated as a three-factor update rule, and suggest that although the method is currently only validated on shallow networks, alternate architectures may allow it to extend to deep and large-scale networks addressing complex tasks.

0.21