CS Paper Radar

Distributional statistics of nonequilibrium

Harrison Hartle, Aanjaneya Kumar, David H. Wolpert

cs.IT cond-mat.stat-mechmath.PRphysics.data-an ·submitted 2026-10-24 00:49 UTC · v1

Abstract

We establish a reference modeling framework for random probability distributions subject to constraints. Generalizing the hyperensemble approach of Crooks, we formulate Jaynesian priors of maximal uncertainty over the unit simplex of categorical distributions. This framework is entirely general, but we primarily investigate models with constraints on physically relevant quantities, namely, average energy and Shannon entropy. A constraint on the exact value of average energy uniquely determines a Boltzmann-Gibbs equilibrium, but also defines a hyperplanar subset of the probability simplex consisting of all distributions with that value of average energy. The associated hyperensemble has a uniform probability density across all such distributions, regardless of distance to equilibrium. Such reference models make predictions of generic nonequilibrium properties such as the mean and modal microstate distributions and the expected free energy of a random distribution, which we examine analytically and via exact sampling algorithms. Moreover, introducing constraints on Shannon entropy permits parametric access to tunably nonequilibrium regimes. The approach follows conventional Jaynesian modeling logic and is compatible with standard Bayesian inference, and reduces to Boltzmann-Gibbs statistics when Shannon entropy is maximized. We examine the physical predictions of this scheme, and demonstrate its wide compatibility with other approaches to nonequilibrium modeling. We illustrate the framework in three scenarios: (i) comparison to superstatistics quasi-equilibrium modeling, (ii) generation of random nonequilibrium steady states, and (iii) Bayesian inference under a prior hyperensemble. We highlight numerous prospects for theoretical extensions and empirical application.

Comment
23 pages, 5 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Physics of Information Geometry - Part II: Small-Step Active Inference on the Probability Simplex

cs.IT cond-mat.stat-mech ·2026-09-10 ·C. Emre Koksal, Deniz Sargun
theory similarity 0.64
Abstract

This paper is the second in a two-part investigation of the physics of information geometry. While Part I develops a physical foundation for distributional motion on the probability simplex, the present paper studies how that framework manifests in active inference. The treatment is fully self-contained and does not require familiarity with Part I. We focus in particular on active inference through small distributional steps and the geometric structure induced by such local motion. Starting from an initial distribution, an agent evolves its belief state toward a final target distribution through a sequence of constrained updates. We define a relative free energy functional with respect to the preferred distribution and extend it to a relative potential energy analogous to the Helmholtz/Gibbs free-energy decomposition. The evolution is subject to a per-step kinetic constraint expressed through the Kullback-Leibler (KL) divergence between consecutive distributions, which serves as a discrete kinetic energy on the probability simplex. Using the information-geometric Pythagorean theorem on KL balls, we show that sufficiently small local moves dominate large direct jumps, and that greedy maximization of free-energy reduction is globally optimal under the kinetic constraint. This leads to a sequential variational principle in which the optimal trajectory minimizes the associated Lagrangian of the optimization problem. Similar to classical mechanics, the Lagrangian takes on the form as the difference between the kinetic and potential terms, establishing a least-action principle for distributional motion on the simplex. The resulting optimal update admits a closed form as an exponentially tilted version of the current distribution toward the preferred distribution, parametrized by an inverse-temperature-like multiplier. We further extend the framework to incorporate state-dependent geodesic...

0.17

Physics of Information Geometry - Part I: Principle of Least Action on the Probability Simplex

cs.IT cond-mat.stat-mech ·2026-09-08 ·C. Emre Koksal, Deniz Sargun
similarity 0.62
Abstract

We develop a least-action framework for describing how a probability distribution can evolve from an equilibrium state to a prescribed nonequilibrium state under constrained incremental changes. Taking a Gibbs distribution as the equilibrium reference, the framework gives a direct physical meaning to the geometry of the probability simplex: distance from equilibrium corresponds to nonequilibrium free energy, while changes between successive distributions carry an informational kinetic cost. The Pythagorean structure of relative entropy then provides the central insight of the work. It shows that intermediate distributions chosen via sequential information projections can reduce the kinetic cost of large transitions and establishes an energy-conservation-like relation between the kinetic expenditure along a path and the free energy accumulated in reaching the target distribution. Motivated by this geometry, we construct a greedy least-action path through successive information projections, obtain a closed-form characterization of each projection through the Lambert W function, and establish a finite-step performance guarantee. We further show that state-dependent costs can be incorporated naturally by reshaping the underlying Gibbs reference, providing a thermodynamic interpretation of path penalties as modifications of the effective energy landscape. Together, these results provide a unified view of distributional evolution through least action, information geometry, and nonequilibrium thermodynamics.

0.41

A Structural Characterization of Entropy Functionals

cs.IT math.ST ·2026-08-14 ·Daniel Lazarev
similarity 0.62
Abstract · 31 pages, 1 figure

Entropy functionals and their associated divergences underlie many statistical methods, including maximum entropy inference, minimum divergence estimation, and goodness-of-fit testing, yet choosing among Shannon, Rényi, Tsallis, and more general entropies is often a matter of convention rather than structural principle. We introduce a measure theoretic framework in which admissibility requires the entropy of an input measure to be bounded above by that of its reference measure whenever the former is absolutely continuous with respect to the latter. Under generalized mean-value composition, we characterize all such entropy functionals and obtain a four-level hierarchy determined successively by the mean generator, entropy scale, and additivity assumptions. A continuous strictly monotone generator $g$ is admissible exactly when $t\mapsto g(1/t)$ is strictly convex for increasing $g$, or strictly concave for decreasing $g$. This resolves a question posed by Rényi (Proc. 4th Berkeley Sympos. Math. Statist. Prob., 1961) concerning which generalized means may replace the arithmetic mean in his entropy axiomatization. The same criterion is equivalent to strict convexity of an associated Csiszár $f$-divergence generator and therefore yields data processing under Markov kernels with an exact equality condition. Within this hierarchy, product additivity singles out the Rényi family, while internal additivity, or product additivity together with arithmetic mean-value composition, singles out Shannon entropy. The characterization is constructive and yields new admissible entropy and divergence families, including integral-transform examples.

0.10

Microcanonical Hamiltonian Monte Carlo and the Helmholtz Theorem

cs.AI cond-mat.stat-mech ·2026-09-07 ·Heinrich von Campe, Bjoern Malte Schaefer
publishedpositiontheory similarity 0.61
Abstract

The recently proposed Microcanonical Hamiltonian Monte Carlo algorithm has not yet been studied in detail from a thermodynamic point of view; this work aims to fill that gap. We demonstrate how thermodynamical state variables and potentials can be derived and thereby demonstrate that the construction of the algorithm formally represents a microcanonical thermodynamic ensemble. In particular, we demonstrate (analytically and numerically) that the algorithm fulfils the Helmholtz theorem, an alternative formulation of the first law of thermodynamics. Furthermore, we construct a new sampling algorithm that extends the original to lower-dimensional inference problems. Finally, we argue that canonical Markov Chain Monte Carlo algorithms are more natural than Microcanonical Hamiltonian Monte Carlo from the thermodynamic and information-theoretic point of view.

0.49

Revisiting Shannon's Source Coding Theorem with Distributional Uncertainty under the Nonlinear Expectation Theory

cs.IT ·2026-08-17 ·Wen-Xuan Lang, Shaoshi Yang, Jianhua Zhang, Zhiming Ma
theory similarity 0.58
Abstract · 6 pages, 2 figures

In classical information theory, a source is modeled by a single, precisely known probability distribution. However, in the increasingly complex communication networks full of unanticipated, nonstationary, and heterogeneous random events, the assumption of precise and well-defined probability distributions to describe random variables appears somewhat idealized. Therefore, it is important to characterize the uncertainty of distributions of source messages, subject to relaxing the assumption of deterministic probability models for analyzing information sources in information theory. Based on the nonlinear expectation theory, a novel axiomatical system that extends classical probability theory, this paper investigates the information sources whose distributions themselves are uncertain, and refers to them as uncertain-distribution sources. We generalize the fundamental concept information entropy to nonlinear information entropy, which describes the measurement of the amount of information contained in a uncertain-distribution source. By using the strong law of large numbers under sublinear expectation, we establish a nonlinear source coding theorem, which not only shows that the nonlinear information entropy is the upper bound for the infimum of achievable coding rate of uncertain-distribution sources under the maximum error probability criterion, but also determines a cluster point of the coding rate of uncertain-distribution sources under the minimum error probability criterion. Our findings reveal that the introduction of nonlinear expectation theory allows for a more comprehensive understanding of information sources.

0.22

On Minimum Divergence for Distribution Reconstruction

cs.IT math.ST ·2026-10-15 ·Justin Bec, Thomas Le Corre, Aurélien Garivier
similarity 0.56
Abstract

We study the possibility of generalizing maximum-entropy reconstruction, which approximately recovers a probability distribution from finitely many (generalized) moments by minimizing the Kullback--Leibler (KL) divergence to a reference measure. The reconstruction lies in an exponential family and is also the best KL approximation of the unknown distribution within that family: moment matching and information projection coincide. Replacing KL by a smooth f-divergence, the reconstruction still exists in an associated family, but the best approximation property does not. Specifically, a family is called entropy-adapted if the projection of any distribution matches its moments. We show that, up to equivalence, KL is the only $C^2$ f-divergence that is entropy-adapted, whatever the choice of the generalized moments. We illustrate the consequences of this failure numerically.

0.22

Expected free energy as an information constraint on the Bethe Lagrangian

cs.IT cs.AIeess.SYstat.ML ·2026-08-17 ·Wouter M. Kouw
similarity 0.56
Abstract · 17 pages, 4 figures, table 2. International Workshop on Active Inference

Active inference selects actions by minimising an expected free energy functional over predicted futures. However, adding an expectation over yet-unobserved outcomes means the free energy functional no longer has a Kullback-Leibler structure, which hinders message passing treatments of inference procedures. We propose an alternative formulation based on a Bethe free energy functional, fully supporting inference by message passing. The epistemic drive is maintained by imposing an information constraint, next to normalisation, marginalisation and form constraints, insisting that the mutual information between future observations, states and parameters given actions must be at least as large as the entropy of the goal prior. For a specific value of the corresponding Karush-Kuhn-Tucker multiplier, the stationary point of this constrained Bethe Lagrangian recovers the expected free energy solution. We show that, as the information demand is varied, the solved multiplier moves through its inactive, interior, and saturated regimes. In the inactive regime the agent's epistemic drive switches off entirely, while in the saturated regime it is maximal. We compare the performance of the constrained Bethe agent on three tasks against EFE and Q-MDP.

0.37

Free Lunch Uncertainty: Sequence-Level Rényi Entropy from Particle Decoding for Adaptive LLM Reasoning

cs.LG cs.AIcs.CL ·2026-11-10 ·Erfan Baghaei Potraghloo, Milad Heidari Sorkhehdizaj, Shahab Rahimirad, Aliasghar Pourghani, Seyedarmin Azizi +4 more
theory similarity 0.56
Abstract

Quantifying uncertainty over an entire LLM completion is critical for selective prediction and adaptive compute. Zero-cost signals such as mean token entropy and log-probability are local to tokens or to one completion, and self-consistency needs many extra completions. We show that Sequential Monte Carlo (SMC) decoding for power sampling already estimates the normalizing constant of the power distribution, which gives the sequence-level R'enyi entropy (SRE) of the model's completion distribution at zero marginal cost, with a guarantee at every particle budget. A R'enyi chain rule extends this readout to a family of stepwise entropies of any order. We prove that orders above one are tied to the most likely token of each step, which explains why they track log-probability, while the order-1/2 member accumulates the log of per-step search effort. This member is the best zero-cost selective-prediction signal in 10 of 13 settings. On MATH500 its AUROC is 0.022 to 0.044 above log-probability with all four models and within length quartiles, and the same order is selected when each setting is held out. An 8-particle probe identifies the problems that benefit from more particles, and a two-pass policy matches 64-particle accuracy on MATH500 with 34.2% fewer particles over three seeds.

0.23