CS Paper Radar

Horizon-Independent Contraction for Continuous-Time Discounted Regularized Mean-Field Games

Junji Yan, Uğur Aydın, Tamer Başar

cs.GT eess.SYmath.OC ·submitted 2026-08-27 21:29 UTC · v1

Abstract

We study contraction properties of non-stationary continuous-time mean-field games (MFGs) under discounting and entropy regularization. The state of the representative agent evolves according to a controlled continuous-time Markov chain, and both the state and action spaces are finite. In contrast to the undiscounted case, we show that, under a sufficiently large discount rate, finite-horizon MFGs admit a horizon-independent contraction condition, which also coincides with the corresponding infinite-horizon non-stationary contraction condition. As a byproduct, we obtain an explicit convergence rate between finite- and infinite-horizon mean-field equilibria. For each finite horizon, we further derive a refined contraction criterion from the spectral radius of a positive operator that majorizes the propagation of policy errors, and show that its large-horizon limit agrees with the horizon-independent contraction factor. Finally, we provide an explicit error bound between discounted and undiscounted finite-horizon regularized equilibria.

Comment
34 pages. Full version including complete proofs

Closest papers

Nearest neighbours by meaning across the whole index.

Infinite Horizon Mean-Field Terminal Value Problem

eess.SY math.OC ·2026-09-11 ·Uğur Aydın, Tamer Başar, Naci Saldi
similarity 0.78
Abstract · 39 pages, 2 figures

We introduce a class of finite- and infinite-horizon discrete-time control problems for studying macroscale systems in discrete time, which we refer to as mean-field terminal value problems (MFTVPs). An MFTVP takes a terminal \(Q\)-function as input and, starting from this terminal \(Q\)-function, seeks an indefinite backward evolution of optimal policies and state measures, as required for an infinite-horizon non-stationary mean-field equilibrium (MFE) of a discrete-time mean-field game (MFG). We study the existence and uniqueness properties of this problem. As an application, using the uniqueness properties of MFTVPs, we show that, in the average-cost setting (and in the discounted setting for sufficiently small discount factors), MFTVPs can be used as an intermediate step to develop approximation schemes between finite-horizon MFEs and average-cost MFEs (respectively, stationary MFEs). In contrast to approaches that use infinite-horizon MFGs as intermediate problems, MFTVPs can be employed when the discount factor is sufficiently small (or, in the average-cost setting, when the drift factor is sufficiently small), together with a regularizer, to approximate a stationary MFE. Assuming uniqueness of the solution of a MFTVP, the restrictions we impose on the Lipschitz parameters of the system are weaker than the contraction conditions available for regularized stationary MFGs, and our approximation scheme remains applicable even in the presence of multiple stationary MFEs.

0.31

Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning

cs.GT math.OCcs.LG ·2026-10-11 ·Gunes Akbas, Naci Saldi, Berkay Anahtarci, Can Deha Kariksiz
theory similarity 0.67
Abstract · 37 pages, 1 figure, 2 tables

We study stationary discrete-time mean-field games with finite state and action spaces under the average-reward criterion, with an entropy-regularized objective and population-dependent dynamics and rewards. Under a minorization condition, the soft Q-operator is a span-seminorm contraction for the full transition kernel, so relative value iteration does not require the minorizing measure. When the Lipschitz constant of the kernel is below the minorization mass and the temperature exceeds an explicit threshold, the equilibrium fixed-point operator is a contraction: an equilibrium exists, its population distribution is unique, and a fixed-point iteration converges to it geometrically. We then learn the equilibrium model-free from a simulator, combining soft Q-learning at a fixed population distribution with an empirical transition kernel. We prove a finite-sample bound for soft Q-learning in the span seminorm, with temperature-independent constants, and derive convergence rates for the learned population distribution and policy. A congestion-aware routing game illustrates the results.

0.26

Normal-Form Correlation in Markov Games

cs.GT cs.CCcs.DScs.LG ·2026-10-02 ·Ioannis Anagnostides, Constantinos Daskalakis, Gabriele Farina, Noah Golowich, Tuomas Sandholm +1 more
similarity 0.65
Abstract

There has been a surge of recent work on correlated equilibrium concepts in Markov games. However, existing results focus on concepts weaker than normal-form correlated equilibria (NFCEs), leaving open the more challenging question of computing such equilibria, which goes back to the seminal work of Papadimitriou and Roughgarden (JACM'08). Here, we establish the first efficient algorithm for NFCEs in finite-horizon Markov games with a fixed number of players $n$. In particular, with $S$ states, horizon $H$, and at most $A$ actions per player, it computes an $ε$-NFCE in time $S(AH/ε)^{O(n)}$. This is the first algorithm polynomial in $1/ε$ and the description of the game for NFCEs in an interesting class of problems beyond the normal-form setting. Moreover, under the usual assumption that recommendations are independent across states, we show PPAD-completeness---that is, computational equivalence to Nash equilibria---either in many-player games or when the precision is exponentially small. The key idea behind our approach is to run backward induction on a sequence of auxiliary stage games, but with the twist that in each step we compute a constant-expectation correlated equilibrium. This is a natural refinement of correlated equilibrium in which the conditional expected payoff from obeying is independent of the recommendation. In fact, our reduction goes both ways, establishing an equivalence between constant-expectation CEs and NFCEs in Markov games. For a fixed number of players, we observe that a constant-expectation CE can be computed approximately by combining linear programming with suitable discretization. In contrast, it is PPAD-hard in i) polymatrix (many-player) games at constant precision, and ii) two-player games at exponentially small precision. The latter result follows from an unexpected connection to rank-2 two-player games.

0.43

Reinforcement learning for constrained stochastic linear--quadratic control in continuous time

cs.LG math.OC ·2026-10-26 ·Weiping Wu, Qitong Pan, Xun Li, Peng Wu
similarity 0.64
Abstract

We study continuous-time stochastic linear--quadratic control with a scalar state and nonnegative control constraints. With quadratic running costs, randomized exploration can change the optimal mean feedback. We introduce an auxiliary problem without entropy in which the conditional policy mean is nonnegative and the covariance is prescribed in proportion to the squared state. Moment reduction and state separation yield two regional Riccati equations and characterize the finite-horizon value function and optimal conditional mean. We give a sufficient control--cost compatibility condition under which this mean coincides with the optimal feedback of the original constrained problem. For time-invariant coefficients, we characterize the undiscounted infinite-horizon optimum and establish mean square asymptotic stability under the stated assumptions. For unknown dynamics, we develop model-free actor--critic schemes for finite and infinite horizons. Numerical experiments compare the learned policies with their analytical targets and examine the effect of exploration on the optimal mean.

0.25

Decentralized Bandit Learning of Markov Perfect Equilibria

cs.LG math.OC ·2026-10-09 ·Felix Höfer, H. Mete Soner
theoryhot direction similarity 0.63
Abstract · 23 pages, 1 figure

We study decentralized bandit learning in discounted infinite-horizon, general-sum Markov games with continuous actions. Agents observe the public state, but only their own realized rewards, and use simultaneous perturbation stochastic approximation to update their actions. Under strong monotonicity of the negative running-reward gradients and sufficiently small time steps, we prove that the fully decentralized procedure converges almost surely to the unique stationary Markov perfect equilibrium. We obtain an $O(n^{-1/2})$ convergence rate and present a natural extension to episodic finite-horizon learning.

0.25

Receding Horizon Control and Dissipativity - Optimal Control, Games and Uncertainty

eess.SY math.OC ·2026-09-29 ·Sophie Hall, Jonas Schießl, Sergio Grammatico, Timm Faulwasser
publishedsurvey similarity 0.62
Abstract · To appear at IEEE CDC 2026

We provide a tutorial overview of receding horizon control across deterministic, stochastic, multi-agent, and game-theoretic settings. We contrast the minimizer of an optimal control problem with the equilibrium of a game in terms of cost and constraint handling to motivate the difference of MPC schemes involving multiple agents/players. Interestingly, dissipativity theory and the turnpike property have turned out to be a unifying thread and system-theoretic backbone across all these schemes. This paper is the first to give an overview of results, summarizing fundamental insights, drawing parallels, and highlighting technical differences in dissipativity and turnpike analysis across economic, game theoretic, and stochastic settings. We also explore recent trends in numerical methods for games. We conclude by pointing to a list of open challenges.

0.32

Beyond Nominal Equilibria: Risk-Averse Multi-Population Mean-Field Games

cs.GT math.OCcs.LGeess.SY ·2026-10-07 ·Bhavini Jeloka, Siddhartha Ganguly, Panagiotis Tsiotras
similarity 0.62
Abstract · Submitted to a conference; comments are welcome

Recent advances in mean-field games and its multi-population variants enable large-scale heterogeneous multi-agent systems to be modeled through representative agents and their associated mean-field distributions. However, existing approaches do not explicitly account for uncertainty in the behavior of other populations. To this end, we introduce a new paradigm: risk-averse multi-population mean-field games, where each population optimizes a worst-case expected reward over dynamically feasible ambiguity sets of mean-field flows of a subset of the other populations. Employing an occupation-measure formulation along with tools from set-valued analysis, we establish, under mild assumptions, several theoretical properties of the multi-population game, including the geometric properties of the ambiguity sets and the existence of a novel risk-averse multi-population mean-field equilibrium. Further, we derive contractivity results of the fixed-point operator under entropy regularization and show that it can be utilized to learn the equilibrium. Finally, we propose a risk-averse fictitious-play scheme and show that exploitability decays to zero, despite the additional nonlinearity introduced by the worst-case objective. We report several numerical experiments to illustrate convergence and risk-averse behavior.

0.28

A Horizon-Independent Regret Bound for Optimistic Hedge in General-Sum Games

cs.GT cs.LG ·2026-09-19 ·Junsoo Ha
theory similarity 0.61
Abstract

Can simple learning rules keep their regret bounded in self-play? Recent work achieves constant regret bounds through modified regularization and higher-order prediction. Yet for Optimistic Hedge, arguably the most canonical method in games, the best known individual regret bound remains logarithmic. In this work, we prove that plain Optimistic Hedge with a constant step size can attain $O_{n,d}(1)$ individual regret in general-sum games with $n$ players and $d=(d_1,\ldots,d_n)$ actions, under expected loss-vector feedback. As a corollary, its time-averaged play enjoys an $O_{n,d}(1/T)$ coarse correlated equilibrium (CCE) gap. Our analysis represents Optimistic Hedge as a real-analytic recurrence on a compact space, which yields an exact finite-order difference relation that eliminates horizon dependence. Our proof hinges on nonconstructive Noetherianity argument of Frisch (1967), so the $(n,d)$-dependence remains implicit.

0.34