CS Paper Radar

Beyond Nominal Equilibria: Risk-Averse Multi-Population Mean-Field Games

Bhavini Jeloka, Siddhartha Ganguly, Panagiotis Tsiotras

cs.GT math.OCcs.LGeess.SY ·submitted 2026-10-07 00:15 UTC · v1

Abstract

Recent advances in mean-field games and its multi-population variants enable large-scale heterogeneous multi-agent systems to be modeled through representative agents and their associated mean-field distributions. However, existing approaches do not explicitly account for uncertainty in the behavior of other populations. To this end, we introduce a new paradigm: risk-averse multi-population mean-field games, where each population optimizes a worst-case expected reward over dynamically feasible ambiguity sets of mean-field flows of a subset of the other populations. Employing an occupation-measure formulation along with tools from set-valued analysis, we establish, under mild assumptions, several theoretical properties of the multi-population game, including the geometric properties of the ambiguity sets and the existence of a novel risk-averse multi-population mean-field equilibrium. Further, we derive contractivity results of the fixed-point operator under entropy regularization and show that it can be utilized to learn the equilibrium. Finally, we propose a risk-averse fictitious-play scheme and show that exploitability decays to zero, despite the additional nonlinearity introduced by the worst-case objective. We report several numerical experiments to illustrate convergence and risk-averse behavior.

Comment
Submitted to a conference; comments are welcome

Closest papers

Nearest neighbours by meaning across the whole index.

Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations

cs.MA cs.LGstat.ML ·2026-08-26 ·Aditya Makkar, Benjamin Unger, Jeongyeol Kwon, Mathieu Laurière, Eugene Vinitsky +1 more
similarity 0.66
Abstract · 33 pages, 3 figures

Modern multi-agent systems are increasingly deployed at scale over large populations of agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each agent's policy independently, treating the other agents as part of a fixed single-agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.

0.19

Infinite Horizon Mean-Field Terminal Value Problem

eess.SY math.OC ·2026-09-11 ·Uğur Aydın, Tamer Başar, Naci Saldi
similarity 0.64
Abstract · 39 pages, 2 figures

We introduce a class of finite- and infinite-horizon discrete-time control problems for studying macroscale systems in discrete time, which we refer to as mean-field terminal value problems (MFTVPs). An MFTVP takes a terminal \(Q\)-function as input and, starting from this terminal \(Q\)-function, seeks an indefinite backward evolution of optimal policies and state measures, as required for an infinite-horizon non-stationary mean-field equilibrium (MFE) of a discrete-time mean-field game (MFG). We study the existence and uniqueness properties of this problem. As an application, using the uniqueness properties of MFTVPs, we show that, in the average-cost setting (and in the discounted setting for sufficiently small discount factors), MFTVPs can be used as an intermediate step to develop approximation schemes between finite-horizon MFEs and average-cost MFEs (respectively, stationary MFEs). In contrast to approaches that use infinite-horizon MFGs as intermediate problems, MFTVPs can be employed when the discount factor is sufficiently small (or, in the average-cost setting, when the drift factor is sufficiently small), together with a regularizer, to approximate a stationary MFE. Assuming uniqueness of the solution of a MFTVP, the restrictions we impose on the Lipschitz parameters of the system are weaker than the contraction conditions available for regularized stationary MFGs, and our approximation scheme remains applicable even in the presence of multiple stationary MFEs.

0.31

Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning

cs.GT math.OCcs.LG ·2026-10-11 ·Gunes Akbas, Naci Saldi, Berkay Anahtarci, Can Deha Kariksiz
theory similarity 0.63
Abstract · 37 pages, 1 figure, 2 tables

We study stationary discrete-time mean-field games with finite state and action spaces under the average-reward criterion, with an entropy-regularized objective and population-dependent dynamics and rewards. Under a minorization condition, the soft Q-operator is a span-seminorm contraction for the full transition kernel, so relative value iteration does not require the minorizing measure. When the Lipschitz constant of the kernel is below the minorization mass and the temperature exceeds an explicit threshold, the equilibrium fixed-point operator is a contraction: an equilibrium exists, its population distribution is unique, and a fixed-point iteration converges to it geometrically. We then learn the equilibrium model-free from a simulator, combining soft Q-learning at a fixed population distribution with an empirical transition kernel. We prove a finite-sample bound for soft Q-learning in the span seminorm, with temperature-independent constants, and derive convergence rates for the learned population distribution and policy. A congestion-aware routing game illustrates the results.

0.26

Horizon-Independent Contraction for Continuous-Time Discounted Regularized Mean-Field Games

cs.GT eess.SYmath.OC ·2026-08-27 ·Junji Yan, Uğur Aydın, Tamer Başar
similarity 0.62
Abstract · 34 pages. Full version including complete proofs

We study contraction properties of non-stationary continuous-time mean-field games (MFGs) under discounting and entropy regularization. The state of the representative agent evolves according to a controlled continuous-time Markov chain, and both the state and action spaces are finite. In contrast to the undiscounted case, we show that, under a sufficiently large discount rate, finite-horizon MFGs admit a horizon-independent contraction condition, which also coincides with the corresponding infinite-horizon non-stationary contraction condition. As a byproduct, we obtain an explicit convergence rate between finite- and infinite-horizon mean-field equilibria. For each finite horizon, we further derive a refined contraction criterion from the spectral radius of a positive operator that majorizes the propagation of policy errors, and show that its large-horizon limit agrees with the horizon-independent contraction factor. Finally, we provide an explicit error bound between discounted and undiscounted finite-horizon regularized equilibria.

0.32

Equilibrium Statistics and Stability in Random Two-Population and Three-Population Evolutionary Games

cs.GT ·2026-11-17 ·Zhao Song, Manh Hong Duong, The Anh Han
similarity 0.61
Abstract

Multi-population matrix evolutionary games provide a framework for studying adaptation and strategic interactions among heterogeneous agents. When payoffs are uncertain, however, analysing a specified game does not reveal how frequently different evolutionary outcomes occur or how their probabilities depend on modelling assumptions. We study two- and three-population games with two strategies per population and both within- and between-population pairwise interactions, with payoffs fixed during each evolutionary trajectory. For two populations, we derive equilibrium-existence probabilities and analytical stability results for generic games and three structured classes: diversity, exclusive, and avoidance games. For generic two-population games, independent Gaussian sampling of payoff entries and of replicator coefficients produces different equilibrium statistics. Interaction structure further distinguishes coexistence from attraction: diversity and exclusive games share the same interior-existence probability, but only exclusive games admit asymptotically stable interior equilibria. Avoidance games admit transversely attracting equilibrium families without locally asymptotically stable individual points. For generic three-population games under independent payoff sampling, we obtain existence probabilities for all equilibrium types and joint existence-and-stability probabilities for boundary equilibria. Monte Carlo simulations support the analytical predictions and quantify multistability, while phase portraits illustrate initial-condition dependence. Under the same Gaussian payoff-sampling protocol, simulations show a lower frequency of fully mixed equilibria but a higher frequency of multistability for three populations than for two.

0.37

Symmetry-breaking in a discrete-choice LQG mean field game

eess.SY math.DSmath.APmath.OC+1 ·2026-10-05 ·Ali Akbar Rezaei Lori, Piyush Grover
similarity 0.61
Abstract · 26 pages, 4 figures. Originally submitted for review 5/18/26. v2 fixes issue with proof of lemma 4.3 related to terminal differentiability o…

Mean-field games provide a continuum framework for modeling the dynamics of large, interacting populations of non-cooperative agents. This paper studies symmetry-breaking in a finite-horizon, two-choice min-LQG mean-field game in which identical agents with linear stochastic dynamics choose one of two equally desirable terminal destinations, while trading off control effort against social pressure to conform. The model has an odd symmetry between the two destinations and therefore always admits a symmetric, dynamic Nash equilibrium in which the population splits evenly between them, producing a deadlock collective state at final time. Numerical studies have suggested that, as the penalty for social nonconformity increases, this symmetric equilibrium loses stability as a fixed point of an associated scalar, self-consistency map, and asymmetric consensus Nash equilibria emerge, where most agents select the same destination. By analyzing the linearized forward-backward PDE system through the scalar map representation, we provide a proof of this loss of stability of the symmetric equilibrium. Together with the odd symmetry of the map, this implies the existence of symmetry-broken mean-field game equilibria corresponding to consensus on either destination.

0.35

Stochastic Games with Distributional Entropy Objectives

cs.GT cs.FL ·2026-11-13 ·Shankaranarayanan Krishna, Durgam Latha, Dion Reji, Ashutosh Trivedi
similarity 0.60
Abstract

We introduce \emph{stochastic games with distributional entropy objectives}, two-player zero-sum games whose configurations are probability distributions over the states of a finite arena and whose stage reward is Shannon entropy. Over an infinite discounted horizon, MIN seeks to concentrate the distribution, while MAX seeks to disperse it. Because entropy is nonlinear, it matters whether strategic and transition randomness are averaged before entropy is evaluated or sampled first. We formalise three resulting semantics: averaged-distribution, sampled-profile, and sampled-transition. This semantic choice fundamentally changes the game-theoretic structure. Averaged-distribution games need not be determined, even on a four-state deterministic arena, whereas both sampled semantics are determined and admit optimal stationary randomised strategies. We give contractive Bellman characterisations and convergent finite-horizon approximations for the upper and lower averaged values and the two sampled values. Moreover, even the lower value under averaged-distribution semantics dominates the value under either sampled semantics, while the sampled-profile and sampled-transition values are incomparable in general. For fixed rational discount and positive rational promise gap, the averaged-distribution (upper and lower value) gap problems and both sampled gap problems lie in PSPACE; for a single entropy minimiser, the averaged and sampled-profile problems lie in NP. Finally, under all three semantics, entropy optimisation is NP-hard to approximate within a suitable fixed additive constant, already for deterministic one-player games, where pure memoryless strategies are optimal.

0.21

Optimization Frameworks & Learning Algorithms for Discrete-Time Mean Field Games

cs.LG math.OC ·2026-11-01 ·Xin Guo, Mathieu Laurière
similarity 0.60
Abstract

This monograph introduces discrete-time mean field games with finite state and action spaces, emphasizing optimization formulations and learning algorithms. Starting from Markov decision processes and finite-player stochastic games, it develops the notion of mean field Nash equilibrium, establishes existence and uniqueness results under suitable assumptions, and explains how mean field equilibria yield approximate equilibria in large finite-player games. Building on the linear programming formulation of Markov decision processes, occupation measures and primal-dual formulations connect equilibrium computation with mathematical optimization. The monograph presents fixed-point iterations, fictitious play, online mirror descent, and optimization-based methods, discussing their convergence properties and limitations. Numerical examples illustrate crowd motion, traffic routing, epidemic management, and the computation of multiple equilibria. It combines examples, theoretical results, and implementable algorithms, with bibliographic notes providing connections to the broader literature.

0.22