CS Paper Radar

Stabilization Limits of Payoff-Based Higher-Order Replicator Dynamics

Hassan Abdelraouf, Vijay Gupta, Jeff S. Shamma

eess.SY ·submitted 2026-08-15 16:25 UTC · v1

Abstract

Replicator dynamics (RD) is a fundamental model in learning in games, connecting evolutionary game theory and online learning. This paper studies payoff-based higher-order variants of RD represented as a cascade interconnection between an integrator in parallel with an auxiliary linear time-invariant (LTI) system and the softmax mapping. We investigate learnability of Nash equilibria under this Nash-stationary learning rule. First, we revisit recent results that establish convergence to Nash Equilibrium whenever the auxiliary LTI system is strictly passive and prove a converse passivity result: if the auxiliary LTI system is not passive, then there exists a static strictly contractive game whose interior Nash equilibrium is unstable under the closed-loop learning dynamics. Second, we show that there exists a class of games with isolated interior Nash equilibria that cannot be locally asymptotically stabilized by any payoff-based higher-order RD whose auxiliary LTI system is asymptotically stable and strictly proper. Finally, we show that if Nash stationarity (i.e., all Nash equilibria are stationary points of the learning dynamics) is relaxed, then generalized exponential RD (Ex-RD) can locally asymptotically stabilize a logit equilibrium for any continuously differentiable game. The stabilized equilibrium can be viewed as an entropy-regularized approximate Nash equilibrium.

Closest papers

Nearest neighbours by meaning across the whole index.

Learning in Large Populations With Anticipatory Payoffs: Bregman Meets Willems and Lyapunov

cs.GT math.OCeess.SY ·2026-10-02 ·Tzuyu Jeng, Nuno C. Martins
similarity 0.71
Abstract

We study evolutionary Nash equilibrium learning in large populations whose payoffs are generated by dynamic mechanisms. Counterclockwise dissipativity (CCW) provides a passivity-like framework for establishing learning/convergence under a broad convex cone of so-called well-behaved learning rules satisfying a natural payoff-alignment condition according to which, away from equilibria, the population's state (strategic profile) moves in a direction positively correlated with the payoffs. The cone is closed under conic combinations and encompasses a broad range of noise-free rules, including discontinuous rules such as best response, imitative rules such as the replicator rule that is not $δ$-passive, and all noise-free $δ$-passive rules. Existing CCW results require no knowledge of the particular rule, but establish convergence without providing a Lyapunov function or Lyapunov stability. We develop a methodology that closes this gap by constructing rule-independent Lyapunov functions from CCW storage functions of the payoff mechanism. For a nonlinear class of anticipatory mechanisms, in which agents compare current payoffs with those associated with a smoothed population state, we show that a storage can be derived as a Bregman divergence. The resulting Lyapunov function combines a potential gap with this divergence and establishes global asymptotic stability. A congestion game with a nonlinear congestion-growth toll illustrates the results.

0.09

Planning Against Learning in Rank-1 Games

cs.GT ·2026-08-18 ·William Overman
similarity 0.65
Abstract

Learning algorithms are often used to make decisions in repeated multi-agent environments. When another player understands how a learner adapts from past experience, that player can plan strategically across rounds to influence the learner's future behavior. Recent work shows that optimizing against Replicator Dynamics, the continuous-time analogue of Multiplicative Weights Update, is tractable in zero-sum games but can be hard in unrestricted general-sum games. We study the first structured class beyond zero sum: bimatrix games satisfying $\text{rank}(A+B)=1$, for which Nash equilibria can be computed in polynomial time. Our main result shows that this equilibrium tractability does not extend to planning against learning dynamics. Unless $\mathsf{P}=\mathsf{NP}$, approximating the optimizer's optimal continuous-time reward within a fixed additive constant is NP-hard even when $\text{rank}(A+B)=1$, the learner starts from the uniform state, and the optimizer is restricted to constant strategies. The hardness persists for bounded payoff matrices and polynomially bounded horizons. We complement this result with structural characterizations of several tractable special cases. Thus rank-one games already separate efficient equilibrium computation from strategic planning against a learning opponent.

0.06

On the Convergence of First-Order Methods in Signaling Games

cs.GT ·2026-11-07 ·Rilind Sahitaj, Artem Tsikiridis, Martin Bichler
top venue similarity 0.63
Abstract · Accepted at NeurIPS 2026

We study first-order learning dynamics in Bayesian signaling games, a minimal extensive-form model of strategic interaction under asymmetric information. In these games, a Sender observes a private type and chooses a signal, after which a Receiver forms a posterior belief and selects an action. This Bayesian belief update makes the induced learning field fundamentally different from the affine game-gradient fields of normal-form games. Empirically, we find a sharp dichotomy: standard online learning algorithms reliably converge to strict Perfect Bayesian Equilibria (PE), but not to non-strict ones. We show that this behavior cannot be explained by the global geometric conditions commonly used to prove last-iterate convergence, such as monotonicity or the Minty variational inequality. Indeed, these conditions can fail even in binary signaling games. Instead, we identify the relevant local geometry. Our main result characterizes variational stability in finite signaling games: a PE is variationally stable if and only if it is strict. This yields local last-iterate convergence guarantees for online mirror descent and regularized dual averaging near every strict PE. We then study global average-iterate convergence and identify a class of signaling games in which coarse-trigger regret-minimizing dynamics converge to the unique PE. Finally, we complement the theory with experiments comparing projected gradient ascent, mirror-based methods, optimistic variants, and PPO across canonical signaling-game families. The results position signaling games as a compact testbed for understanding how Bayesian belief formation reshapes the convergence geometry of multi-agent learning.

0.35

Fully Online Decentralized Learning in Stochastic Games with Unknown Independent Chains

cs.LG cs.GTcs.MAeess.SY+1 ·2026-10-01 ·S. Rasoul Etesami
similarity 0.61
Abstract

We consider stochastic games with independent controlled chains and unknown transition kernels, where players observe only their local states and realized payoffs. We develop a fully online, decentralized, and uncoordinated mirror-descent algorithm that operates in the dual space of occupancy measures for approximating stationary Nash equilibrium (NE) policies. The algorithm uses a single transition/reward sample at every primitive time step, relies only on local information, and requires neither coverage of the joint state space nor synchronized episodes. Under uniform-ergodicity and finite-coverage assumptions, we show that, with high probability, the time-averaged fixed-comparator regret decays at the canonical $O(T^{-1/2})$ rate, up to logarithmic factors and polynomial dependence on the game parameters. In particular, the complexity depends on the cover times of the individual local state spaces rather than the product state space, avoiding exponential dependence on the number of players and the sizes of the joint state and action spaces. The resulting finite-time regret bound further yields an approximate coarse-correlated-equilibrium guarantee, which is natural for arbitrary reward functions since computing a stationary $ε$-NE is PPAD-hard in this setting. Under an additional global variational-stability condition, we show that the same fully online algorithm converges asymptotically in the last iterate to a stationary $ε$-NE. Our results provide a fully online and scalable learning framework for stochastic games with unknown independent chains. The algorithm can also be viewed as a primal-dual framework for Markov games that exploits the independence and local structure of the players' controlled transition chains.

0.11

Stable Coexistence in Ecologies and Games

cs.GT math.DSq-bio.PE ·2026-09-08 ·Türkü Özlüm Çelik, Vincenzo Antonio Isoldi, Irem Portakal, Giulio Zucal
similarity 0.61
Abstract · Comments are welcome!

We study feasible stable equilibria of Lotka-Volterra systems and their higher-order extensions. We complete the classification of impossible ecological interaction networks with at most four species and extend several of these impossibility results to families with arbitrarily many species. We then show that these sign-pattern obstructions are specific to the pairwise Lotka-Volterra model: arbitrary prescribed growth rates and pairwise coefficients can be supplemented by higher-order interactions so as to admit a feasible asymptotically stable equilibrium. Through the correspondence with replicator dynamics, we interpret feasible equilibria of higher-order Lotka-Volterra systems as totally mixed symmetric Nash equilibria of symmetric multiplayer games, derive bounds on their number, and study their robustness under perturbations of the payoff tensors. We conclude by showing that every impossible ecology determines a nonempty open class of symmetric two-player games with no totally mixed evolutionarily stable strategy.

0.25

Homological Barriers to Stable Learning of First-Order Nash Equilibria

cs.GT cs.LG ·2026-11-08 ·Ashkan Soleymani, Gabriele Farina, Patrick Jaillet, Georgios Piliouras
published similarity 0.60
Abstract · The Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS) 2026

We identify a homological barrier to stable learning of local Nash equilibria, also known as first-order Nash equilibria (FONE), in constrained non-concave differentiable games. We consider continuous deterministic learning dynamics that fix every FONE point and aim to attract all initial conditions to the full FONE set. We show that this convergence guarantee is impossible when the target set has nontrivial positive-degree cohomology, such as that arising from a loop, and the state space is compact and contractible. We exhibit this barrier in a simple non-concave quadratic zero-sum game on a box whose exact FONE set consists of a diagonal six-edge loop and an isolated point. Consequently, no such dynamics can make the full FONE set a stable global attractor. We further show that degree two is the minimal polynomial degree at which this barrier can arise and extend the construction to obtain nontrivial homology in arbitrarily high degrees. Finally, we establish that the barrier persists for sufficiently small approximate-FONE targets under small perturbations, yielding an open, positive-volume family of quadratic zero-sum games with the same homological barrier.

0.41

Bayesian Replicator Dynamics and the Inversion of Strict Dominance in Bimatrix Games

cs.GT cs.MA ·2026-10-26 ·Jaime Ruiz-Serra, Nihat Ay, Michael Harré
similarity 0.60
Abstract · 18 pages, 3 figures

Replicator dynamics, the central model of evolutionary game theory, describe populations evolving under selection and agents learning by trial and error. They also admit a third use: continuous-time Bayesian inference, in which an agent planning against an opponent accumulates evidence for each candidate strategy. Here we show that this use divides all two-player games into two classes. With the standard expected-utility payoff, the dynamics can be expressed as Bayesian updating only in games (with any number of actions) where no player has a dominant strategy. The logarithmic payoff of Ay and Erb (2005) admits the Bayesian representation in every game. Coordination, anti-coordination, and cyclic games share qualitative dynamics under both payoff definitions, with equilibria displaced and selection thresholds shifted. In contrast, the flows of every game with a dominant strategy are reversed between the payoff definitions, with the strictly dominant strategy eliminated under logarithmic payoffs. Cooperation therefore emerges in the Prisoner's Dilemma, and games of common interest converge to mutual defection. For two-action games the classification is complete and rests on an exact Hamiltonian structure shared by both payoff definitions. The structure gives a behavioural test: cycles in the classic Matching Pennies game have amplitude-independent period (isochronous) under logarithmic payoffs, but not under expected utility. Logarithmic evidence integration thus determines which strategic outcomes dynamical selection can reach, wherever learning approximates Bayesian updating.

0.23

Evolutionary Stability Does Not Guarantee Learning Accessibility: A Multi-Agent Reinforcement Learning Perspective on Cooperation Emergence

cs.AI ·2026-09-23 ·Yijie Wang
similarity 0.59
Abstract

Cooperation emergence is a central problem in multi-agent systems because decentralized agents must coordinate while adapting to the changing behavior of others. Evolutionary game theory identifies strategically stable outcomes, but stability under a population adjustment dynamic need not imply that finite-sample learning agents can reach the same outcome through local reward feedback. We study this distinction in a transparent three-agent governance-motivated game involving a government, a platform firm, and users. We derive replicator dynamics for the fixed stage-game incentives, evaluate the cooperative evolutionary basin on a symmetric initial-condition grid, and compare it with learning-basin estimates for three decentralized value-based learners. The learning analysis uses independent Q-learning with $\varepsilon$-greedy action selection, scaled Boltzmann exploration, and SA--EA BQL under the same payoff environment and outcome criterion. The evolutionary basin has volume $V_E=1.00$ on the sampled grid. The empirical learning basin is $0.88$ for $\varepsilon$-IQL and $0.00$ for both scaled Boltzmann and SA--EA BQL. Diagnostic traces show that broader action diversity and nonzero value separation can coexist with failure to sustain the cooperative joint action in this fixed configuration. These results indicate that evolutionary stability and learning accessibility are distinct properties of a coupled game--learning system. The shared-bike setting is a motivating application; the broader contribution is a framework for comparing population-level stability with the finite-sample accessibility of cooperation under specified multi-agent learning dynamics.

0.13