CS Paper Radar

Policy Synthesis for Finite Populations of MDP Agents under Aggregate Reach-Avoid Chance Constraints

Jie Fu, Anamika Dubey

eess.SY cs.MA ·submitted 2026-10-08 14:23 UTC · v1

Abstract

Consider a finite population of agents with decoupled Markov transition dynamics and empirical-density feedback, subject to the following constraints: with probability at least $1-δ_r$, at least a fraction $α_r$ of agents must reach a target region at some time $t^*$, while, at each time up to $t^*$, the unsafe population fraction must remain below $β_u$ with probability at least $1-δ_u$. However, standard mean-field methods enforce these constraints only in expectation, which fails to account for stochastic fluctuations at finite fleet size $N$. To address this control problem, we propagate the second-order moment (variance) of the empirical density alongside the mean-field trajectory via a discrete-time Lyapunov recursion, and apply the Cantelli inequality to convert chance constraints into tractable deterministic conditions on the moments of the empirical density. We then incorporate these moment-based surrogate constraints into a gradient-based sequential convex approximation procedure for density-feedback policy synthesis. We further introduce additional moment-error bounds to construct a rigorous finite-$N$ certificate. The method is evaluated on a gridworld environment and a power-system EV-charging aggregation problem and compared with a standard deterministic population-level LP baseline.

Comment
8 pages, 2 figures. Submitted to the 2027 American Control Conference

Closest papers

Nearest neighbours by meaning across the whole index.

Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control

cs.LG math.OC ·2026-10-08 ·Adonis Jamal, Samy Mekkaoui, Yadh Hafsi, Huyên Pham
theory similarity 0.64
Abstract

We develop a model-free policy gradient method for discrete-time mean-field control (MFC). In MFC, the policy affects the objective both through the controlled dynamics and through the population distribution. Standard REINFORCE estimators capture the first effect but not the second. We introduce Transport REINFORCE, a transport map-based approach that perturbs a suitable transformation of the population distribution to estimate this missing mean-field contribution. The method applies to both finite and continuous state spaces. In finite state spaces, we perturb the population distribution directly on the probability simplex through a convex combination of the current population weights and random weights. In continuous state spaces, we project the population distribution onto the manifold of Gaussian mixtures, and then randomize it via a transport map that ensures the perturbed law remains within this manifold. We prove consistency of the perturbed objective and gradient as the perturbation vanishes, and derive bias and mean-square error bounds for the resulting sample-based gradient estimator. Numerical experiments on several MFC benchmarks show that Transport REINFORCE improves over standard REINFORCE.

0.31

Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations

cs.MA cs.LGstat.ML ·2026-08-26 ·Aditya Makkar, Benjamin Unger, Jeongyeol Kwon, Mathieu Laurière, Eugene Vinitsky +1 more
similarity 0.63
Abstract · 33 pages, 3 figures

Modern multi-agent systems are increasingly deployed at scale over large populations of agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each agent's policy independently, treating the other agents as part of a fixed single-agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.

0.19

Learning Chance-Constrained MDPs with Bellman Distributional Certificates

cs.LG ·2026-09-25 ·Chenbei Lu, Hongyu Yi
top venuebenchmarktheory similarity 0.62
Abstract · Accepted to NeurIPS 2026

Safe reinforcement learning (RL) commonly enforces expected-cost constraints, but such expectation safety may fail to control the probability of rare high-cost trajectories. Chance-constrained MDPs (CCMDPs) impose a stronger probability-level requirement, but are widely viewed as harder because the chance constraint is nonconvex and depends on the full trajectory rather than a Bellman-linear expectation. In this paper, we reveal that this computational difficulty does not necessarily imply a higher statistical price. For tabular discounted CCMDPs with fixed bounded successor support and access to a certified planning oracle, we establish a model-based upper bound, with a matching lower bound up to logarithmic terms. Technically, our key idea is the \emph{Bellman distributional certificate}, which constructs a Bellman recursion for constraint violation probabilities before policy selection. The certificate can be reused across candidate policies; combined with shared row-wise reverse-KL confidence sets, it gives a policy-uniform trajectory-KL transfer without a union bound over policies or time--budget Bellman tables. For stochastic policies, we give a model-free variance-reduced policy-gradient algorithm with a finite-sample expected KKT-residual guarantee and independent validation of every accepted policy. Numerical experiments on synthetic CCMDPs and an IEEE 14-bus energy storage control benchmark illustrate the safety and mechanism behavior of the proposed algorithms.

0.36

Reinforcement learning for constrained stochastic linear--quadratic control in continuous time

cs.LG math.OC ·2026-10-26 ·Weiping Wu, Qitong Pan, Xun Li, Peng Wu
similarity 0.61
Abstract

We study continuous-time stochastic linear--quadratic control with a scalar state and nonnegative control constraints. With quadratic running costs, randomized exploration can change the optimal mean feedback. We introduce an auxiliary problem without entropy in which the conditional policy mean is nonnegative and the covariance is prescribed in proportion to the squared state. Moment reduction and state separation yield two regional Riccati equations and characterize the finite-horizon value function and optimal conditional mean. We give a sufficient control--cost compatibility condition under which this mean coincides with the optimal feedback of the original constrained problem. For time-invariant coefficients, we characterize the undiscounted infinite-horizon optimum and establish mean square asymptotic stability under the stated assumptions. For unknown dynamics, we develop model-free actor--critic schemes for finite and infinite horizons. Numerical experiments compare the learned policies with their analytical targets and examine the effect of exploration on the optimal mean.

0.25

Safe Leader-Follower Density Control via Higher-Order Mean-Field Control Barrier Functions

eess.SY math.OC ·2026-10-03 ·Cinzia Tomaselli, Mario di Bernardo
similarity 0.61
Abstract

Mean-field control barrier functions (MF-CBFs) enforce safety of large-scale multi-agent systems at the level of densities, but existing formulations require the control to act directly on the population to be kept safe. We consider populations that are not directly actuated and are influenced only through other, controlled populations, possibly in the presence of diffusion. We show that if the constrained population lies at directed graph distance d from the actuated one, then, under a nondegeneracy condition, the safety functional has relative degree d+1 and the control enters its (d+1)-th time derivative affinely. Higher-order safety constraints can therefore still be enforced through a convex quadratic program. We apply the framework to leader-follower density control among dangerous regions, where the follower safety functional has relative degree two and the leader safety functional has relative degree one. We characterize exactly when the two constraints are jointly infeasible and bound the tracking errors induced by the safety correction. Simulations in cluttered environments show that the proposed filter preserves follower safety, whereas enforcing leader safety alone does not.

0.18

Equivalence Between Continuous-Time Risk-Sensitive Control and Rényi Divergence Minimization

eess.SY math.OC ·2026-09-07 ·Shinji Kataoka, Kaoru Teranishi, Yasumasa Fujisaki
theory similarity 0.61
Abstract · 6 pages, 1 figure

In this study, we show that a continuous-time risk-sensitive control problem is equivalent to a Rényi divergence minimization problem over trajectory path measures. Reformulating stochastic optimal control as probabilistic inference via Kullback-Leibler (KL) divergence minimization avoids the computational intractability of the Hamilton-Jacobi-Bellman equation. However, standard KL control is inherently risk-neutral, and recent minimax extensions remain restricted to risk-averse settings. Our equivalence result resolves this limitation by offering a unified probabilistic framework for arbitrary risk attitudes in continuous-time nonlinear systems. Based on Girsanov theorem, we explicitly map the risk sensitivity to the Rényi divergence order, deriving a noise-dependent control penalty scaled by risk preference. This formulation seamlessly modulates tail-weighting behaviors, interpolating between zero-forcing for risk-averse policies and mass-covering for risk-seeking policies. These findings bridge stochastic control and information-theoretic inference, providing a foundation for sampling-based control algorithms.

0.31

Control-Gibbs Distribution: Sampling for Optimal Control and Diverse Actions

eess.SY ·2026-11-12 ·Shengling Shi
similarity 0.61
Abstract

While conventional optimal control selects an optimal input sequence, probabilistic control assigns probability mass over all possible inputs and exploits a distribution of control sequences. Control distributions help enable exploration, capture diverse solutions, and solve deterministic optimal control problems. This work reviews an important probabilistic model for control distributions, the control-Gibbs distribution, i.e., the Gibbs distribution with an optimal control objective as the energy function. We point out its fundamental role in various probabilistic control frameworks, e.g., maximum entropy control, control-as-inference framework, model predictive path integral control, and energy-based models for control. This provides a model-central perspective to unify these probabilistic control frameworks. We also review the associated extensions of dynamic programming for computing closed-loop control distributions. To sample the control-Gibbs distribution, we consider projected Langevin sampling and develop non-asymptotic guarantees. The guarantees show the dual benefits of sampling the control-Gibbs distribution: obtaining diverse control solutions and solving the deterministic optimal control problem. We provide a numerical example where diverse arm trajectories are sampled from a control-Gibbs distribution.

0.29

Log-Derivative Trick for Parametric Markov Chains

cs.LG cs.LO ·2026-10-27 ·Arthur Gall, Ichiro Hasuo, Thomas Seiller
theory similarity 0.60
Abstract

Optimizing expected rewards and reachability probabilities in parametric Markov chains (pMCs) is central to the analysis and synthesis of probabilistic systems with loops. While traditional symbolic synthesis approaches--such as parameter lifting and rational function generation--suffer from scalability bottlenecks, gradient-based optimization emerges as an effective alternative. However, existing gradient-based techniques still rely on symbolic gradient computation--such as solving large linear systems--over the global state space, inheriting scalability limits. Finite-horizon approximation is possible but introduces uncontrollable bias. In this paper, we introduce a scalable Monte Carlo gradient estimator based on a measure-theoretic generalization of the log-derivative trick. We prove that our estimator provides unbiased gradient estimates under almost-sure termination and standard regularity conditions. Crucially, our method avoids global state-space materialization and linear system solving entirely: it operates purely along sampled trajectories, requiring only black-box evaluations of local transition probabilities (or densities) and their derivatives. Unlike existing pMC techniques that are fundamentally restricted to finite discrete systems, our measure-theoretic framework naturally extends to continuous and general $σ$-finite state spaces. Empirical evaluations on synthetic discrete pMCs and continuous-state case studies demonstrate that our method scales to state spaces far beyond the reach of existing solvers, significantly reducing execution time without introducing bias.

0.22