CS Paper Radar

Critic-Free Policy Iteration for Continuous-Time Zero-Sum Games: A Policy-Space Riccati Approach

Jiacheng Wu, Yang Zhu, Hongye Su

eess.SY ·submitted 2026-09-10 13:57 UTC · v1

Abstract

This paper develops a critic-free policy iteration (PI) method for continuous-time linear zero-sum games. The central idea is to characterize the saddle-point policies directly in the joint policy space, rather than treating the quadratic value matrix as an iterative variable. A policy game Riccati equation (PGRE) is introduced whose unknowns are policy gains only. Its solutions are shown to be in one-to-one correspondence with the symmetric solutions of the game algebraic Riccati equation. Based on a stabilizing anchor, PI is performed directly in the actor space. The actor-space Jacobian is nonsingular at every stabilizing policy, and the resulting policy sequence coincides with that of simultaneous PI. For unknown dynamics, a data-driven algorithm uses a single batch of data and nullspace projection of endpoint increments to eliminate the value matrix, yielding an actor-only regression. A necessary and sufficient rank condition for unique policy recovery is established and shown to be iteration-invariant, demonstrating that critic identifiability is unnecessary. A power systems frequency-regulation example verifies convergence and policy recovery, while scalability tests demonstrate substantial reductions in computational and memory requirements.

Closest papers

Nearest neighbours by meaning across the whole index.

Data-Driven Policy Iteration Without an Initial Stabilizing Policy: A Finite-Horizon Bootstrap Method

eess.SY ·2026-09-15 ·Jiacheng Wu, Yang Zhu
similarity 0.64
Abstract

This article investigates data-driven policy iteration (PI) for continuous-time linear systems without requiring an initially stabilizing policy. Standard infinite-horizon PI is not self-starting because its policy-evaluation step is well posed only when the feedback gain is stabilizing. However, verifying this property is difficult when the system matrices are unknown. To remove this requirement, we develop a finite-horizon bootstrap method. The key idea is to perform policy evaluation over a compact interval for a shifted system, where the evaluation equation is well defined for arbitrary bounded time-varying policies. We show that, for a sufficiently long horizon, the initial-time optimal gain of the shifted finite-horizon problem, when applied as a constant feedback gain, achieves a prescribed stability margin for the original system. We then derive a data-driven implementation from an off-policy identity evaluated along trajectories of the original plant. We use basis-function approximations to reconstruct the finite-horizon value matrix and policy, and we characterize the resulting error through a perturbed policy-improvement recursion. A data-driven Lyapunov certificate is further introduced to verify admissibility of the candidate gain before it is used to initialize infinite-horizon PI. Numerical studies of a batch reactor and a two-mass-spring system demonstrate the effectiveness of the proposed bootstrap method.

0.15

Iterative State- and Control-Dependent Model Predictive Control: A Jacobian-Free Formulation for Constrained Nonlinear Systems

eess.SY math.OC ·2026-08-15 ·Mohammadreza Kamaldar
theory similarity 0.62
Abstract

This paper presents an iterative model predictive control algorithm that stabilizes constrained nonlinear systems without evaluating a single plant derivative. By factoring the exact nonlinear dynamics into a pseudo-linear form using state- and control-dependent coefficients (SCDCs), we replace the standard nonconvex optimization with a sequence of constrained linear-quadratic programs. Refreezing the coefficient matrices along the previously predicted trajectory drives the iteration. Near the origin, we prove this sequence contracts to a unique fixed point. We explicitly bound the number of iterations required to reach any stopping tolerance, and we quantify the distance from the fixed point to a true Karush-Kuhn-Tucker point, showing this optimality gap vanishes quadratically as the state approaches the origin. Inflating the discrete algebraic Riccati equation generates terminal ingredients that guarantee recursive feasibility and asymptotic stability, even when the solver terminates early. We adapt the terminal penalty online, proving it remains uniformly bounded, and we secure output feedback through the block-observable canonical form, which extracts the exact system state directly from past inputs and outputs. Retaining the block-banded structure of the subproblem forces the computational cost to scale linearly with the horizon length $\ell$. This $O(\ell)$ complexity matches the iterative linear quadratic regulator (iLQR) but sharply undercuts the $O(\ell^3)$ scaling of dense sequential quadratic programming (SQP). Numerical studies on a saturated quadrotor, a nonholonomic integrator, and a nonminimum-phase plant illustrate the theoretical bounds and map how the algorithm compares with iLQR, SQP, and linear-parameter-varying MPC.

0.34

Continuous-Time Critic as a Lyapunov Function: Projected Adaptation and Cart-Pole Stabilization

eess.SY math.DSmath.OC ·2026-10-07 ·Pavel Osinenko
theory similarity 0.60
Abstract

We study adaptive stabilization of known control-affine systems by treating the current parameterized critic itself as a Lyapunov candidate. Its total derivative contains a parameter-velocity term that can compensate unfavorable state evolution. A projection onto the intersection of a bounded admissible velocity set and one decay halfspace enforces parameter invariance and Lyapunov decrease simultaneously. We give an exact pointwise feasibility test, a conditional state-stability theorem, and a counterexample showing why a positive critic family containing a stabilizing member does not imply persistent feasibility. A convex endpoint constraint provides a distinct sampled implementation. A generic periodic polynomial-factor critic with a local Riccati anchor is evaluated on a force-limited cart-pole without mechanical-energy features, replay training, waypoints, or controller switching. The frozen controller passes stabilization and numerical decay checks on 600 sampled initial states across two draw sets; 400 matched continuous trajectories also pass. Independent reintegration audits and a conservative local feasibility certificate are distinguished from a global or machine-arithmetic guarantee. Compared with nonlinear model-predictive control, the measured implementation uses less CPU but has higher cost, settling time, and cart travel. The results support a conditional adaptive stabilization mechanism.

0.26

General-Sum Linear Regulator Games for Positive Systems

eess.SY math.OC ·2026-08-24 ·Alba Gurpegui, Monika Tomar, Takashi Tanaka
publishedtheory similarity 0.60
Abstract · 6 pages and 3 figures. Accepted for presentation at the 65th IEEE Conference in decision and control 2026 (CDC)

This paper studies a continuous-time general-sum non-cooperative game with linear costs, positive linear system dynamics, and elementwise linear input constraints. In the finite-horizon case, we present a verification theorem characterizing feedback Nash equilibria, in terms of absolutely continuous solutions of a coupled system of vector-valued ordinary differential equations, realized by time-varying feedback laws. Unlike linear-quadratic differential games, whose Riccati-based equilibria scale quadratically with the state dimension, this formulation scales linearly. However, the resulting piecewise-constant feedback saturates between its constraint bounds rather than varying smoothly, and additional mathematical challenges arise when characterizing the solutions of the differential equations, which are generally discontinuous due to the switching nature of the feedback gains. In this work, we study the case where switching occurs only at isolated time instants. In the infinite-horizon case, under stabilizability assumptions, the equilibrium is characterized by coupled vector-valued algebraic equations. For this game, we propose iterative methods to compute both finite and infinite-horizon equilibria. The approach is illustrated through a large-scale pollution game.

0.26

Reinforcement learning for constrained stochastic linear--quadratic control in continuous time

cs.LG math.OC ·2026-10-26 ·Weiping Wu, Qitong Pan, Xun Li, Peng Wu
similarity 0.58
Abstract

We study continuous-time stochastic linear--quadratic control with a scalar state and nonnegative control constraints. With quadratic running costs, randomized exploration can change the optimal mean feedback. We introduce an auxiliary problem without entropy in which the conditional policy mean is nonnegative and the covariance is prescribed in proportion to the squared state. Moment reduction and state separation yield two regional Riccati equations and characterize the finite-horizon value function and optimal conditional mean. We give a sufficient control--cost compatibility condition under which this mean coincides with the optimal feedback of the original constrained problem. For time-invariant coefficients, we characterize the undiscounted infinite-horizon optimum and establish mean square asymptotic stability under the stated assumptions. For unknown dynamics, we develop model-free actor--critic schemes for finite and infinite horizons. Numerical experiments compare the learned policies with their analytical targets and examine the effect of exploration on the optimal mean.

0.25

An α-potential linear approximation for LQ differential games

eess.SY math.OC ·2026-11-18 ·Balint Varga, Gergely Lorant Kiss, Andreas Schwung
similarity 0.58
Abstract

Feedback Nash equilibria of infinite-horizon linear-quadratic (LQ) differential games are characterized by coupled algebraic Riccati equations (cAREs), whose numerical solution generally relies on iterative algorithms and can be highly sensitive to their initialization. Motivated by the $α$-potential-game framework, this letter proposes an $α$-potential linear approximation for constructing a systematic, game-dependent initialization. For an N-player LQ game, a reference potential game is constructed. Its equilibrium requires solving only one standard algebraic Riccati equation. The original cAREs are subsequently linearized around this reference solution, yielding a coupled linear system whose solution provides player-specific corrections and hence an approximate feedback Nash equilibrium. The resulting approximation is used to initialize three standard cARE solvers. Numerical experiments on $10^4$ randomly generated two-player games show that the proposed initialization reduces the mean number of iterations. Further experiments for $N=2,\ldots,13$ show that the reduction in iteration count persists as the number of players increases.

0.29

Parallel Policy-Gradient Methods for Parameter Optimization of Nonlinear Feedback Controllers

eess.SY math.OC ·2026-09-12 ·An Nguyen, Leilei Cui
similarity 0.57
Abstract

Structured feedback controllers provide rigorous stability guarantees, but often require manual parameter tuning to achieve good closed-loop performance. Policy-gradient methods offer a systematic approach to parameter optimization; however, conventional gradient evaluation requires sequential forward state rollout and backward costate propagation. This letter develops a time-parallel policy-gradient framework for discrete-time nonlinear control-affine systems. We derive the policy-gradient expression where the state and costate rollouts required for policy-gradient evaluation are formulated as residual-minimization problems and solved using Gauss-Newton (GN) iterations with parallel associative scans. For closed-loop systems that are globally asymptotically stable and locally exponentially stable, we show that the residual-minimization problems satisfy a local Polyak-Lojasiewicz (PL) inequality and that the GN iterates converge locally at a quadratic rate. Moreover, the PL constant, the size of the convergence neighborhood, and the quadratic convergence bound are independent of the rollout horizon T. We also prove that, for any finite horizon T, the state solver recovers the exact trajectory from any initialization in at most T iterations. Finally, an inertia-wheel pendulum example with interconnection and damping assignment passivity-based control (IDA-PBC) demonstrates improved closed-loop performance and the computational benefits of the proposed parallel policy-gradient framework.

0.19

Improved Sample Complexity Analysis For First Order Bilevel Reinforcement Learning

cs.LG cs.AI ·2026-11-01 ·Mudit Gaur, Naman Saxena, Vaneet Aggarwal
theory similarity 0.56
Abstract

Bilevel reinforcement learning optimizes an upper-level objective through the policy response to a lower-level reinforcement learning problem. Such problems arise when learning a reward model while adapting a policy to that reward. However, a gap remains between sample-complexity guarantees and practical actor--critic training. Tabular analyses restrict applicability to finite spaces, while continuous-state analyses often impose a uniform Polyak--Łojasiewicz (PL) condition. We show that this condition need not hold even for smooth positive policies with exactly representable values. We develop a single-loop, fully first-order method for continuous states and finite actions with specified multilayer neural actors and a persistently trained nonlinear critic built from fixed features. Under explicit initialization, regularity, feature-excitation, and value-realizability conditions, fixed reference-KL regularization yields local curvature, and we prove containment of the coupled iterates. Shared trajectories control sampling noise, while paired critic analysis controls normalized critic-error differences as the response perturbation shrinks. A joint descent and tracking argument establishes an improved $\widetilde{\mathcal O}(\varepsilon^{-2})$ sample complexity for $O(\varepsilon)$ expected squared gradient norm of the upper objective induced by the selected regularized policy response, including critic training.

0.15