CS Paper Radar

KL-Regularized Policy Iteration for Adaptive LQR: Gain Updates and Probing from One Policy Improvement

Ajinkya Bhole, Mohammad Mahmoudi Filabadi, Guillaume Crevecoeur, Tom Lefebvre

eess.SY math.OC ·submitted 2026-11-09 17:22 UTC · v1

Abstract

Policy-gradient methods for adaptive linear quadratic control improve the feedback gain of an unknown plant online by gradient-type steps computed from closed-loop data. A recent framework of such methods, policy-gradient adaptive control (PGAC), takes one gradient-type step per sample, which its indirect form computes from an estimated model and its direct form directly from the data. Because data generated by a fixed linear feedback cannot identify the plant, PGAC adds a probing perturbation to the input and leaves its covariance as a design parameter. We study the adaptive LQR problem through a KL-regularized formulation, in which the probing is the randomness of a Gaussian policy kept close to a Gaussian reference policy, so that the cost of the probing is part of the objective. One policy improvement of this formulation gives a gain update together with a probing covariance. We show that, for suitable references centred at the current gain, this gain update is exactly PGAC's Gauss-Newton or natural-gradient update with any step size below a bound. We repeat this policy improvement, each time against a tempered copy of the current policy, and refer to this scheme as KL-regularized policy iteration (KLPI). On a known plant and with tempering, the cost of KLPI's gains converges linearly to the optimal cost, and the probing covariance keeps a floor proportional to the regularization weight. On an estimated plant, under conditions like PGAC's (small updates and sufficiently informative data), the closed loop stays stable, the cost of the gains converges linearly up to a bias, and the probing covariance again keeps this floor. In simulations at equal probing cost, the shape of KLPI's probing covariance, which PGAC can adopt, lowers the extra cost caused by suboptimal gains on nearly all tested plants, whereas KLPI's use of past iterations in its gain update changes this cost little.

Closest papers

Nearest neighbours by meaning across the whole index.

Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC

cs.RO cs.LGeess.SY ·2026-09-01 ·Baha Zarrouki, Arslan Thobani, Jasper Hoffmann, Mattia Piccinini, Rudolf Reiter +4 more
similarity 0.64
Abstract · Submitted to IEEE Transactions on Robotics (T-RO). 18 pages, 13 figures

In Model Predictive Control (MPC), cost-function weights shape closed-loop behavior, yet changing conditions often make fixed parametrizations suboptimal and motivate context-dependent online adaptation. Learning such policies is difficult because behavior depends implicitly on numerical MPC solutions, producing nonlinear, potentially nonsmooth, long-horizon dependencies on policy parameters. This creates a bias-variance tradeoff: Reinforcement Learning (RL) optimizes realized closed-loop return from environment samples but is sample-inefficient, whereas Gradient-Based Policy Learning (GB-PL) uses low-variance solver gradients from differentiable MPC to optimize surrogate losses on predicted trajectories but can be biased under model mismatch. We propose Solver-Gradient Guided Reinforcement Learning (SG-RL), a solver-sensitivity augmentation for RL-based online MPC cost-weight adaptation. SG-RL keeps sampled closed-loop return as the objective and uses bounded solver-derived gradients as auxiliary guidance to improve stability and sample efficiency. We instantiate SG-RL in Proximal Policy Optimization (PPO) with four modular algorithms that inject solver-gradient guidance into actor-update scaling, policy loss, advantage estimation, and value-function learning. On two full-scale autonomous racing platforms with intentional model mismatch, SG-RL reaches PPO's best closed-loop return with up to 70.6% fewer samples, outperforms GB-PL baselines by at least 54% in closed-loop return, and generalizes zero-shot to unseen environments.

0.41

Data-Driven LQR with Uncertainty Refinement and Domain Randomization---Part~II: Fisher-Majorized Certified Policy Iteration

eess.SY ·2026-10-24 ·Hamidreza Modares
theory similarity 0.64
Abstract

Building on the uncertainty description developed in Part~I, this paper introduces a Fisher-majorized domain-randomized generalized policy iteration framework for data-driven linear quadratic regulator (LQR) synthesis. At each iteration, partial Bellman evaluation over a fixed batch of uncertainty realizations produces a structured Sylvester policy-improvement step. A residual-weighted Fisher curvature controls the policy-induced variation in the stationary covariance and converts the Sylvester model into a certified upper bound on the cost change, thereby guaranteeing descent for every accepted step. A check-and-correct mechanism then certifies stability over the entire uncertainty support, rather than only over the sampled realizations. Consequently, every iterate is robustly stabilizing and monotonically decreases the fixed-batch objective. The Fisher truncation and stability correction become inactive after finitely many times. The objective gap then contracts linearly to a floor determined by the dispersion of the realization-wise Riccati targets and by measurement noise. In the absence of measurement noise, this floor vanishes when the realization-wise targets coincide.

0.30

Exploration as Feedback in Model Predictive Path Integral Control

eess.SY ·2026-11-14 ·Junsei Ito, Kohei Honda, Yasuaki Wasa
hot direction similarity 0.64
Abstract · 20 pages, 3 figures. Submitted to the 9th Annual Conference on Learning for Dynamics and Control (L4DC 2027)

This paper investigates how exploration affects closed-loop stability in model predictive path integral (MPPI) control. With a warm start, MPPI optimizes a control sequence by sampling candidates around the previous solution, and their covariance, the exploration covariance, is typically tuned to balance solution quality and computational cost. Small exploration is attractive because most samples then contribute to the solution, and existing stability analyses treat reduced exploration only as a bias toward the previous solution. Whether insufficient exploration can destabilize the closed loop has therefore remained open. We show that, for linear quadratic (LQ) problems with infinitely many samples, the exploration covariance also determines the feedback gain, i.e., how strongly MPPI updates the previous solution in response to the current state. With small exploration at a fixed temperature, MPPI nearly repeats the previous solution and fails to stabilize an exponentially unstable plant. Because a single covariance governs both exploration and feedback, stability and sample efficiency are in structural conflict: stabilization requires large exploration, which leaves few samples that contribute to the solution. To resolve this conflict, we propose Stabilizing Subspace MPPI (SS-MPPI), which computes the control inputs required for stabilization and samples only the rest. For LQ problems, SS-MPPI remains stable with small exploration and is thus sample-efficient. Simulations of an inverted pendulum and a quadruped robot confirm the analysis, showing that MPPI fails under small exploration, whereas SS-MPPI remains stable even with few samples.

0.32

First-Order Model-Free Policy Gradient for Undiscounted LQR with Global Optimality and Convergence Guarantees

eess.SY math.OC ·2026-11-16 ·Vipul K Sharma, S Sivaranjani
similarity 0.63
Abstract

We provide a first-order model-free policy gradient (PG) framework for learning undiscounted infinite-horizon linear quadratic regulator (LQR) policies with global optimality and convergence guarantees. We first construct a stochastic wrapper policy class parameterized by the LQR gain to enable exploration around a deterministic linear control law. We analyze the additional cost induced by exploration over infinite-horizon, and establish a weak gradient dominance condition that characterizes global optimality under exploration. We then construct a model-free first-order PG algorithm using sampled trajectory information, without identifying the dynamics or approximating a value function. Finally, we establish finite-time convergence guarantees, providing the sufficient number of samples required for near-global optimality with our proposed PG algorithms.

0.21

Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization

cs.CL ·2026-08-24 ·Xianlei Zhou, Xiangdi Meng, Yu He, Tianyu Qi, Shuyan Guan +5 more
top venuepositioncode/data similarity 0.63
Abstract · Accepted to EMNLP 2026 main conference

Policy optimization (PO) for Large Language Models faces a stability--exploration trade-off, currently mediated by an action-side Policy-KL regularizer. This puts practitioners in a double bind: keeping Policy-KL constrains response behavior and consumes the action-side exploration budget, while dropping it leaves the optimization without an explicit drift control. We argue for an alternative that breaks the dilemma by moving regularization to the input side. As training progresses, the distribution over training queries induced by the current policy drifts unchecked from its pre-RL reference distribution. Concretely, Environment-Regularized Policy Optimization (ERPO) introduces a Query-KL (QKL) term that bounds this query distribution shift, together with a dataset-static reference-derived per-query weight that biases each per-query update toward queries typical under the reference. The QKL gradient flows strictly through the query likelihood; the response score function used by policy-gradient estimators does not appear in the QKL term, so QKL exerts no direct gradient pressure on the response distribution---exploration is preserved. ERPO plugs into GRPO/PPO/REINFORCE-style pipelines without additional forward passes. On six mathematical reasoning benchmarks, ERPO replaces the standard Policy-KL regularizer while achieving effective control over query distribution drift, delivering stronger accuracy and substantially more stable behavior under high-temperature decoding and long-horizon training. Our source code are available at https://github.com/AlibabaResearch/ERPO

0.58

On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces

cs.LG math.OC ·2026-11-04 ·Jingyi Jin, Martha White, Hao Wang
similarity 0.62
Abstract · 66 pages, 1 figure. Submitted to the Journal of Machine Learning Research

Temporal-difference (TD) learning underlies many prediction and control algorithms, but can diverge when combined with off-policy learning, bootstrapping, and function approximation. TD with Regularized Corrections (TDRC) is theoretically sound and practically simple, but existing convergence theory covers only the one-step case under i.i.d. sampling, while the Huber Bellman-error formulation has not been extended to TDRC or eligibility traces. In this work, we establish convergence of constrained TDRC($λ$) with diminishing stepsizes and non-i.i.d. off-policy data, for state-dependent and history-dependent $λ$-parameters. We also propose TDRC($λ$)-Huber, which preserves the TDRC($λ$) fixed point while reducing sensitivity to large Bellman errors. For both algorithms, we apply an ODE-based weak-convergence method from stochastic approximation theory. For the nonlinear mean dynamics of TDRC($λ$)-Huber, we give an integral small-gain condition that guarantees uniqueness and global asymptotic stability of the target equilibrium. We then establish convergence of the constrained algorithm, derive computable sufficient conditions for convergence, and identify a regularization-based convergence mechanism shared by linear TDRC($λ$) and TDRC($λ$)-Huber. When the mean TD matrix is positive stable, both constrained algorithms converge under sufficiently strong regularization without requiring positive definiteness.

0.23

On sparsity and directional forgetting in adaptive control

eess.SY ·2026-09-20 ·Tochukwu E. Ogri, Trivikram Satharasi, Muzaffar Qureshi, Kyle Volle, Rushikesh Kamalapurkar
similarity 0.62
Abstract

This paper develops a sparsity-promoting memory regressor extension (MRE) adaptation law with directional forgetting for nonlinear control-affine systems with linearly parameterized uncertainty. The objective is to use directional forgetting to selectively discount obsolete information and leverage $\ell_1$ regularization to promote sparsity of the parameter estimates. While $\ell_1$ regularization has been applied to the system identification problem in an offline setting, a contribution of this paper is to develop a recursive least squares update law to implement $\ell_1$ regularization in online adaptive control. In particular, we show that $\ell_1$-regularized recursive least squares is realized via a sliding mode update law. A nonsmooth Lyapunov-based stability analysis is then used to show that the tracking and parameter estimation errors are ultimately bounded under a subspace excitation condition. Simulation results on a Van der Pol oscillator demonstrate the ability of the developed sparsity-promoting MRE controller to recover sparse dynamics while maintaining stable tracking.

0.09

Policy-Gradient Transfer under a Shared State Estimator

eess.SY ·2026-10-15 ·Dongming Wang, Wei Ren
similarity 0.62
Abstract

This paper considers the problem of policy-gradient transfer from a simulator to a physical plant when both are controlled through the same fixed state estimator. The plant and the simulator are partially observed nonlinear systems with additive Gaussian noise of common known covariance whose dynamics and observation models differ, and the controller is a squashed Gaussian policy with uniformly bounded latent mean and first two parameter derivatives. We derive a bound on the difference between the plant and simulator policy gradients that is linear in the noise-weighted model error and requires no estimator-accuracy guarantee, because the shared estimator is a common post-processing of the measurements and the data-processing inequality compares the trajectory laws without an estimation-error term. Based on this bound, we derive an occupancy-weighted refinement, a sufficient condition for the simulator gradient to be an ascent direction for the plant, and a bound on the expected squared plant gradient at a random iterate of a finite simulator-training run with optimization, truncation and model-error terms. Simulation results on a scalar system and a pendulum are presented to validate the theoretical results.

0.17