CS Paper Radar

Adaptive dynamic programming using Lyapunov function constraints

Thomas Göhrt, Pavel Osinenko, Stefan Streif

eess.SY math.OCmath.DS ·submitted 2026-10-08 15:07 UTC · v1
publishedhighly cited

Abstract

This work is concerned with a stabilizing adaptive dynamic programming (ADP) approach to approximate solution of a given infinite-horizon optimal control problem. Since the latter problem cannot, in general, be solved exactly, a parametrized function approximator for the infinite-horizon cost function is introduced in ADP (so called ``critic''). This critic is used to adapt the parameters of the function approximator. The so called ``actor'' in turn derives the optimal input of the system. It is a notoriously hard problem to guarantee closed-loop stability of ADP due to the use of approximation structures in the control scheme. Since at least stabilizability is always assumed in the analyses of ADP, it is justified to invoke a respective Lyapunov function. The proposed ADP scheme explicitly uses the said Lyapunov function to simultaneously optimize the critic and guarantee closed-loop stability. A Hessian-free optimization routine is utilized for the actor and critic optimization problems. Convergence to prescribed vicinities of the optima is shown. A computational study showed significant performance improvement for the critic-based approach compared a nominal stabilizing controller for a range of initial conditions.

Journal ref
IEEE Control Systems Letters, vol. 3, no. 4, pp. 901-906, Oct. 2019
DOI
10.1109/LCSYS.2019.2919439

Closest papers

Nearest neighbours by meaning across the whole index.

Approximate Feedback Nash Equilibria in Constrained Differential Games via Model Predictive Control with Upper Bound Guarantees

eess.SY cs.GT ·2026-10-03 ·Balint Varga, Karl Handwerker, Imre Rudas, Peter Galambos
theory similarity 0.60
Abstract

Physical human--machine interaction and other multi-agent control settings require decision-making policies that adapt online. Differential games provide a principled framework where each agent optimizes an individual objective while anticipating the other's response. The relevant solution concept in many applications is the feedback Nash equilibrium (FNE), which yields time-consistent state-feedback strategies. However, computing the FNE is demanding and becomes intractable when state and input constraints must be enforced, motivating the need for approximate methods. This paper presents a Model Predictive Control (MPC) approach that approximates infinite-horizon FNE trajectories through repeated solution of finite-horizon open-loop games. An auxiliary-game formulation is introduced that selects prediction horizons and terminal costs to approximate the feedback-game optimality conditions. The approach is extended to incorporate hard constraints via a constrained open-loop game formulation. For the unconstrained setting, an analytic upper bound on the state-trajectory deviation between the MPC-induced and FNE trajectories is derived, enabling quantitative performance certification. Numerical examples illustrate the effectiveness of the proposed method compared with baseline approaches from the literature.

0.26

Safe Newton-Based Extremum Seeking for Static Maps with Delayed Output Measurements

eess.SY ·2026-09-14 ·Azad Ghaffari, Tiago Roux Oliveira
similarity 0.60
Abstract

This work presents a delayed safe Newton-based extremum seeking (SANES) framework for minimizing an unknown static map subject to an unknown safety constraint. The objective and safety measurements are assumed to be affected by the same constant time delay. To compensate for delayed measurements, a model-free predictor is developed to construct the quantities required for optimization, including the nominal Newton-based extremum-seeking control input and the gradient information used to formulate control Lyapunov function (CLF) and control barrier function (CBF) conditions. Robust CLF--CBF quadratic programs (QPs), subject to parameter-update constraints, are then formulated to account explicitly for derivative-estimation and prediction errors. For the delay-free case, robustness margins are derived from bounds on the extremum-seeking estimation errors, whereas for the delayed case, the margins incorporate both estimation and prediction errors. Practical stability of the nominal Newton-based extremum-seeking dynamics is established directly through a Lyapunov analysis, thereby providing convergence guarantees over the admissible parameter set without relying exclusively on local averaging arguments. Robust CLF and CBF conditions are subsequently derived to establish practical convergence and forward invariance of a robust subset of the prescribed safe set. A numerical case study demonstrates the effectiveness of the proposed SANES framework in achieving constrained optimization despite unknown objective and safety maps and delayed measurements.

0.45

Weak Convexity and Proximal Bundle Methods for Nonsmooth Policy Optimization in Robust Control

eess.SY math.OC ·2026-09-05 ·Yuto Watanabe, Feng-Yi Liao, Yang Zheng
similarity 0.58
Abstract · 36 pages, 15 figures

We study policy optimization for discrete-time robust $\mathcal{H}_\infty$ control with static output-feedback, and present the first feasibility-preserving algorithm with a deterministic, non-asymptotic complexity guarantee. This problem naturally leads to a nonsmooth and nonconvex optimization over the set of stabilizing feedback gains. We first establish several structural properties of the $\mathcal{H}_\infty$ cost. In particular, we show that the cost is weakly convex on every convex subset of a sublevel set. For the state-feedback case, we further establish a weak Polyak--Łojasiewicz inequality, which ensures that every stationary point is globally optimal. Building on these properties, we develop a proximal bundle method for $\mathcal{H}_\infty$ policy optimization. The proposed method can be viewed as an implementable approximation of the proximal point method and uses only function value and subgradient information. We show that all iterates remain stabilizing and establish a deterministic non-asymptotic complexity bound of $\mathcal{O}(\max\{η^{-4},ε^{-2}\})$ for finding an $(η,ε)$-stationary point. Numerical experiments illustrate our theoretical results.

0.30

Infinite-Horizon Sparse Optimal Control: Solution through a Finite-Horizon Subproblem and Its Receding-Horizon Implementation

eess.SY math.OC ·2026-08-18 ·Yasuaki Oishi, Takumi Iwata, Masaaki Nagahara
similarity 0.58
Abstract · 17 pages, 4 figures

Sparse optimal control is considered in the infinite horizon. In the literature, sparse control has been considered mostly in a finite horizon for its formulation into a finite-dimensional optimization problem. It is shown in this paper that an optimal solution of the infinite-horizon sparse control problem can be obtained through a solution of some finite-horizon subproblem. This is due to sparsity of the optimal solution in the sense that the optimal control input is constantly equal to zero at its tail. An estimate is given on the horizon length required by this subproblem and its adaptive choice is also discussed. Implementation with a receding-horizon technique is considered and its optimality and sparsity are guaranteed.

0.59

Continuous-Time Critic as a Lyapunov Function: Projected Adaptation and Cart-Pole Stabilization

eess.SY math.DSmath.OC ·2026-10-07 ·Pavel Osinenko
theory similarity 0.58
Abstract

We study adaptive stabilization of known control-affine systems by treating the current parameterized critic itself as a Lyapunov candidate. Its total derivative contains a parameter-velocity term that can compensate unfavorable state evolution. A projection onto the intersection of a bounded admissible velocity set and one decay halfspace enforces parameter invariance and Lyapunov decrease simultaneously. We give an exact pointwise feasibility test, a conditional state-stability theorem, and a counterexample showing why a positive critic family containing a stabilizing member does not imply persistent feasibility. A convex endpoint constraint provides a distinct sampled implementation. A generic periodic polynomial-factor critic with a local Riccati anchor is evaluated on a force-limited cart-pole without mechanical-energy features, replay training, waypoints, or controller switching. The frozen controller passes stabilization and numerical decay checks on 600 sampled initial states across two draw sets; 400 matched continuous trajectories also pass. Independent reintegration audits and a conservative local feasibility certificate are distinguished from a global or machine-arithmetic guarantee. Compared with nonlinear model-predictive control, the measured implementation uses less CPU but has higher cost, settling time, and cart travel. The results support a conditional adaptive stabilization mechanism.

0.26

Online Constraint Optimization-Based Indirect-Adaptive Robust Control for Autonomous Vehicles with Unknown Friction

eess.SY cs.RO ·2026-10-15 ·Yupeng Zhou, I-Chia Chang, Ziran Wang, Bin Yao
code/data similarity 0.57
Abstract

Friction uncertainty is a significant challenge for autonomous vehicle control, undermining feasibility and performance. This paper proposes a novel two-layer control framework to achieve vehicle motion control under large friction uncertainty by combining optimization-based planning with indirect adaptive robust control (IARC). The upper layer solves a constrained optimal control problem to generate a feasible trajectory that satisfies all the constraints in a time-optimal manner along the desired path. The lower layer, IARC, adapts the unknown parameters related to friction uncertainty and combines model compensation with robust feedback to ensure transient and steady-state tracking performance. An estimation monitor is designed to prevent poor learning when data indicate a model mismatch, while a smooth parameter reset reduces the estimated tire capacity and limits abrupt tightening of constraints. The updated parameters are fed back into the model and constraints in the planner, enabling trajectory replanning as friction conditions change. Simulation results indicate that the proposed method achieves accurate tracking of the desired trajectory and online friction adaptation under large friction uncertainty while ensuring time-optimal vehicle motion. The simulation results demos are available at https://github.com/TedZhou0603/CO-IARC.

0.27

Safe Receding-Horizon Control of Fixed-Wing Aircraft Using Constrained Approximate Dynamic Programming

eess.SY ·2026-10-05 ·Felipe Arenas-Uribe, Ricardo Gutierrez, Jesse B. Hoagg
similarity 0.57
Abstract

We present a receding-horizon optimal control for fixed-wing aircraft subject to state constraints. These constraints include operational constraints such as altitude, geofencing, and obstacle avoidance, as well as bounds on flight-path angle, roll angle, and airspeed. The state constraints are composed into a single soft-minimum control barrier function (CBF). We use this composite CBF in a constrained-approximate dynamic program to obtain a sequence of analytic closed-form control functions that approximately minimize a quadratic finite-horizon integral cost subject to the CBF constraint. The resulting receding-horizon control is non-myopic in the sense that it approximately optimizes the integral cost while satisfying the state constraint at all times along the entire prediction horizon. We demonstrate constraint satisfaction and performance in simulations of a fixed-wing aircraft navigating an obstacle-filled airspace under wind uncertainty. We also compare this receding-horizon control with 2 other methods.

0.14

Predefined-Time Resilient Integral Reinforcement Learning for Input-Constrained Unknown Nonlinear Systems Under FDI Attacks and Disturbances: A Fully Data-Driven Approach

eess.SY ·2026-09-10 ·Tien Dat Vu, Minh Doan
hot direction similarity 0.57
Abstract

This paper investigates optimal control for nonlinear systems with unknown dynamics, input constraints, disturbances, and adversarial signals. The objective is to develop a learning-based control method that allows the designer to prescribe the desired convergence time in advance. An integral reinforcement-learning framework is proposed to avoid requiring exact knowledge of the system dynamics while ensuring that the control input always satisfies the actuator constraints. Current and recorded data are combined to train the critic without requiring persistent excitation. The learning gain is selected directly from the prescribed convergence deadline. Lyapunov analysis is then used to establish practical predefined-time convergence of the coupled state-critic system in the presence of disturbances and adversarial channels. The effectiveness of the proposed method is validated through the stabilization control of a two-link robot manipulator.

0.46