CS Paper Radar

Safe Reinforcement Learning with Hard Constraints: Policy Gradient Methods with Intersample Guarantees

Vipul K. Sharma, S. Sivaranjani

eess.SY math.OC ·submitted 2026-11-16 23:15 UTC · v1

Abstract

We develop a policy-gradient (PG) reinforcement learning (RL) framework with hard inter-sample control barrier function (CBF)-based safety constraints for nonlinear sampled-data systems. We first design a class of stochastic policies with support truncated to sampling-period-dependent safe control sets, that guarantee safety-by-construction at sampling instants. We then propose a model-free PG algorithm that searches over the class of safe-by-design policies and establish sufficient conditions on sampling-period such that any action sampled from safe control sets remains safe throughout the sampling interval, providing inter-sample guarantees. We further establish finite-time convergence and sample complexity guarantees for the proposed PG algorithm and characterize its global optimality under gradient dominance. Finally, we illustrate inter-sample safety guarantees on a safe quadrotor navigation environment.

Closest papers

Nearest neighbours by meaning across the whole index.

SafePG: Safe and Globally Optimal Reinforcement Learning with Hard Constraints

eess.SY ·2026-09-12 ·Vipul K. Sharma, Wesley A. Suttle, S. Sivaranjani
published similarity 0.81
Abstract · Accepted to the IEEE Conference on Decision and Control (CDC) 2026

We present an optimal and convergent model-free policy gradient (PG) reinforcement learning (RL) framework for controlling nonlinear dynamical systems under hard safety constraints. We first construct a class of stochastic wrapper policies centered around a deterministic controller, thereby enabling exploration in unknown environments while preserving the underlying deterministic control structure. We then define a class of parameterized safe-by-construction control policies by truncating these stochastic policies onto hard safety constraints. We next establish, via measure-theoretic arguments, that the potentially nonconvex RL objective under the truncated policy class, as well as its policy gradients, are well-defined. We then develop a model-free PG algorithm based on stochastic gradient ascent that directly searches over these truncated policies and leverage gradient dominance to establish convergence and optimality guarantees. Finally, we validate this framework through simulations on a safe quadrotor navigation problem.

0.30

First-Order Model-Free Policy Gradient for Undiscounted LQR with Global Optimality and Convergence Guarantees

eess.SY math.OC ·2026-11-16 ·Vipul K Sharma, S Sivaranjani
similarity 0.74
Abstract

We provide a first-order model-free policy gradient (PG) framework for learning undiscounted infinite-horizon linear quadratic regulator (LQR) policies with global optimality and convergence guarantees. We first construct a stochastic wrapper policy class parameterized by the LQR gain to enable exploration around a deterministic linear control law. We analyze the additional cost induced by exploration over infinite-horizon, and establish a weak gradient dominance condition that characterizes global optimality under exploration. We then construct a model-free first-order PG algorithm using sampled trajectory information, without identifying the dynamics or approximating a value function. Finally, we establish finite-time convergence guarantees, providing the sufficient number of samples required for near-global optimality with our proposed PG algorithms.

0.23

Towards Probabilistically Safe Reinforcement Learning: Theory and Deployment on Humanoid and Aerial Robots

cs.RO eess.SY ·2026-11-14 ·Pol Mestres, Varun Gabbita, Jason H. K. Siu, Sergio A. Esteban, Hampus Serneke +2 more
similarity 0.71
Abstract

Reinforcement learning (RL) enables robots to acquire complex skills, but standard RL formulations do not guarantee that learned policies satisfy safety constraints, which limits their use in safety-critical settings. Existing safe RL methods with provable guarantees typically support only restricted classes of constraints and require unbiased estimators of the objective and constraint functions, which prevents their direct integration with state-of-the-art algorithms such as proximal policy optimization (PPO). In this paper, we extend the recently introduced robust safe gradient flow-based reinforcement learning (RSGF-RL) framework to overcome these limitations. Our formulation accommodates a general class of trajectory-level safety constraints, including chance constraints such as collision avoidance with a prescribed probability, and our analysis allows for biased value and gradient estimators. We show that, with high probability, the constraint violation throughout training is bounded by the estimator bias plus a term that vanishes as the number of rollouts grows, and decays to this level if the initial policy is unsafe. We also establish the convergence properties of the iterates to approximate KKT points of the original safe RL problem. Finally, we show that RSGF-RL implemented with PPO-style estimates enables zero-shot sim-to-real deployment of whole-body control policies on a Unitree G1 humanoid and collision-free waypoint navigation on a Crazyflie drone.

0.19

Constrained Differentiable Policy Learning for Safe Quadrotor Navigation

cs.RO ·2026-10-12 ·Yan Rui Tan, Praveen Krishnapur, Zeyu Feng, Han Bao, Yew-Soon Ong +1 more
similarity 0.71
Abstract

Differentiable simulation enables efficient policy optimization through analytic gradients, yet safety is commonly incorporated as a weighted penalty, requiring manual tuning of the trade-off between task performance and constraint satisfaction. We instead formulate obstacle avoidance as an explicit constraint and train quadrotor policies using primal--dual gradient descent--ascent through a differentiable simulator. We introduce two key components to enable effective constrained training. First, we propose a bounded clearance-based constraint function that provides informative gradients around the safety boundary while preventing large violations from destabilizing primal--dual optimization. Second, we introduce log-mean-exp (LME) aggregation to smoothly emphasize critical per-step safety violations across a trajectory and training batch within a single constraint governed by one dual variable. Together, these components enable the desired safety level to be specified directly while balancing constraint satisfaction and task completion. We theoretically prove that the proposed optimization with differentiable simulation converges to feasible policy under mild assumptions. Experiments demonstrate improved safety--performance trade-offs over fixed-penalty and existing policy-learning baselines, as well as generalization to denser out-of-distribution environments. Finally, the learned policy transfers zero-shot to Gazebo/PX4 and a physical quadrotor without additional fine-tuning.

0.18

Successive Convexification in Learned World Models for Safe Offline Planning

cs.LG cs.ROeess.SY ·2026-11-14 ·Samuel Younger
code/data similarity 0.69
Abstract · 28 pages, 13 figures. Under review at L4DC 2027

World models learned from offline data allow an agent to plan without further interaction with the environment, but the sampling-based planners used with them handle safety constraints only by scoring or rejecting sampled action sequences. We instead pose safe planning in the latent space of a learned world model as a successive convex program (SCP), in which a learned discrete-time control barrier function is imposed as a constraint at every step of the horizon. SCP requires smooth dynamics, so we build the networks from smoothed MaxMin activations and spectrally normalised layers, and bound the resulting linearisation error in terms of the Lipschitz bound and the smoothing width. As SCP is a local method, we start it from a sampled plan. On two Safety-Gymnasium navigation tasks with offline data, this planner has fewer violations than MPPI and CEM drawing 128 times as many samples at equal wall-clock time in our main comparison, though not significantly, and earns greater return at longer horizons, and is much safer than SCP without a sampled start. SCP converged more often on Lipschitz-bounded models. Unlike sampling, it also reports at each step whether its plan is a feasible, first-order optimal KKT point of the planning problem in the model, or that no feasible plan was found nearby. Our implementation is publicly available at https://github.com/youngers2006/scwm.

0.45

RL-Guided PAC-NMPC for Probabilistically-Safe Perception-Based Navigation in Unknown Environments

cs.RO ·2026-09-30 ·Adam Polevoy, Dillon Capalongo, Katherine Tang, Mark Gonzales, Marin Kobilarov +1 more
similarity 0.69
Abstract

In this paper, we present an approach for combining stochastic nonlinear model predictive control (SNMPC) and reinforcement learning (RL) to enable probabilistically-safe perception-based navigation in unknown environments. Our method first uses RL to train probabilistic actor-critic and sensor prediction models. We then leverage these probabilistic models in a sampling-based SNMPC framework known as Probably Approximately Correct (PAC)-NMPC, which uses hard constraints to enforce finite-time statistical guarantees on the probability of collision and value function improvement. By ensuring that our finite-horizon SNMPC policies decrease the value function in expectation, we can approach the long-horizon performance of the RL approach while satisfying probabilistic safety constraints. Through simulation experiments, we show that our approach can improve the safety of perception-based RL navigation policies and scale to high dimensional systems with large sensor input spaces and complex nonlinear dynamics. We also demonstrate our approach through hardware experiments, showing improved performance for vision-based navigation with an agile fixed-wing aerial vehicle in unknown environments.

0.12

Forward-Invariant Policy Classes for Safe Reinforcement Learning in Multicopter Control

eess.SY ·2026-09-29 ·Chieh Tsai, Muhammad Junayed Hasan Zahed, Jinzhi Shen, Yi Xie, Ruoshan Lan +3 more
similarity 0.69
Abstract

This paper proposes a reinforcement learning (RL) framework for safe gain scheduling based on forwardinvariance-induced action-space design. Under stated nominalmodel and inversion-domain assumptions, rather than enforcing safety through runtime shielding or penalty-based constraints, safety is embedded directly into the policy class. Specifically, we construct a finite library of feedback controllers sharing a common Lyapunov certificate that establishes forward invariance of a prescribed admissible set under arbitrary switching. Consequently, any policy whose actions are restricted to this library, including policies encountered during RL exploration, inherits the same certificate. Policy optimization can therefore focus on closed-loop performance without runtime safety filtering or action projection. The framework is instantiated for quadcopter hover regulation, where a DQN schedules among certified feedback controllers. Nonlinear MuJoCo simulations demonstrate state-dependent gain scheduling and empirically evaluate robustness to wind, model mismatch, sensor noise, and sensing delay. The results illustrate how safety certification can be separated from policy optimization by learning over a forward-invariant policy class.

0.12

Sample-Efficient Model Predictive Control via Proximal Policy Optimization

eess.SY ·2026-11-17 ·Jiarui Wang, Sina Sharifi, Mahyar Fazlyab
similarity 0.67
Abstract

Sampling-based model predictive control (MPC) enables real-time control of complex nonlinear systems but often incurs substantial computational costs due to repeated trajectory simulations. In this paper, we bridge policy optimization in reinforcement learning and sampling-based MPC to improve computational efficiency through sample reuse. Starting from a KL-regularized distributional formulation of trajectory optimization, we derive an exact performance-difference identity based on a signed exponential advantage function. This identity motivates two algorithms, PPO-MPPI and SPO-MPPI, which adapt proximal policy optimization techniques to sampling-based MPC, enabling multiple optimization updates using previously evaluated trajectories without additional forward simulations. We establish population-level monotonic improvement guarantees for both methods and prove convergence to first-order stationarity for PPO-MPPI under fixed-covariance Gaussian sampling distributions. Experiments on quadruped velocity tracking and quadrotor path tracking demonstrate that the proposed methods achieve competitive control performance at substantially lower computational cost than multi-stage sampling approaches. Our results establish a principled connection between policy optimization and sampling-based MPC and highlight the potential of trajectory reuse for sample-efficient predictive control.

0.29