CS Paper Radar

Forward-Invariant Policy Classes for Safe Reinforcement Learning in Multicopter Control

Chieh Tsai, Muhammad Junayed Hasan Zahed, Jinzhi Shen, Yi Xie, Ruoshan Lan, Majed Obaid, Salim Hariri, Hossein Rastgoftar

eess.SY ·submitted 2026-09-29 23:22 UTC · v1

Abstract

This paper proposes a reinforcement learning (RL) framework for safe gain scheduling based on forwardinvariance-induced action-space design. Under stated nominalmodel and inversion-domain assumptions, rather than enforcing safety through runtime shielding or penalty-based constraints, safety is embedded directly into the policy class. Specifically, we construct a finite library of feedback controllers sharing a common Lyapunov certificate that establishes forward invariance of a prescribed admissible set under arbitrary switching. Consequently, any policy whose actions are restricted to this library, including policies encountered during RL exploration, inherits the same certificate. Policy optimization can therefore focus on closed-loop performance without runtime safety filtering or action projection. The framework is instantiated for quadcopter hover regulation, where a DQN schedules among certified feedback controllers. Nonlinear MuJoCo simulations demonstrate state-dependent gain scheduling and empirically evaluate robustness to wind, model mismatch, sensor noise, and sensing delay. The results illustrate how safety certification can be separated from policy optimization by learning over a forward-invariant policy class.

Closest papers

Nearest neighbours by meaning across the whole index.

SafePG: Safe and Globally Optimal Reinforcement Learning with Hard Constraints

eess.SY ·2026-09-12 ·Vipul K. Sharma, Wesley A. Suttle, S. Sivaranjani
published similarity 0.71
Abstract · Accepted to the IEEE Conference on Decision and Control (CDC) 2026

We present an optimal and convergent model-free policy gradient (PG) reinforcement learning (RL) framework for controlling nonlinear dynamical systems under hard safety constraints. We first construct a class of stochastic wrapper policies centered around a deterministic controller, thereby enabling exploration in unknown environments while preserving the underlying deterministic control structure. We then define a class of parameterized safe-by-construction control policies by truncating these stochastic policies onto hard safety constraints. We next establish, via measure-theoretic arguments, that the potentially nonconvex RL objective under the truncated policy class, as well as its policy gradients, are well-defined. We then develop a model-free PG algorithm based on stochastic gradient ascent that directly searches over these truncated policies and leverage gradient dominance to establish convergence and optimality guarantees. Finally, we validate this framework through simulations on a safe quadrotor navigation problem.

0.50

Constrained Differentiable Policy Learning for Safe Quadrotor Navigation

cs.RO ·2026-10-12 ·Yan Rui Tan, Praveen Krishnapur, Zeyu Feng, Han Bao, Yew-Soon Ong +1 more
similarity 0.69
Abstract

Differentiable simulation enables efficient policy optimization through analytic gradients, yet safety is commonly incorporated as a weighted penalty, requiring manual tuning of the trade-off between task performance and constraint satisfaction. We instead formulate obstacle avoidance as an explicit constraint and train quadrotor policies using primal--dual gradient descent--ascent through a differentiable simulator. We introduce two key components to enable effective constrained training. First, we propose a bounded clearance-based constraint function that provides informative gradients around the safety boundary while preventing large violations from destabilizing primal--dual optimization. Second, we introduce log-mean-exp (LME) aggregation to smoothly emphasize critical per-step safety violations across a trajectory and training batch within a single constraint governed by one dual variable. Together, these components enable the desired safety level to be specified directly while balancing constraint satisfaction and task completion. We theoretically prove that the proposed optimization with differentiable simulation converges to feasible policy under mild assumptions. Experiments demonstrate improved safety--performance trade-offs over fixed-penalty and existing policy-learning baselines, as well as generalization to denser out-of-distribution environments. Finally, the learned policy transfers zero-shot to Gazebo/PX4 and a physical quadrotor without additional fine-tuning.

0.18

Safe Reinforcement Learning with Hard Constraints: Policy Gradient Methods with Intersample Guarantees

eess.SY math.OC ·2026-11-16 ·Vipul K. Sharma, S. Sivaranjani
similarity 0.69
Abstract

We develop a policy-gradient (PG) reinforcement learning (RL) framework with hard inter-sample control barrier function (CBF)-based safety constraints for nonlinear sampled-data systems. We first design a class of stochastic policies with support truncated to sampling-period-dependent safe control sets, that guarantee safety-by-construction at sampling instants. We then propose a model-free PG algorithm that searches over the class of safe-by-design policies and establish sufficient conditions on sampling-period such that any action sampled from safe control sets remains safe throughout the sampling interval, providing inter-sample guarantees. We further establish finite-time convergence and sample complexity guarantees for the proposed PG algorithm and characterize its global optimality under gradient dominance. Finally, we illustrate inter-sample safety guarantees on a safe quadrotor navigation environment.

0.06

CALOS: Control-Affine Lyapunov On-manifold Safety Layer for Safe Deep Reinforcement Learning for Quadrotors

cs.RO cs.AI ·2026-09-15 ·Fabrizio Cesareo, Sebastiano Mengozzi, Nicola Mimmo, Andrea Acquaviva
similarity 0.67
Abstract

Deep Reinforcement Learning has demonstrated remarkable capability in quadrotor control, yet learned policies offer no guarantee of respecting safety constraints during training or deployment. We present CALOS (Control-Affine Lyapunov On-manifold Safety), a runtime safety layer that enforces attitude constraints on a quadrotor without modifying the underlying learning algorithm. CALOS formulates four tilt-angle inequalities and a Lyapunov descent condition as a single quadratic program whose solution is the minimum-norm correction to the nominal torque output of the policy. The quadratic program is solved exactly via active-set enumeration over the three-dimensional torque space, with a computational cost low enough to enforce constraints in real time across thousands of parallel simulation environments, as required by modern massively parallel Deep Reinforcement Learning training. Evaluated on trajectory-tracking tasks in NVIDIA Isaac Lab, CALOS reduces lateral tracking error by 55-60% relative to an unconstrained Proximal Policy Optimization baseline while achieving zero attitude-constraint violations on the training trajectory. By restricting exploration to safe regions of the state space, the safety layer also accelerates training convergence and improves data efficiency without producing suboptimal policies.

0.19

Towards Safe Reinforcement Learning with Reduced Conservativeness: A Case Study on Drone Flight Control

eess.SY ·2026-08-27 ·Loizos Hadjiloizou, Michael C. Welle, Hang Yin, Danica Kragic
published similarity 0.67
Abstract · 7 pages, 6 figures

Incorporating formal methods into reinforcement learning (RL) has the potential to result in the best of both worlds, combining the robustness of formal guarantees with the adaptability and learning capabilities of RL, though careful design is needed to balance safety and exploration. In this work, we propose a framework to mitigate this loss of exploration while still allowing for the safety of the system to be ensured. Specifically, we introduce a less restrictive method that can reduce the conservativeness of formal methods by refining a disturbance model using online collected data and it evaluates the safety of a learning-based controller, using computationally efficient zonotopic reachability analysis for the safety analysis to facilitate a real-time implementation. We validate the framework in a real-world drone flight through a canyon, where the drone is subjected to unknown external disturbances and the framework is tasked with learning those disturbances online and adjusting the safety guarantees accordingly. The results show that the framework enables a less restrictive online training of learning-based controllers without compromising the safety of the system.

0.23

A Risk-Sensitive and Uncertainty-Aware Decision-Making and Control Framework for Safe and Robust Autonomous Driving

cs.RO ·2026-09-09 ·Zhuoren Li, Ran Yu, Weiqi Zhang, Ming Liu, Lu Xiong +2 more
similarity 0.66
Abstract · 14 pages, 8 figures, 5 tables

Reinforcement learning (RL) has demonstrated considerable potential for autonomous driving decision-making. However, its deployment in urban autonomous driving, particularly at highly interactive unsignalized intersections, remains challenging, as learned policies may struggle to maintain both safety and robust decision-making in complex traffic situations. Conventional safety-filtering approaches typically employ fixed conservative constraints, which may improve safety at the cost of excessive intervention and degraded traffic efficiency. To address these limitations, we propose a Risk-sensitive and Uncertainty-aware Decision-making and Control (RUDC) framework for safe and robust autonomous driving. RUDC couples risk-sensitive distributional RL with ensemble-based policy uncertainty quantification, jointly accounting for tail risks in return distributions and uncertainty in learned policies. An uncertainty-aware high-order control barrier function (HOCBF)-based safety correction mechanism adaptively adjusts constraint strictness according to policy uncertainty, while a learnable residual predictor compensates for CBF model mismatches and discretization errors. Extensive simulations at unsignalized intersections demonstrate that RUDC achieves a favorable balance among safety, efficiency, and robustness, outperforming representative safe RL baselines under both nominal and challenging OOD and long-tail scenarios while satisfying real-time requirements.

0.40

Recoverability, Not Invariance: Certified Safety for Robots at the Limits of Performance

cs.RO eess.SY ·2026-11-17 ·Thomas A. Berrueta, Joshua D. Ibrahim, Gilchrist X. Johnson, John Lathrop, Soon-Jo Chung
theory similarity 0.66
Abstract · For our supplementary video, see https://www.youtube.com/watch?v=h1kTaiRAQEI

Safety certification in robotics typically rests on forward invariance of the deployed controller, a property that demands high-fidelity models at the edge of the operating envelope, exactly where models are least reliable and robots perform best. As a result, many of the behaviors we most want robots to run, from learned policies to aggressive maneuvers at the limits of traction, cannot be certified at all. We introduce recoverability, a contraction-theoretic safety concept that resolves this tension by certifying a robot's ability to reengage a trusted recovery mode rather than the invariance of a deployed behavior. We prove that switching to the recovery mode from anywhere inside a certified tube guarantees exponential convergence, even under disturbances and model error. To decide when to switch, we derive the contraction insertion gradient, the first-order effect of a recovery insertion on the certificate, and embed it in a switching-time program in which feasibility alone guarantees safety and optimality only shapes performance. To supply the certificates, we develop sequential convex synthesis that monotonically grows the certified tube toward the exact viability kernel. We validate our approach in simulated quadrotor flight through uncertain obstacles, where it outperforms invariance-based safety filters, and in hardware deployments aboard a full-scale IndyNXT autonomous racecar, where it repeatedly recovers the car from an aggressive raceline at the limits of handling in the rain. Recoverability provides a certified exit strategy, extending formal safety guarantees to aggressive, learned, and experimental behaviors without requiring their invariance.

0.41

Safety-Focused Reinforcement Learning Using Risk-Sensitive Coupled Control

cs.LG ·2026-11-06 ·Jiabin Lin, Shana Moothedath, Vivek S. Borkar
similarity 0.65
Abstract

This paper investigates the coupled control problem within safety-focused reinforcement learning, addressing the essential challenge of achieving optimal performance while ensuring safety in systems operating within complex environments. We define safety as the requirement to prevent the system from reaching undesirable or hazardous states. We propose a coupled control approach that prioritizes safety by employing effective methods to prevent the system from entering unsafe states while optimizing the performance metric, defined as the average cost. We propose a novel problem transformation that reformulates it into a risk-sensitive problem combined with an average reward criterion. We present a two-timescale Q-learning algorithm where in the faster timescale, the algorithm solves a risk-sensitive control problem to ensure safety, while on a slower timescale, the algorithm focuses on average cost control. We provide theoretical guarantees for the convergence of our algorithms, establishing their significance in ensuring safe, optimal control in dynamic environments. We validated the performance of our approach using GridWorld and CarGoal task in Safety Gymnasium.

0.17