CS Paper Radar

Deep Reinforcement Learning with Smith-Predictor Delay Compensation

Moh Kamalul Wafi

eess.SY ·submitted 2026-10-31 06:06 UTC · v1
published

Abstract

Time delays can degrade feedback performance and destabilize a closed loop that is stable without delay. This paper studies Smith-predictor-assisted reinforcement learning for systems with constant input delays. A deep deterministic policy gradient (DDPG) actor is trained on a nominal delay-free environment and deployed with a Smith predictor using a fixed nominal delay. For linear controllers, we derive exact closed-loop maps under plant and delay mismatch and provide small-gain and $ν$-gap robustness conditions. A separate delay-independent Razumikhin condition gives a sufficient stability test for a fixed Lipschitz state-feedback policy. We also study output-error delay identification from a fixed input--output record, deriving the loss gradient and establishing projected-flow descent and local exponential convergence under exact model matching and an excitation condition. These identification results do not imply stability of an online adaptive predictor. Numerical experiments retained from the conference study compare PID, LQR, and DDPG controllers on stable and open-loop unstable first-order plants with valve stiction. The results illustrate the benefit of predictor compensation in the tested configurations and the sensitivity of standalone DDPG to delay and plant changes.

Comment
This is an extended version of the paper published in the proceedings of IEEE American Control Conference (ACC)

Closest papers

Nearest neighbours by meaning across the whole index.

Robust Data-Driven Output Feedback for Systems With Unknown Input Delays

eess.SY ·2026-10-31 ·Ngoc Tuan Dinh, Ba Huy Nguyen, Alexey Peregudin, Egor Dogadin
similarity 0.63
Abstract · 6 pages, 1 figure. Submitted jointly to IEEE Control Systems Letters (L-CSS)

In a history-feedback realization, an input delay increases memory without increasing the number of unknown physical coefficients. We use this structure to design output-feedback controllers for discrete-time systems with one input, one output, and known undelayed order, using noisy input-output data. For a known delay, we derive a necessary and sufficient linear matrix inequality for common-quadratic stabilization of all models consistent with the data. Optimizing its equation-error bound gives a robustness margin for the measured record. For an unknown constant delay in a known interval, we characterize noiseless delay ambiguity and give a sufficient condition for one controller to stabilize all models for every delay consistent with the error bound. Under stated excitation and nondegeneracy assumptions, this procedure is feasible for all sufficiently small equation errors. Numerical examples compare robustness at equal controller memory and show that certified stabilization can succeed while distinct delay models remain compatible with the same data.

0.31

Stability-aware Residual Reinforcement Learning Framework for Robotic Manipulator Disturbance Compensation

cs.RO ·2026-09-18 ·Jihong Kim, Joonhyuk Kwon, Hwa Soo Kim, TaeWon Seo, Hyung-Tae Seo
similarity 0.59
Abstract · 14 pages, 9 figures

Although conventional controllers and disturbance observers (DOBs) are the standard for precision tracking in manipulators, they suffer from parameter uncertainty, nonlinear friction, and compound disturbances. This study proposes a residual reinforcement learning DOB framework that pairs an analytical observer with an RL policy. The deterministic baseline operates within a reliable region, whereas the RL policy explicitly targets the residuals that the model cannot capture. To make this compensation disturbance-aware, an estimator network aligns the observation history with a privileged disturbance context, organizing the latent space by disturbance regime and enabling rapid adaptation across disturbance transitions. To guarantee stability, we derived and enforced a state-dependent action bound on the RL policy from an input-to-state stability (ISS) analysis such that the closed loop provably confines the tracking error to a certified envelope for arbitrary policy outputs. Experiments on a 6-DOF manipulator demonstrated consistent improvements in disturbance estimation and tracking, including a 27.8% tracking-error reduction on real hardware under zero-shot sim-to-real transfer and a 38.0% reduction under a base-vibration disturbance that was not observed during training.

0.24

Augmenting PID Control with Deep Reinforcement Learning: A Hybrid Approach to the Industrial Benchmark

cs.LG ·2026-09-18 ·Zhengyang, Gu, Joseph E. Hernandez, John Burtenshaw, Sean Scott +2 more
top venuebenchmark similarity 0.59
Abstract · Published in: 2026 7th International Conference on Artificial Intelligence, Robotics, and Control (AIRC)

As industrial processes grow in complexity, traditional Proportional-Integral-Derivative (PID) controllers are often insufficient for handling their non-linear, multi-input dynamics. We propose using advanced Deep Reinforcement Learning (DRL) to prove its advantages in these complex environments. To do this, we rely on the Industrial Benchmark (IB). The IB is a realistic simulation that tests DRL algorithms against the key challenges of industrial applications: high-dimensional state spaces, delayed effects, and conflicting multi-criterial objectives. This testbed highlights DRL's core trade-off: while its final policies can often be unstable, its unique strength is the ability to autonomously discover optimal, non-obvious policies in multi-dimensional spaces where simple controllers fail. In this paper, we propose a novel hybrid PID-RL controller that leverages DRL's discovery capability while ensuring Reliability. After developing a multi-objective reward function to make DRL viable, we use a twin-delayed deep deterministic (TD3) agent as a discovery tool to find the optimal, non-obvious settings for the IB's 'Gain' and 'Shift' parameters. By feeding these discovered parameters to a simple, tuned PID controller, our hybrid model successfully combines all three characteristics: it achieves the optimal Performance and Efficiency of the best DRL agent with the Reliability of a classical controller. This work demonstrates a practical methodology for using DRL to augment, rather than replace, trusted industrial control systems.

0.47

Asynchronous Model Predictive Control Under Model Mismatch: Stability and Performance Guarantees

eess.SY ·2026-09-08 ·Changrui Liu, Anil Alan, Shengling Shi, Bart De Schutter
similarity 0.58
Abstract · This is the first work to characterize the joint effects of model mismatch, general asynchronous feedback, and prediction horizon on stabili…

Certainty-equivalence model predictive control (CE-MPC) is widely used for its simplicity and efficiency, but theoretical guarantees under asynchronous feedback remain limited. This paper establishes stability and performance guarantees for asynchronous CE-MPC of input-constrained nonlinear systems. We first derive a nominal stability condition and competitive-ratio bound that explicitly account for inter-execution intervals without prescribing a feedback mechanism. A value-function perturbation analysis for quadratic stage costs then accommodates additive, potentially non-smooth model mismatch without constraint qualification conditions. Combining these results yields stability criteria and competitive-ratio bounds for CE-MPC under general asynchronous feedback, including event/self-triggered and multi-step MPC. The guarantees explicitly relate prediction horizon, inter-execution time, and uncertainty magnitude, quantifying performance degradation relative to an ideal infinite-horizon controller. These results clarify tradeoffs between feedback frequency, model accuracy, and horizon length, guiding asynchronous MPC design using approximate or learned models.

0.30

Velocity-Free Funnel Control of Euler-Lagrange Systems: An Artificial-Delay Approach

eess.SY ·2026-11-12 ·Chayan Kumar Paul, Pushpak Jagtap
similarity 0.58
Abstract

Classical funnel control keeps the tracking error within a prescribed, time-varying envelope by increasing the feedback gain as the error approaches the boundary. For mechanical systems, this guarantee typically relies on velocity measurements. We show that it can be retained for fully actuated Euler--Lagrange systems using only position measurements. The proposed controller replaces the velocity error with a difference quotient of the position error over a fixed delay, while preserving the classical unbounded gain. It requires no plant model, reference derivatives, observer, or filter states. The central challenge is to reconcile the averaging inherent in delayed feedback with a gain that can grow without bound near the funnel boundary. We resolve this by establishing an a priori bound on the gain along closed-loop trajectories, independent of the delay. This bound yields a computable delay threshold that guarantees tracking within the prescribed funnel and boundedness of all closed-loop signals. The threshold depends only on bounds on the inertia, forces, and reference. Simulations demonstrate tracking accuracy comparable to full-state funnel control and tolerance of delays substantially longer than certified. With quantized position measurements, the controller also requires substantially less torque than an existing derivative-free funnel controller.

0.30

Making Koopman Control Guarantees Hold on Robots

cs.RO cs.AIeess.SY ·2026-11-16 ·Abulikemu Abuduweili, Changliu Liu
similarity 0.58
Abstract · https://walleclipse.github.io/tik-koopman/

A Koopman controller can stabilize its learned latent-space linear model yet fail to stabilize the nonlinear plant. The learned model propagates predictions in latent space, while the deployed controller repeatedly re-encodes the measured physical state. We show that nominal closed-loop stability transfers to the nonlinear plant when the learned and true closed-loop dynamics agree to first order on the embedding's tangent space at the target. We derive a robustness margin for approximate matching and show that a stable open-loop latent model cannot exactly match an unstable equilibrium. We introduce Tangent-Invariant Koopman (TIK), which estimates a local plant linearization from offline transitions and enforces first-order matching to this estimate. TIK recovers the fitted LQR gain at the target, while a bilinear predictor captures state-dependent actuation during large maneuvers. Across four simulated unstable benchmarks, all 40 Standard Koopman controllers have stable latent closed loops, but their deployed nonlinear systems are locally unstable. In contrast, all 40 TIK controllers locally stabilize their respective nonlinear plants. An offline stability test using empirical curvature estimates identifies stable TIK controllers on two cart-pole systems, but is inconclusive under high noise. Under explicit model-error and curvature bounds, an invariant terminal set guarantees constraint satisfaction after entry under local feedback. TIK achieves 88\% constrained cart-pole swing-up success with a 0.25 ms planning budget and controls simulated seven-joint arms within 1-5 ms per step.

0.35

Delay-Aware Reinforcement Learning for O-RAN Control under Stochastic Reward Feedback

cs.MA cs.LG ·2026-11-14 ·Xingqi Wu, Junaid Farooq, Tao Li, Juntao Chen, Ying Wang
published similarity 0.56
Abstract

Reinforcement learning (RL) is increasingly employed for real-time control in Open Radio Access Networks (O-RANs). However, the asynchronous nature of O-RAN feedback introduces stochastic delays between actions and their observable rewards, leading to misaligned credit assignment and unstable training. This letter formulates the O-RAN control problem as a Markov decision process (MDP) with delayed and stochastic rewards, where the true reward corresponding to an action is received after an unknown delay. To address this, we propose a delay-aware deep RL framework that integrates a long short-term memory (LSTM)-based delay predictor and a temporal reward realignment module within an off-policy learning pipeline. Using empirical delay traces collected from the Open AI Cellular (OAIC) testbed, we demonstrate that the proposed framework achieves faster convergence and significantly reduces latency violations and packet drops compared to conventional RL methods that ignore feedback delay. The results confirm that reward-delay compensation is critical for reliable AI-driven O-RAN control.

0.60

RAPAC-DP: Response-Aligned Pending-Action Compensation for Diffusion Policies under Delayed Execution

cs.RO ·2026-08-16 ·Tao Wang, Wei Wang, Jianhui Wang, ZhengYang Ye, Qi Wang +2 more
similarity 0.56
Abstract

Cloud-side inference expands the computational resources available to imitation-learning policies, but communication and inference latencies create a mismatch between prediction and execution. We propose RAPAC-DP, a response-aligned pending-action compensation framework for diffusion- and flow-based action generators. By conditioning on actions committed during inference, RAPAC-DP generates a full-horizon action chunk aligned with the scheduled handover time. A lightweight pending-action encoder and LoRA adapters learn this compensation while keeping the pretrained base policy frozen. We construct delay-conditioned training samples from delay-free demonstrations, requiring neither an explicit dynamics model nor additional data collected under delayed execution. In simulation benchmarks at the largest evaluated delays, RAPAC-DP retains 81.4\% of its overall delay-free performance on Kinetix and achieves a mean success rate of 63.3\% across three RoboMimic tasks. Across 450 physical trials on a Franka Panda manipulator, RAPAC-DP increases the high-delay mean success rate from 28.9\% for RTC to 60.0\% across static and conveyor-based pick-and-place tasks.

0.04