CS Paper Radar

Learning to Tune with Biased Gradients: Iterative Feedback Tuning of Model-Free Control on Throttle Valves

Félix Martinez-Jiménez, Sander Miller-Murphy, Tobie Ondoa-Essomba, Emmanuel Witrant

cs.LG cs.AI ·submitted 2026-11-04 11:01 UTC · v1

Abstract

Iterative feedback tuning (IFT) learns the parameters of a fixed-structure controller by stochastic gradient descent. At each iteration, two closed-loop experiments are run on the plant: the first one measures the tracking error, and the second one, driven by this error, provides the sensitivities from which the gradient of the cost is estimated. This estimate is unbiased for linear plants only. We study the biased case on automotive electronic throttle valves, whose friction and stiction vary between valves and runs, controlled by an intelligent proportional (model-free) controller, which is an incremental PI controller. We show that the mismatch of the controller state at the start of the two experiments, which stiction makes unavoidable, adds an error inversely proportional to the amplitude of the second experiment. On digital twins of ten valves, the IFT gradient is a descent direction in only 69\% of the low-cost region, friction being the cause, and all update rules plateau and then drift. On two hardware valves, the best update rule depends on the noise structure of the gradient. On the first valve, where one gradient component is buried in noise, Adam outperforms Gauss--Newton, its normalisation and its momentum each improving one parameter. On the second valve, where both components are consistent, the same normalisation amplifies a direction that Gauss--Newton rightly discounts, and Adam drifts to a controller worse than the initial one. The twins reproduce the mechanism but not this ranking.

Closest papers

Nearest neighbours by meaning across the whole index.

Augmenting PID Control with Deep Reinforcement Learning: A Hybrid Approach to the Industrial Benchmark

cs.LG ·2026-09-18 ·Zhengyang, Gu, Joseph E. Hernandez, John Burtenshaw, Sean Scott +2 more
top venuebenchmark similarity 0.58
Abstract · Published in: 2026 7th International Conference on Artificial Intelligence, Robotics, and Control (AIRC)

As industrial processes grow in complexity, traditional Proportional-Integral-Derivative (PID) controllers are often insufficient for handling their non-linear, multi-input dynamics. We propose using advanced Deep Reinforcement Learning (DRL) to prove its advantages in these complex environments. To do this, we rely on the Industrial Benchmark (IB). The IB is a realistic simulation that tests DRL algorithms against the key challenges of industrial applications: high-dimensional state spaces, delayed effects, and conflicting multi-criterial objectives. This testbed highlights DRL's core trade-off: while its final policies can often be unstable, its unique strength is the ability to autonomously discover optimal, non-obvious policies in multi-dimensional spaces where simple controllers fail. In this paper, we propose a novel hybrid PID-RL controller that leverages DRL's discovery capability while ensuring Reliability. After developing a multi-objective reward function to make DRL viable, we use a twin-delayed deep deterministic (TD3) agent as a discovery tool to find the optimal, non-obvious settings for the IB's 'Gain' and 'Shift' parameters. By feeding these discovered parameters to a simple, tuned PID controller, our hybrid model successfully combines all three characteristics: it achieves the optimal Performance and Efficiency of the best DRL agent with the Reliability of a classical controller. This work demonstrates a practical methodology for using DRL to augment, rather than replace, trusted industrial control systems.

0.47

On Convergence of Gradient Temporal-Difference Learning with Regularized Corrections and Eligibility Traces

cs.LG math.OC ·2026-11-04 ·Jingyi Jin, Martha White, Hao Wang
similarity 0.57
Abstract · 66 pages, 1 figure. Submitted to the Journal of Machine Learning Research

Temporal-difference (TD) learning underlies many prediction and control algorithms, but can diverge when combined with off-policy learning, bootstrapping, and function approximation. TD with Regularized Corrections (TDRC) is theoretically sound and practically simple, but existing convergence theory covers only the one-step case under i.i.d. sampling, while the Huber Bellman-error formulation has not been extended to TDRC or eligibility traces. In this work, we establish convergence of constrained TDRC($λ$) with diminishing stepsizes and non-i.i.d. off-policy data, for state-dependent and history-dependent $λ$-parameters. We also propose TDRC($λ$)-Huber, which preserves the TDRC($λ$) fixed point while reducing sensitivity to large Bellman errors. For both algorithms, we apply an ODE-based weak-convergence method from stochastic approximation theory. For the nonlinear mean dynamics of TDRC($λ$)-Huber, we give an integral small-gain condition that guarantees uniqueness and global asymptotic stability of the target equilibrium. We then establish convergence of the constrained algorithm, derive computable sufficient conditions for convergence, and identify a regularization-based convergence mechanism shared by linear TDRC($λ$) and TDRC($λ$)-Huber. When the mean TD matrix is positive stable, both constrained algorithms converge under sufficiently strong regularization without requiring positive definiteness.

0.23

Policy-Gradient Transfer under a Shared State Estimator

eess.SY ·2026-10-15 ·Dongming Wang, Wei Ren
similarity 0.56
Abstract

This paper considers the problem of policy-gradient transfer from a simulator to a physical plant when both are controlled through the same fixed state estimator. The plant and the simulator are partially observed nonlinear systems with additive Gaussian noise of common known covariance whose dynamics and observation models differ, and the controller is a squashed Gaussian policy with uniformly bounded latent mean and first two parameter derivatives. We derive a bound on the difference between the plant and simulator policy gradients that is linear in the noise-weighted model error and requires no estimator-accuracy guarantee, because the shared estimator is a common post-processing of the measurements and the data-processing inequality compares the trajectory laws without an estimation-error term. Based on this bound, we derive an occupancy-weighted refinement, a sufficient condition for the simulator gradient to be an ascent direction for the plant, and a bound on the expected squared plant gradient at a random iterate of a finite simulator-training run with optimization, truncation and model-error terms. Simulation results on a scalar system and a pendulum are presented to validate the theoretical results.

0.17

Rethinking MoE Load Balancing as Feedback Control

cs.LG ·2026-11-06 ·Yikun Hou, Alp Yurtsever
similarity 0.55
Abstract

Mixture-of-Experts (MoE) training is often not stationary: the data mixture changes at curriculum boundaries, when a model is continually pretrained into a new domain, or when the corpus is refreshed. Routing is learned against the current mixture, so each such change disrupts the per-expert load the router had converged to, and existing balancing methods have only been evaluated on stationary data. We rethink load balancing as a feedback control problem, in which the deviation of each expert's load from its fair share is the control error and the per-expert routing bias is the control input, and define a family of controllers, the PID-Balancer, from the proportional, integral, and derivative terms of classical PID control. Two widely used methods belong to this family: the auxiliary balancing loss behaves like a proportional controller, and the loss-free bias update is an integral controller acting on the sign of the error. Reading the router as a static map with a one-step delay, we show that the integral term is the only one that both removes persistent imbalance and averages out the step-to-step fluctuation of the measured load, whereas the proportional and derivative terms pass or amplify that fluctuation into the routing bias. We therefore adopt PID (I), the pure-integral member of the family with a single gain. Two settings are evaluated: (i) training an OLMoE backbone from scratch with 16 to 64 experts per layer, through a mid-run mixture switch, and (ii) continual pretraining of the released OLMoE-1B-7B on a new corpus. PID (I) holds balance before the shift and restores it more tightly afterwards than either the auxiliary loss or loss-free balancing, at comparable perplexity.

0.34

Two-Timescale Reinforcement Learning for Real-Time Optimization and Economic NMPC: Experimental Validation

eess.SY ·2026-09-28 ·Saket Adhau, Jose Matias, Sebastien Gros, Sigurd Skogestad
similarity 0.55
Abstract · 12 pages, submitted to Journal of Process Control

We propose a reinforcement learning (RL) framework that tunes both Real-Time Optimization (RTO) and Economic Nonlinear Model Predictive Control (ENMPC) to address plant--model mismatch in process systems. Drawing on modifier-adaptation concepts, the method parameterizes the dynamic model, stage costs, constraints, and RTO modifiers, and uses Q-learning to adjust these parameters at two timescales: a fast update for the ENMPC layer and a slow update for the RTO layer. The framework is experimentally validated on a laboratory rig emulating a three-well subsea oil-production network. Using plant measurement data, the proposed RTO-RLMPC scheme achieves 8.6% higher economic profit than nominal ENMPC, preserves input feasibility of the deployed ENMPC policy and empirically satisfies the path constraints under disturbances and measurement noise, and drives the learned model parameters toward reference plant values, independently identified from experimental data, along the directions that most affect the economic objective. This work provides one of the first experimental demonstrations of RL-tuned ENMPC integrated with RTO.

0.24

Certified Neural Approximation of Robust Nonlinear Model Predictive Control

eess.SY cs.LG ·2026-10-21 ·Hendrik Alsmeier, Maik Pfefferkorn, Rolf Findeisen
similarity 0.55
Abstract · 12-page article plus 4-page supplement, 3 figures

Model predictive control is a high-performance control method, but it incurs comparatively high online computational costs across many control tasks and is challenging to realize in low-level hardware settings. To address this, we present a framework for generating neural network controllers that provably inherit the safety and stability guarantees of a robust model predictive policy. In contrast to approaches based on post-verification or additional online safety filters, our method provides constructive, a priori design conditions that ensure preservation of recursive feasibility and stability before deployment. We treat the approximation error as a bounded input disturbance, which is addressed in the MPC design. This allows the neural network controller to directly inherit robustness from the underlying model predictive controller, which enables the transfer of guarantees to the approximation. The design process is an offline, constructive synthesis pipeline and can be implemented once Lipschitz bounds are specified. We further demonstrate in simulation that the learned controller reproduces the closed-loop behavior of the original controller while reducing online computation.

0.11

Adaptive Control for General Nonlinear Stochastic Systems

eess.SY math.OC ·2026-10-26 ·Haizheng Li
similarity 0.55
Abstract

This paper extends the least-squares and certainty-equivalence minimum-variance framework of classical linear self-tuning regulators to a broad class of nonlinearly parameterized systems with compact parameter spaces, including commonly used deep neural network models. First, historical nonlinear least-squares fitting guarantees are converted into online one-step prediction bounds under a general modulus of continuity, with explicit regret rates under parameter Lipschitz continuity and finite-moment noise. Second, approximate NLS estimation combined with certainty-equivalence control achieves closed-loop stability and asymptotically optimal tracking under mild conditions without requiring excitation or parameter identifiability. Explicit closed-loop regret and tracking error rates are established in the parameter-Lipschitz case, and closed-loop examples show that the regret rates cannot, in general, be improved for this regulator. Finally, robustness to small unmodeled dynamics is established by showing that closed-loop stability is preserved and the tracking error bounds comprise a vanishing term and a residual term determined by the model discrepancy.

0.28

Generalization as a robust performance property of learning-enabled dynamical systems

eess.SY cs.LGmath.OC ·2026-08-31 ·Filippo Fabiani
similarity 0.54
Abstract

By focusing on algorithmic stability as a means of establishing out-of-sample bounds, we provide a system-theoretic interpretation of generalization in learning-enabled dynamical systems arising in data-driven optimization and feedback control approximation. Given two neighboring datasets, we specifically model sample replacement as an exogenous disturbance acting on a sensitivity system, while the incremental behavior of the data-dependent operator is encoded through an integral quadratic constraint. By relying on dissipativity arguments, we establish a matrix inequality-based certificate and a uniform stability bound that separates the one-sample sensitivity of the learned operator, and an algorithm-dependent dynamical gain. The latter can then be optimized, offering a tractable tool for certifying and comparing generalization capabilities of learning dynamics. We show that our results recover classical ones for gradient descent, apply naturally to momentum-based methods such as heavy-ball and Nesterov acceleration, and extend to data-driven control.

0.20