CS Paper Radar

Generalizable Output Feedback Optimal Control via Transformers Without Explicit State Estimation

Maitham F. AL-Sunni, Turki Bin Mohaya, John M. Dolan, Peter Seiler

eess.SY cs.RO ·submitted 2026-11-09 03:23 UTC · v1

Abstract

Classical optimal control is used to design a separate controller for each plant. This paper asks a different question. Can one learned policy serve a whole family of partially observed, noisy linear systems? We show that one transformer can. The policy is trained to imitate optimal LQG control on a heterogeneous family of Linear Time-Invariant (LTI) systems with Multiple Inputs and Multiple Outputs (MIMO), whose dimensions, cost matrices, and noise levels all vary. A shared representation with system-wise standardization, zero-padding, masking, and an encoding of the cost and noise covariances lets one set of weights control every member of the family. At run time, a short window of noisy measurements is all the policy needs to produce a control action, with no plant matrices and no state estimator. We characterize the relation between the optimal LQG policy (represented as a dynamic state-space model) and a static transformer acting on a finite window of measurements. We then express the closed-loop excess cost of any causal policy as the energy of its deviation from the optimal action. Finally, we evaluate the approach on 28 simulated systems. The trained policy reaches the optimal cost to within a median gap of $0.4\%$ for seen systems. For unseen systems, a light fine-tuning step brings the median gap to $1.0\%$.

Comment
7 pages

Closest papers

Nearest neighbours by meaning across the whole index.

Generalizable Optimal Control with Transformers: Closed-Loop Certification and Near-Optimality Guarantees

eess.SY ·2026-09-19 ·Turki Bin Mohaya, Maitham F. AL-Sunni, John M. Dolan, Peter Seiler
benchmark similarity 0.67
Abstract

This letter develops closed-loop performance certificates for a transformer-based feedback policy. The policy is trained to imitate optimal Linear Quadratic Regulator (LQR) control across a family of heterogeneous Multiple-Input, Multiple-Output (MIMO) Linear Time-Invariant (LTI) systems. First, we establish a finite-sample excess-risk bound for the imitation loss minimized during training. Second, for each fixed problem instance, we derive regional closed-loop guarantees consisting of a forward-invariant operating region and a worst-case bound on deviation from the optimal rollout. Our main result is a probabilistic certificate for finite-horizon closed-loop near-optimality. Using an exact LQR cost identity, we express excess cost as a measurable per-rollout statistic and use independent calibration and validation rollouts to obtain a high-confidence bound on its violation probability. We evaluate the certificate on $28$ benchmark systems. This uses the base policy on seen systems and system-specific fine-tuned copies on unseen systems, with each rollout drawing the plant, cost, and initial condition from the corresponding certification distribution. All per-system certificates have violation probabilities below $3.1\%$, each at $95\%$ confidence; twenty systems certify suboptimality below $10\%$, with the tightest threshold equal to $4.8\times10^{-6}$.

0.44

Policy-Gradient Transfer under a Shared State Estimator

eess.SY ·2026-10-15 ·Dongming Wang, Wei Ren
similarity 0.61
Abstract

This paper considers the problem of policy-gradient transfer from a simulator to a physical plant when both are controlled through the same fixed state estimator. The plant and the simulator are partially observed nonlinear systems with additive Gaussian noise of common known covariance whose dynamics and observation models differ, and the controller is a squashed Gaussian policy with uniformly bounded latent mean and first two parameter derivatives. We derive a bound on the difference between the plant and simulator policy gradients that is linear in the noise-weighted model error and requires no estimator-accuracy guarantee, because the shared estimator is a common post-processing of the measurements and the data-processing inequality compares the trajectory laws without an estimation-error term. Based on this bound, we derive an occupancy-weighted refinement, a sufficient condition for the simulator gradient to be an ascent direction for the plant, and a bound on the expected squared plant gradient at a random iterate of a finite simulator-training run with optimization, truncation and model-error terms. Simulation results on a scalar system and a pendulum are presented to validate the theoretical results.

0.17

Reinforcement Learning-Based Output Feedback LQR for Continuous-Time MIMO Systems

eess.SY ·2026-08-12 ·Qihua Chen, Mingxiang Liu, Lili Wang, Zhiyun Lin, Minyue Fu
similarity 0.58
Abstract · 22 pages, 3 figures, 1 table

This article studies model-free output feedback linear quadratic regulation (LQR) for continuous-time linear systems with an $n$-dimensional state, an $m$-dimensional input, and a $p$-dimensional output, using filtered input--output data. Since the system state is unavailable, existing methods rely on dynamic filters to parameterize the hidden state using measurable input--output signals. However, the intrinsic dimension of the resulting filter-based parametrization can be smaller than the dimension of the complete filtered vector, and this deterministic redundancy can make the Bellman regressions rank deficient. We characterize this intrinsic dimension and show that the conventional filtered vector contains only $2n$ independent components for single-input multi-output (SIMO) systems and $n(m+1)$ independent components for general multi-input multi-output (MIMO) systems. Based on this characterization, a reduced filtered vector is extracted directly from data and used to develop reduced model-free output feedback policy iteration and value iteration equations, eliminating the redundant directions and decreasing the number of unknown parameters while retaining a fully input--output data-based implementation. A numerical example illustrates the rank reduction and the effectiveness of the learned controller.

0.10

From Policy Evaluation to Off-Policy Control: Provable Emergence of In-Context Q-Learning

cs.LG cs.AI ·2026-10-13 ·Yingjun Mo, Yajie Zhao, Fang Kong
theory similarity 0.57
Abstract

Transformers can implement reinforcement-learning algorithms in context, yet how off-policy control emerges within Transformers remains poorly understood. We study Q-learning, a foundational model-free control algorithm that combines off-policy bootstrapping with greedy policy improvement. We prove that a two-layer attention-only Transformer can implement a finite-temperature form of Q-learning, where the temperature $τ>0$ controls the approximation to greedy action selection, recovering window Q-learning as $τ\to0$. We train the structured Transformer on randomly generated MDP tasks using output-only supervision. Experiments show that the theoretically predicted parameter structure emerges during training, including the internal greedy action-selection mechanism, and that the trained Transformer achieves performance comparable to Q-learning on unseen tasks. Our theoretical analysis further shows that the equivalent realizations form a solution manifold and that, under suitable task-diversity conditions, the training iterates converge locally to this manifold at an exponential rate. For the random-task model analyzed here, the local worst-case number of gradient steps needed to reduce the distance to the manifold by a factor $ε$ is $Θ\left(τ^{-5}\log(1/\varepsilon)\right)$, quantifying the increasing optimization cost as the learned update approaches window Q-learning.

0.14

Provable Transformer Context Compression for In-Context Reinforcement Learning

cs.LG cs.AI ·2026-10-14 ·Yajie Zhao, Yingjun Mo, Zhiyong Wang, Bo Tang, Chao Xu +1 more
similarity 0.57
Abstract

Transformers can perform in-context reinforcement learning without parameter updates. However, existing theoretical constructions often require context sizes that grow with the number of sampled transitions, leading to increasing memory costs in large-scale problems. This raises a fundamental scalability question: how much context is intrinsically necessary to carry out such learning? We answer this question for fixed-policy TD evaluation. We show that, with a shared linear decoder, the minimum context dimension required to exactly reproduce the first $L$ TD iterates is the Krylov dimension induced by the sampled transitions. When the available context dimension $m$ falls below this threshold, exact reproduction is no longer possible. In this regime, we characterize the optimal approximation error and show that it decays as $Θ(γ^m\max\{1,1/(m(1-γ))\})$ where $γ$ is the discounting factor. Moving from representation complexity to actual memory cost, we characterize the optimal retained memory for exact continuous representations as $Θ(nk_L)$ and show experimentally that on a sampled task whose exact rank is full but whose numerical rank is small, a context accurate to $10^{-2}$ is $88.7\%$ smaller than the sparse sufficient statistics of TD. Finally, we establish the learnability of the shared executor through pretraining by proving global convergence of an auxiliary objective over effective attention parameters under suitable task diversity. Experiments across diverse TD evaluation problems validate parameter recovery and the predicted context--accuracy tradeoff.

0.13

When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?

cs.LG math.OCmath.PR ·2026-10-29 ·Kağan Akman, Jake Feldman Starosta, Naci Saldi, Serdar Yüksel
similarity 0.55
Abstract · 62 pages, 12 figures

There exists little mathematical theory explaining why Transformer architectures are empirically successful even for non-Markovian data. To this end, this paper studies next-token prediction via stochastic optimal control theoretically trained Transformers for such sources, where we consider either stationary and ergodic non-Markovian sources or those which are hidden Markov models of some finite order (possibly non-stationary). In either case, we first obtain error bounds via finite models with finite window approximations. For stationary ergodic sources, the resulting error is bounded by an entropy gap; for $N$th order Markovian sources, it decays at an exponential rate $α^{\lfloor M/N\rfloor}$, where $M$ is the window length and $α$ is determined by filter stability (and the Dobrushin coefficients of the transition and measurement kernels). We then construct a Transformer of depth $M+1$, one head per layer, realizing the truncated filter, and we optimally train it with stochastic control developed in earlier work, which rigorously returns a near optimal Transformer. In particular, we show that for a non-Markovian source, either under stationarity and ergodicity for general models or under filter stability for hidden Markov models, an optimally trained Transformer has arbitrarily small performance error with increasing context length, data, and approximation refinement. Numerical studies accompany the theory.

0.24

DER Allocation without Load Prediction via Reinforcement Learning

eess.SY ·2026-08-17 ·Abed AlRahman Al Makdah, Aravind Ramana, Shaofeng Zou, Oliver Kosut, Lalitha Sankar
published similarity 0.55
Abstract · 5 pages. Presented at the 2026 IEEE Power & Energy Society General Meeting (PES GM)

The growing variability of renewable generation increases the need for fast and flexible grid-balancing mechanisms. Existing frameworks for distributed energy resource aggregations (DERAs) rely on short-term forecasts of net demand, making their performance highly sensitive to prediction errors. In this paper we present a forecast-free reinforcement learning (RL) framework for DERA allocation that learns optimal policies directly from operational data. We model the DERA dynamics as a deterministic linear system and the exogenous net load as a feature-based linear Markov process, capturing short-range temporal dependencies without explicit forecasting. We derive a closed-form expression for the optimal policy, which is learned through a least-squares value iteration (LSVI) algorithm using data collected across episodes. The proposed framework preserves the interpretability and constraint satisfaction of DER model while adapting to stochastic demand variations through data-driven updates. Numerical experiments on real California Independent System Operator (CAISO) net-demand data demonstrate that the learned controller achieves high tracking accuracy and stable regulation across heterogeneous DER aggregators without requiring any demand prediction.

0.37

A Reinforcement Learning Approach to Space-Filling Input Design for Nonlinear System Identification

eess.SY ·2026-11-13 ·Charly Witmeur, Gonçalo G. Cruz, Maarten Schoukens, Mark C. Runacres, Jan Decuyper
benchmark similarity 0.54
Abstract

In this paper, we propose a reinforcement learning (RL) approach to space-filling input design for nonlinear system identification, in which an agent learns a control policy that actively drives the system to explore its state space. Classical input design, developed mainly for linear models, minimises the covariance of the estimated parameters. For nonlinear black-box models, however, the dominant error is the bias induced by the model structure over the region of interest, which motivates space-filling strategies that seek to cover the system state space. Existing space-filling approaches rely on predefined input signal classes and require solving explicit optimisation problems. Unlike these methods, the proposed approach requires no input class to be selected and no frequency band to be tuned for the system at hand, satisfies an amplitude bound by construction, and adapts to varying initial conditions through its observation-based policy without re-optimisation. The learned excitation signals exhibit physically meaningful behaviours, such as resonance excitation and amplitude modulation. The approach is evaluated against tuned swept sine and crest-factor-optimised multisine excitations, and against a state-of-the-art space-filling multisine design, on two nonlinear benchmark systems: a mass-spring-damper with nonlinear stiffness and a Van der Pol oscillator. Under the same amplitude bound, the learned policy achieves the best overall space-filling performance on both systems and distributes its samples more uniformly than the state-of-the-art design.

0.23