CS Paper Radar

Dynamic Programming for A Linear-Gaussian Dual Control Problem

Ying Wang, Yuchao Li, Håkan Hjalmarsson

eess.SY math.OC ·submitted 2026-10-13 16:05 UTC · v1

Abstract

We consider an infinite-horizon discounted dynamic programming (DP) problem for a countable-innovation belief-state model. The model is a surrogate for a linear-Gaussian dual control problem involving an unknown parameter vector and Gaussian observation noise. The surrogate preserves the first two moments of the Gaussian innovation and the resulting posterior second-moment structure. We allow unbounded belief and control spaces and nonnegative stage costs with quadratic growth in the posterior mean and covariance. Using a posterior-moment weight, we show that the Bellman operator is a contraction in a weighted sup-norm space. We characterize the optimal cost as the unique solution of the Bellman equation in this space, establish the existence of optimal stationary policies. We further prove weighted sup-norm convergence of value iteration, policy iteration, and optimistic policy iteration under vanishing approximation errors. The convergence results under vanishing approximation errors rely on general weighted sup-norm contraction properties and are applicable beyond the particular dual control problem considered here.

Closest papers

Nearest neighbours by meaning across the whole index.

Dual Effect and Certainty Equivalence Revisited for a Linear-Gaussian Dual Control Problem

eess.SY math.OC ·2026-10-21 ·Ying Wang, Yue Ju, Håkan Hjalmarsson
similarity 0.69
Abstract

This paper revisits the relation between dual effect and certainty equivalence (CE) through a scalar linear-Gaussian model. In dual control, the current input affects not only the immediate cost but also the information available for future decisions, and hence the future cost. Dynamic programming (DP) provides a natural framework for formulating dual control problems. At each step of the Bellman recursion, an optimal input minimizes the sum of the stage cost and the expected cost-to-go. We restrict attention to stage costs minimized by the CE input. However, CE may not minimize the Bellman objective because of the input-dependent expected cost-to-go. Classical no-dualeffect (NDE) conditions guarantee CE optimality in certain problem classes by requiring the conditional distribution of the estimation error to be independent of past controls. In the present model, we observe that CE can remain optimal at the level of a single Bellman recursion even when the classical NDE conditions fail, provided that the expected cost-to-go is independent of the current input. Motivated by this observation, we introduce a notion of NDE formulated directly at the level of the Bellman objective, which we call Bellman-side NDE. Under some mild regularity conditions, we show that Bellman-side NDE can be characterized by a heat equation. This characterization identifies the class of Bellman-side NDE functions, and further yields a decomposition of a generic cost-to-go function into a Bellman-side NDE component and a dual-effect component.

0.21

Vector Bellman Theory for Multichain Robust Average-Reward Markov Decision Processes

cs.LG ·2026-09-23 ·Yue Wang, George Atia
similarity 0.63
Abstract · preprint, work in progress

Robust average-reward Markov decision processes provide a fundamental framework for long-term performance optimization under uncertainty, and can have optimal long-run rewards that depend on the initial state. This state dependence requires a vector Bellman theory that accounts for both recurrent-class rewards and transition uncertainty. We develop such a theory for finite models with compact, post-action $(s,a)$-rectangular ambiguity. A gain-first, bias-second optimization principle yields a coupled vector gain-bias system, and every finite solution identifies the optimal robust gain and supplies stationary saddle strategies against history-dependent opponents, simultaneously from all initial states. We further characterize solvability through stationary gain conditions and a uniform bound on canonical transient corrections, and give sufficient conditions that permit distinct recurrent-class gains. The certificates also yield asymptotically affine trajectories of the robust Bellman operator, based on which we design a robust approximately shifted Halpern planning algorithm. Under finite Bellman solvability, the gain estimates and Bellman displacements converge to the optimal gain vector, and every extracted greedy controller is average-optimal after a finite, instance-dependent budget. These results thus connect finite Bellman certificates to undiscounted planning for state-dependent robust average rewards, providing theoretical understandings.

0.27

Belief-Informed Hybrid Control with Almost-Sure Target-Set Convergence

eess.SY cs.ROmath.OC ·2026-10-06 ·Clinton Enwerem, Saleh Kemal, John S. Baras, Calin Belta
theorycode/data similarity 0.62
Abstract · 8 pages, 5 figures, 6 tables. Project page: https://clintonenwerem.com/belief-hybrid-control/

Controlling a hybrid system to a target set under parameter uncertainty can require informative actions that temporarily drive the system state away from the specified set. We propose a belief-informed dual-control algorithm that combines belief-space receding-horizon selection with an expected-decrease constraint on a nonnegative target-set progress function. To accommodate exploratory deviations, a scalar controller state bounds the constraint's cumulative slack. Our algorithm's two-step lookahead selection uses predicted observations to update the parameter belief before evaluating the subsequent action's admissibility. Under distance-comparison bounds, correct conditional prediction, and recursive feasibility, we prove almost-sure target-set convergence at decision times and bound both the sum of expected progress-function values and the expected neighborhood-entry time. Upper confidence bounds extend these convergence guarantees to bounded model samples with summable error probabilities. In planar regulation with an unknown control direction, we verify recursive feasibility: the proposed two-step selection reduces the Euclidean state norm below 0.01 within 11 decisions for either sign, whereas a myopic one-step selection loses admissibility immediately after zero input. In a simulated bimanual assembly task, our algorithm's one-step implementation uses clearance feedback to complete the assembly under nominal friction, yielding a 96.4% lower infinity-norm relative-position error than the reference-tracking baseline at the method's completion time. At lower friction, its posterior conditioning successfully detects an empty admissible set, whereas a fixed-prior alternative admits an action that violates the conditional decrease constraint. Project page: https://clintonenwerem.com/belief-hybrid-control/.

0.37

Minimax optimal dual control of positive systems: an exact solution for scalar input-sign uncertainty

eess.SY math.OC ·2026-08-15 ·Fethi Bencherki, Tomas J. Meijer, Anders Rantzer
similarity 0.61
Abstract

While recent advances in minimax dual control have led to exact solutions for uncertain general linear time-invariant systems as well as (sub)optimal dual controllers, corresponding results for linear positive systems are still lacking. This paper aims to fill this gap and thereby pave the way toward scalable dual control algorithms. We study the general minimax optimal dual control problem for positive linear systems with unknown dynamics and reformulate it as a standard zero-sum dynamic game. By allowing randomized control inputs, we solve the corresponding Bellman equation exactly for the scalar case with sign uncertainty in the input. This yields an implicit dual control policy that is optimal both in terms of cost and $\ell_1$-gain. The optimal dual policy uses exploration in a specific region of the hyperstate space to conduct optimal probing. Outside this exploration regime, the controller reduces to a deterministic certainty equivalence policy, indicating that sufficient information has been obtained to identify the correct input direction. In addition, these results allow us to analyze fundamental limitations of minimax dual control for positive systems and provide a foundation for more general dual control problems for positive systems for future work.

0.04

Fundamental Limits of Dynamic Tracking and Communication: A Stationary Policy Formulation

cs.IT ·2026-09-28 ·Mohammadreza Bakhshizadeh Mohajer, Luca Barletta, Alex Dytso, Giuseppe Caire
similarity 0.60
Abstract · 17 pages, 5 figures, submitted to IEEE Transactions on Information Theory

We study a dynamic integrated sensing-and-communication problem in which a multi-antenna transmitter communicates with a receiver while simultaneously tracking a moving target governed by a stable Gauss--Markov model. We formulate the sensing--communication trade-off as a rate--distortion problem, where the communication rate is defined through the Verdú--Han information spectrum and the sensing distortion is measured by the long-run minimum mean square tracking error in angle and distance. We construct a tractable outer region by lower bounding this distortion through the posterior Cramér--Rao bound (PCRB). The control problem is partially observed because the target state is not directly available to the transmitter. We therefore introduce a belief-augmented information state and show, under the stated regularity assumptions, that stationary randomized Markov policies suffice for the covariance-based scalarized problem used in the outer bound. We then derive, under the stated large-block conditions, a Gaussian covariance-control outer bound by showing that, in the large-block regime, the sensing belief transition becomes asymptotically covariance sufficient; consequently, Gaussian signaling with the same stationary covariance policy preserves the stationary PCRB asymptotically while maximizing the communication reward for each transmit covariance. Numerical experiments compare an extended Kalman filter-based achievable frontier with a PCRB-based Gaussian covariance-control outer frontier.

0.21

Control and Estimation Co-Design via Envelope-Theorem Gradients

eess.SY math.OC ·2026-09-28 ·Mohammad S. Ramadan, Philip Dinenis, Mihai Anitescu
theory similarity 0.60
Abstract

Instead of the sequential plant--control--estimator design pipeline, which is in general not optimal, we propose the Control and Estimation Co-Design (ContEst) framework, which poses the entire system design as a single two-stage problem solved by first-order methods. A key enabler is that the gradient of each inner control/estimation optimal cost with respect to the design parameters is available from the dual variables the inner solver already returns, by the envelope theorem: exactly under mild regularity (a unique inner minimizer), and as a subgradient under more relaxed conditions. That is, no differentiation through the optimizer or its optimality conditions is needed. We first present the framework in a linear quadratic Gaussian (LQG) regime, where the control and estimation costs become two semidefinite programs (SDPs) whose design gradients are read from the duals of the Lyapunov constraints. We extend the results to robust ($H_\infty$) formulation to cover worst-case control and filtering problems. We also present an extension to constrained and nonlinear systems, using an extended Kalman filter (eKF) and a local linearization of the information-state dynamics to yield an approximate but convex model predictive control (MPC) problem whose design gradients are read directly from the costates. We apply our methods to a variety of realistic design examples from different fields and report significant design improvements.

0.18

Learning to Control Coupled-Dynamics Environments with Joint Markov Decision Processes

cs.LG math.OC ·2026-08-24 ·Ege C. Kaya, Aliasghar Pourghani, Mahsa Ghasemi, Vijay Gupta, Abolfazl Hashemi
similarity 0.59
Abstract · 14 pages, 5 figures

Coupled-dynamics environments expose the one-step outcomes that would follow from several possible counterfactual actions under a common realization of exogenous randomness. The ordinary Markov decision process formalism allows one to reason about the marginal law of each action but discards dependence across these counterfactual outcomes. The Joint Markov decision process (JMDP) formalism preserves that dependence. Prior work established the formalism and solved the fixed-policy joint moment evaluation problem in JMDPs. This paper develops optimal-control methods. We define a nonparametric distributional Bellman optimality operator for JMDPs, and prove that when the induced marginal MDP has a unique optimal policy, its iterates converge in Wasserstein distance to the optimal joint return law. For the first two moments, we establish convergence under a weaker condition that permits several mean-optimal actions as long as their tie resolutions share a second-moment fixed point. We also derive sampled targets for neural approximation.

0.20

Reinforcement Learning with Multi-Step Transition Predictions: A Bellman Theory

cs.LG ·2026-10-23 ·Chenbei Lu, Zaiwei Chen, Tongxin Li, Chenye Wu, Adam Wierman
similarity 0.59
Abstract

Multi-step predictions of future transition realizations can improve reinforcement learning, but their value remains difficult to quantify. Directly including these predictions in the state can lead to exponential state-space growth, making direct dynamic programming computationally expensive. We study discounted finite MDPs under a block-feedback protocol: at the start of each K-period block, the decision maker observes $K$-step probabilistic transition predictions for a subset of actions and adapts decisions to observed states within the block. We show that the optimal value satisfies a Bayesian Bellman equation on the original state space. This representation reveals that predictive information creates value by recursively changing the local order of maximization and expectation in the Bellman expansion. We formalize this effect through a Bellman--Jensen analysis and quantify the effects of prediction horizon, error, action coverage, and within-block feedback. We propose block-feedback Bayesian offline learning with online adaptation (BF--BOLA), a model-based algorithm for learning near-optimal policies, and establish upper and lower sample-complexity bounds that match up to logarithmic factors in the stated regimes. The results show that predictions can reduce transition learning to uncovered rows and that learning a near-optimal policy can require fewer prediction samples than estimating the Bayesian value function.

0.17