CS Paper Radar

A Multistage Extension of Witsenhausen's Counterexample with Linear Signaling

Quanyan Zhu, Tamer Başar

eess.SY math.OC ·submitted 2026-11-08 06:48 UTC · v1
benchmark

Abstract

We study a multistage scalar extension of Witsenhausen's counterexample in which controller 1 observes the initial state exactly, whereas every subsequent controller observes only a local noise-corrupted state and does not inherit its predecessors' observations, so that every nonterminal action both regulates the state and shapes the signal available to the next controller. Restricting the signaling controllers to linear policies while leaving the terminal controller unrestricted, we compute the exact value within this class and reduce the optimization to a deterministic dynamic program in a single propagated variance. The optimal intermediate gains are shown to have a shrinkage form, and the value functions of that program are concave. For two stages, the benchmark admits a closed form on a matching locus, where for small effort weights it has two distinct minimizers, and at the standard parameter instance on that locus a two-point nonlinear policy strictly improves upon it. For three stages, numerical continuation at a reference instance reveals three distinct local minima, and two values of the initial variance at which the lowest of the three changes abruptly. Under the terminal-cost normalization we further prove, for every finite horizon, that the optimal linear value becomes asymptotically equal to the signaling-idle benchmark as the initial variance tends to infinity.

Comment
8 pages, 3 figures

Closest papers

Nearest neighbours by meaning across the whole index.

The Exact Boundary of Linear Optimality in Witsenhausen's Counterexample

eess.SY math.OC ·2026-10-19 ·Yikai Lu, Tongxin Li
similarity 0.64
Abstract · 39 pages

Witsenhausen's counterexample has resisted a closed form solution for decades. However, it has remained unknown whether a mathematically tractable representation exists or not. In this paper, we solve this open problem, showing that the optimal policies are either linear or not tame and characterizing the exact regions where the two properties hold. Moreover, in the linear optimal region, we give an essentially unique closed form solution for the optimal policy, and show that no optimal first-stage policy is definable in any $o$-minimal expansion of the ordered real field in the nonlinear optimal region.

0.27

Optimal Threshold Type Policies for Partially Observable Restless Bandits

eess.SY ·2026-09-12 ·Anu Krishna, Rahul Meshram, Kesav Ram Kaza
similarity 0.60
Abstract

We study a finite-state partially observable restless multi-armed bandit (PO-RMAB) motivated by resource-constrained wildlife monitoring. The underlying condition of each location evolves independently, while only a limited number of locations can be actively monitored at each decision epoch. Activation reveals the current state, whereas passive operation provides no observation, yielding a collapsing-bandit belief dynamics. Our main contribution is a structural characterisation of optimal policies for the multidimensional belief-state problem. We establish sufficient conditions under which an activation advantage is monotone with respect to the belief state, and hence the optimal policy has the threshold type structure over the $(M-1)$-dimensional belief simplex. The key result is obtained by bounding the variation of the value function through a model-dependent Lipschitz constant. We further specialise the result to one-step birth-death dynamics and derive explicit bounds.

0.05

Optimal Regret for Online Storage Control via Cumulative Policies

eess.SY math.OC ·2026-09-18 ·Kamiar Asgari, Michael J. Neely
theory similarity 0.59
Abstract

We study online control of a scalar storage system with adversarial nonnegative arrivals, known retention coefficient, and convex costs depending on both state and action. Each action must respect current resource availability and is chosen before the current arrival and cost function are revealed. For the existing simplex disturbance-action policy class, we give an exact reparameterization by cumulative allocation fractions and a decay-weighted projected subgradient update. The resulting regret bound is independent of policy memory length. For fixed retention coefficient and cost constants, the controller achieves $O(\sqrt T)$ regret against the best fixed infinite-memory policy in this class, using $O(\log T)$ memory and arithmetic operations per round and one cost-subgradient query. A storage-specific block construction gives a matching lower bound against every causal feasible controller, including randomized controllers. Writing $τ=(1-α)^{-1}$, the minimax expected regret is $Θ(\sqrt T\min\{T,τ\}^{3/2})$ for every finite-memory simplex policy class and its infinite-memory extension, when $α\in[1/2,1)$, $T\ge4$, and the positive cost constants are fixed. This identifies the joint horizon and retention-time dependence for these policy benchmarks.

0.10

The Variance of Thought: Policy Variance, Critical Forks, and Local Credit Assignment

cs.LG ·2026-08-23 ·Yingru Li
similarity 0.58
Abstract

Long-horizon language-model tasks --- multi-step reasoning and tool-using agents alike --- are limited by credit assignment. We analyze it through the policy variance $σ_π^2(s)=\operatorname{Var}_{a\simπ}[Q_π(s,a)]$, which in a deterministic MDP is the sole source of return variance and is injected in discrete pulses at states we call critical forks. Three results follow. (i) Policy variance is a discovery budget: observing an action of advantage $c$ requires $Ω(c^2/σ_π^2(s))$ draws, a bound that is exact on the canonical two-point fork. (ii) Policy variance is bounded by the policy's Gini dispersion, $σ_π^2(s)\le 1-\|π(\cdot|s)\|_2^2$, a rollout-free necessary condition for criticality computable from logits alone. (iii) The remaining horizon sets the estimation cost: at a fork whose downstream success probability is $P$, the Monte Carlo advantage estimate has signal-to-noise ratio of order $\sqrt{P}$, so its sample cost scales as $1/P$ --- a cost that branched sampling shares. Bootstrapping removes it by converting a product of survival probabilities into a sum, provided the value representation is multiplicatively accurate, which argues for log-value parameterization.

0.29

Vector Bellman Theory for Multichain Robust Average-Reward Markov Decision Processes

cs.LG ·2026-09-23 ·Yue Wang, George Atia
similarity 0.58
Abstract · preprint, work in progress

Robust average-reward Markov decision processes provide a fundamental framework for long-term performance optimization under uncertainty, and can have optimal long-run rewards that depend on the initial state. This state dependence requires a vector Bellman theory that accounts for both recurrent-class rewards and transition uncertainty. We develop such a theory for finite models with compact, post-action $(s,a)$-rectangular ambiguity. A gain-first, bias-second optimization principle yields a coupled vector gain-bias system, and every finite solution identifies the optimal robust gain and supplies stationary saddle strategies against history-dependent opponents, simultaneously from all initial states. We further characterize solvability through stationary gain conditions and a uniform bound on canonical transient corrections, and give sufficient conditions that permit distinct recurrent-class gains. The certificates also yield asymptotically affine trajectories of the robust Bellman operator, based on which we design a robust approximately shifted Halpern planning algorithm. Under finite Bellman solvability, the gain estimates and Bellman displacements converge to the optimal gain vector, and every extracted greedy controller is average-optimal after a finite, instance-dependent budget. These results thus connect finite Bellman certificates to undiscounted planning for state-dependent robust average rewards, providing theoretical understandings.

0.27

Near-Minimax-Optimal Rates for Asynchronous TD Learning with Markovian Data

cs.LG stat.ML ·2026-09-30 ·Yang Peng
theory similarity 0.57
Abstract

We study the last iterate of standard importance-weighted temporal-difference (TD) learning from a single trajectory of a finite Markov decision process. The trajectory follows a behavior policy $b$ with an irreducible, possibly periodic state chain; a target policy $π$ covered by $b$ is evaluated with action importance ratios bounded by $\varrho$. Write $H=(1-γ)^{-1}$, with $μ_{\min}$ the minimum stationary probability and $\mathfrak h_\star$ the largest expected hitting time of the reverse behavior state chain. We prove that the last iterate achieves sup-norm error at most $\varepsilon$ with high probability after $\widetilde O(\varrho H^3/(μ_{\min}\varepsilon^2)+\mathfrak h_\star)$ transitions for $0<\varepsilon\le1$. The statistical term retains the cubic effective-horizon dependence of synchronous TD, while the initialization cost is additive in $\mathfrak h_\star$. Setting $b=π$ and $\varrho=1$ recovers the on-policy rate. For aperiodic behavior chains, $\mathfrak h_\star=\widetilde O(t_{\rm mix}/μ_{\min})$ recovers a mixing-time bound. The proof combines an anchored local Poisson equation in reverse time to control fluctuations without a mixing-time factor, a compensation argument for successive hitting costs, and an expected cumulative propagation mass bound for decreasing step sizes. Given the policy ratios and their bound, an empirical state-count rule attains the same rate without prior knowledge of model parameters. The framework covers nonnegative affine importance-weighted prediction updates, including successor features, transition-dependent discounting, and coupled finite-mode predictions. A perturbation estimate yields a time-uniform guarantee for online return-variance estimation. A three-state, two-action construction gives matching minimax lower bounds for both terms up to logarithmic factors on the considered parameter classes.

0.47

A Concentration Bound for Two-Timescale Actor-Critic Algorithm

cs.LG ·2026-09-24 ·Prashansa Panda, Shalabh Bhatnagar
similarity 0.57
Abstract

Significant research effort has been directed in recent years towards establishing both asymptotic and non-asymptotic convergence guarantees for two-timescale actor--critic algorithms, where the actor recursion is run on a slower timescale than the critic recursion. This work derives a uniform all-time concentration bound for the actor--critic algorithm with function approximation in the long-run average-reward setting. This bound helps us analyze the behavior of the actor parameter with high probability. We show that, after some finite time, the actor parameter enters a safe region and remains within it thereafter with high probability. Specifically, with probability at least $1-ε_1-ε_2$, the actor error $\Vert θ_k-θ^{*}\Vert$ is $O\left(\frac{n_0^{3/4}}{k}\frac{1}{\sqrt{ε_2}}+\left(\frac{1}{n_0}\right)^{1/4}\log^{1/4}\left(\frac{1}{ε_1}\right)+\left(\frac{1}{n_0}\right)^{1/4}\right)$ for all $k\geq n_0$ and sufficiently large $n_0$. We also present experimental results demonstrating that the aforementioned actor error diminishes with the number of actor-parameter updates.

0.21

The Bias of Nonlinear Two-Time-scale Stochastic Approximation under Constant Step-Sizes

cs.LG math.OCstat.ML ·2026-09-17 ·Djamel Rassem Lamouri, Dorian Baudry, Nicolas Gast
theory similarity 0.57
Abstract

Two-timescale stochastic approximation (TTSA) is a fundamental tool for analyzing coupled iterative algorithms in reinforcement learning, optimization, and stochastic control. However, finite-time guarantees for nonlinear two-timescale schemes remain difficult to obtain, especially under constant step-sizes. In this paper, we study nonlinear TTSA with step-sizes $α\ggβ$. Under standard stability, regularity, and Markovian noise assumptions, we upper bound the mean-squared error and the bias of both iterates around their limiting equilibria. Our bounds scale as $O(α+β^2/α^2)$, which we prove to be tight when $β\leα^{3/2}$. The analysis separates the contributions of initial conditions, fast-timescale tracking error, Markovian dependence, and timescale coupling, thereby clarifying the origin of the $β^2/α^2$ term. Our results reveal qualitative differences from the linear TTSA setting previously studied, showing that nonlinear dynamics introduce additional finite-time effects that are absent in the linear case.

0.37