CS Paper Radar

Unthrottling the Tanh Jacobian in SAC: A Negative Result on Bang-Bang Control and MetaDrive

Faiq Shamass

cs.LG cs.RO ·submitted 2026-09-08 21:53 UTC · v1
code/data

Abstract

Soft Actor-Critic (SAC) represents a continuous policy as an unbounded Gaussian that is squashed by tanh. The Jacobian of that map is $\partial a/\partial u = 1-a^2$, which vanishes as $|a|\to 1$. A natural concern is that this throttle starves the actor of critic signal exactly where extreme actions (full brake, full throttle) are optimal. We test a minimal intervention that restores the missing signal: one extra term in the actor loss whose gradient on the pre-tanh mean is the detached action-gradient of $Q$, with no gain parameter. On a minimum-time double integrator whose optimum is bang-bang at the action bounds, vanilla SAC already reaches near-optimal return ($-31.6$ vs. a calibrated optimum of $-30.3$) across ten paired seeds. An ungated bypass does saturate the policy (99% of eval steps with $|a|\ge 0.9$) and collapses return to $-195.5$. A gated bypass that fires only on the flat shoulder $|a|\in[0.9,0.999]$ also fails, and does so without leaving a saturated policy. Warm-started MetaDrive fine-tuning shows the same pattern: the bypass does not improve return, and where collision rate falls it is typically traded for out-of-road departures. Auto-tuned entropy coefficient rises against the bypass, which is a push toward the tails. The Jacobian effect is real. Treating it as a bug to be undone is not free, and on the tasks studied here it is not helpful. Saturating a bound is not the same as solving a problem whose optimum lives on that bound.

Comment
8 pages, 2 figures. Technical report. Negative result. Code: https://github.com/fshamass/Tanh-Bypass

Closest papers

Nearest neighbours by meaning across the whole index.

Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO

cs.LG ·2026-08-25 ·Yiyang He, Zhichun Zhou, Ziwei Wang, Tao Xue, Haolin Fei
similarity 0.58
Abstract · 24 pages, 6 figures, 8 tables

Many continuous-control policies are optimized as unbounded Gaussians and then mapped into bounded actions. We show that where entropy is measured changes the policy geometry learned by proximal policy optimization (PPO). In an 80-muscle MyoLeg task, a clipped Gaussian executes 89.07% of actions within 5% of a bound. A same-state decomposition shows that this is not due to variance alone: setting variance to zero still leaves 83.83% of actions near a bound, while 82.12% of state-conditioned means lie outside the executable interval. Replacing clipping with a tanh map does not remove the high-variance regime. For latent Gaussian entropy H(u), the entropy loss has zero gradient with respect to the mean and a constant variance-increasing gradient. For executed-action entropy H(a), the transform Jacobian adds an inward gradient on the mean. Across three matched MyoLeg seeds, near-boundary occupancy is 71.42%, 29.76%, and 18.83% under latent entropy, no entropy, and executed-action entropy. A 38-dimensional Dog-Stand replication with an independent CleanRL-based PPO implementation reproduces the ordering in mean geometry, which also survives shared-state evaluation and boundary margins from 1% to 10%. Direct mean penalties can match or exceed the centering produced by H(a), showing that interior means are not unique to executed entropy. However, matched mean geometry can coexist with substantially different variance and return. Entropy measurement space is therefore a coupled mean-variance design choice, and task return alone does not characterize bounded-policy geometry.

0.35

Convex Pursuit-Evasion Games for Spacecraft Proximity Operations on Circular and Elliptical Orbits

eess.SY math.OCastro-ph.IMmath.DS ·2026-09-13 ·Omer Burak Iskender
published similarity 0.57
Abstract · 38 pages, 10 figures, 19 tables (9 in appendices); accepted at Aerospace Science and Technology, 11 September 2026; ID: AESCTE-D-26-03480R1

Inspecting or servicing a non-cooperative spacecraft is a two-player game: the target can thrust to defeat the inspector's plan. Hamilton-Jacobi-Isaacs reachability answers such games exactly but its cost grows exponentially with state dimension, while learning-based controllers scale yet certify nothing. Convex formulations are the usual escape, typically cast as convex-concave saddle-point problems. That framing fails in the terminal-distance orbital game: with effort regularization, the payoff is convex in both controls, so no pure open-loop saddle point need exist. In its place we derive two exact one-sided guarantees read from the players' terminal reachable sets via support functions: an escape certificate proving the target can hold a guaranteed standoff, and a security strategy bounding the miss distance the inspector can force. Together they bracket the engagement without assuming player rationality, and over two hundred perturbed trials the escape certificate separated capture from escape without error. A projected extragradient method supplies a strategy pair in about 25 ms, certified in place by best-response gaps. The construction carries unchanged to elliptical reference orbits via Yamanaka-Ankersen dynamics, where the orbital phase moves miss distance by nearly a factor of two, reproduced by nonlinear Keplerian propagation. Under receding-horizon play the inspector captures in half of ten representative engagements. Angles-only navigation error, keep-out zones, multiple pursuers, and closed-loop play are treated as extensions.

0.67

A Concentration Bound for Two-Timescale Actor-Critic Algorithm

cs.LG ·2026-09-24 ·Prashansa Panda, Shalabh Bhatnagar
similarity 0.56
Abstract

Significant research effort has been directed in recent years towards establishing both asymptotic and non-asymptotic convergence guarantees for two-timescale actor--critic algorithms, where the actor recursion is run on a slower timescale than the critic recursion. This work derives a uniform all-time concentration bound for the actor--critic algorithm with function approximation in the long-run average-reward setting. This bound helps us analyze the behavior of the actor parameter with high probability. We show that, after some finite time, the actor parameter enters a safe region and remains within it thereafter with high probability. Specifically, with probability at least $1-ε_1-ε_2$, the actor error $\Vert θ_k-θ^{*}\Vert$ is $O\left(\frac{n_0^{3/4}}{k}\frac{1}{\sqrt{ε_2}}+\left(\frac{1}{n_0}\right)^{1/4}\log^{1/4}\left(\frac{1}{ε_1}\right)+\left(\frac{1}{n_0}\right)^{1/4}\right)$ for all $k\geq n_0$ and sufficiently large $n_0$. We also present experimental results demonstrating that the aforementioned actor error diminishes with the number of actor-parameter updates.

0.21

PolicyAttention: Softmax Attention Implements Policy Mirror Descent for Closed-Loop Control

cs.LG cs.AI ·2026-09-24 ·Yuhe Sui, Yingzhi Tang, Shufang Chen
similarity 0.55
Abstract

Can causal softmax attention implement policy mirror descent as a repeated controller rather than a one-step algebraic identity? Negative-entropy policy mirror descent (PMD) has the statewise update $\operatorname{PMD}_η(π,Q)=\operatorname{softmax}(\logπ+ηQ)$. Building on the known Q-TD-PMD recursion, we construct one fixed causal-softmax actor--environment--one-step-critic protocol with explicit actor, routing, sampling, and normalization residuals, and propagate them to the policy actually returned. The construction states the finite-logit/full-support domain, the external tokenization and sampling boundary, and the mean-zero LayerNorm carrier conditions required by the normalized compilation. Separately trained pre-LN Transformers recover the target computation empirically. A frozen one-step audit model is closest to PMD among the tested fixed rules; in a preregistered five-run $S=4$ repeated-control test, the learned actor with an exact one-step critic reaches median returned-policy loss $1.052\times$ the Exact PMD oracle and retains the criterion across four no-retraining shifts. The same checkpoints with their learned critic give descriptive median $1.050\times$ the oracle (no registered margin). At $S=8$, replacing the exact critic by the learned critic raises median $T=20$ loss to $0.0225$ yet leaves the Liang--Lai and Algorithm Distillation adaptations $20.2$--$24.2\times$ higher-loss; this is a one-sided sampled-critic bound because PolicyAttention consumes 144 generative transitions per round versus 20 on-policy transitions for the adaptations. The strict 20-transition comparison remains open. At $S=8,16$, the exact-critic common-harness comparison remains $17.7$--$28.2\times$ lower-loss than those adaptations, with the information asymmetry stated locally.

0.28

Amortized Feedback Planning: Turning Model-Based Rollouts into Executable Policies

eess.SY math.OC ·2026-09-28 ·Jeonggyu Huh
similarity 0.54
Abstract

Closed-loop planning accounts for future observation-dependent actions but can be expensive to repeat at deployment. Bellman-gradient (BG) refinement differentiates conditional rollouts through an existing actor, corrects the current action, and stores the result in an executable policy. A backward sweep reuses deployed future feedback; full-horizon rollouts in an identified Gaussian belief model need no learned value critic. A nonlinear error recursion links quadratic continuation error, local correction, and policy storage. A controlled non-LQG example exhibits second-order policy accuracy with consistent storage, while ideal affine LQG admits exact backward recovery. In nonquadratic thrust, BG reduces actor cost by 2.47-3.20% and remains within 0.13-0.53% of the tested feedback MPC; original policies execute in 5-6 microseconds in a seed-0 native audit. With matched storage, BG attains competitive plant costs at about one ninth (arm) and one sixtieth (docking) of feedback-teacher-plus-student construction time on one GPU, excluding shared learning and node preparation. Richer common maps substantially narrow some student-BG gaps. These results expose the roles of learned feedback, local correction, and storage in executable control.

0.30

Ascent Conditions for Length-Normalized Single-Rollout Policy Updates

cs.LG ·2026-10-10 ·Levin David Schwab
similarity 0.54
Abstract · 16 pages, 1 figure, 3 tables; code and data included as ancillary files

Dividing each trajectory's score contribution by its realized length can change the direction of a reward-centered policy update. In a finite trajectory model, we derive an exact ascent threshold for a three-outcome softmax policy and exhibit a binary-reward example in which the expected Euclidean update decreases task success. Natural-gradient ascent holds for fully controllable outcome distributions with deterministic binary labels, but can fail under a one-parameter restriction. A third-order U-statistic estimates the first-order return change without bias from independent current-policy trajectories, one per request. In the example, a same-sample plug-in estimator instead has the opposite sign in expectation for batch sizes two through forty-nine. Exact policy-grid enumeration confirms the local sign reversal. In a branching process, length-normalized learning initially loses return, then recovers and exceeds the unnormalized comparator. The analysis concerns the expected update field at the current policy.

0.30

When Does Exploration Prevent Collapse? Three Regimes and the Path of the Reward in Self-Consistency Training

cs.LG ·2026-10-31 ·Jaeho Hwang, Uiwon Hwang
theory similarity 0.54
Abstract

Test-time reinforcement learning (TTRL) adapts a model without labels by rewarding answers that agree with the majority, but it collapses the policy onto its majority answer and can remove the correct answer from the samples. Exploration can prevent this collapse, yet its strength is chosen by trial and error, and how the training algorithm changes the required strength is unclear. We model training on a single problem as a continuous flow of the logits of a softmax policy and analyze which policies are stable as the exploration coefficient varies. Only collapsed policies are stable below a lower boundary and only policies spread over many answers above an upper boundary, and we derive these boundaries in closed form for an agreement reward with smoothing toward the uniform policy, compute them for the majority reward of TTRL on policies with one leading answer and bound them with interval arithmetic for actions on ordered bins. We prove that under exact gradients, the learning rate and, in a mean-field model, the normalization of GRPO leave these boundaries unchanged when exploration passes through them together with the reward but shift them when exploration bypasses them. Exact sums on these policies then show that a bonus for rare answers needs a coefficient above $1/\log(G-1)$ for groups of $G$ samples before the normalization and about two to three times that after it. In free-form TTRL on MATH-500, a bonus placed before the normalization prevented collapse and kept more correct answers among the samples, although it lowered pass@1.

0.13

A Multistage Extension of Witsenhausen's Counterexample with Linear Signaling

eess.SY math.OC ·2026-11-08 ·Quanyan Zhu, Tamer Başar
benchmark similarity 0.54
Abstract · 8 pages, 3 figures

We study a multistage scalar extension of Witsenhausen's counterexample in which controller 1 observes the initial state exactly, whereas every subsequent controller observes only a local noise-corrupted state and does not inherit its predecessors' observations, so that every nonterminal action both regulates the state and shapes the signal available to the next controller. Restricting the signaling controllers to linear policies while leaving the terminal controller unrestricted, we compute the exact value within this class and reduce the optimization to a deterministic dynamic program in a single propagated variance. The optimal intermediate gains are shown to have a shrinkage form, and the value functions of that program are concave. For two stages, the benchmark admits a closed form on a matching locus, where for small effort weights it has two distinct minimizers, and at the standard parameter instance on that locus a two-point nonlinear policy strictly improves upon it. For three stages, numerical continuation at a reference instance reveals three distinct local minima, and two values of the initial variance at which the lowest of the three changes abruptly. Under the terminal-cost normalization we further prove, for every finite horizon, that the optimal linear value becomes asymptotically equal to the signaling-idle benchmark as the initial variance tends to infinity.

0.35