CS Paper Radar

Runtime-Incremental Transformer for Reinforcement-Learning-Based Adaptive Control

Giansalvo Cirrincione, Adriano Fagiolini

cs.RO cs.LG ·submitted 2026-09-11 21:16 UTC · v1

Abstract

Learning-based adaptive control of robotic manipulators with non-observable friction memory has been addressed by attention- based meta-controllers whose number of attention heads is fixed before training and is tuned by costly offline search. At long memory horizons, such fixed-capacity controllers are prone to catastrophic failures on a sizeable fraction of training seeds. The present paper introduces a runtime mechanism that grows and prunes the heads of the attention block during reinforcement learning, governed by two signals: the effective rank of the on-policy context distribution, which triggers growth when representational capacity becomes insufficient, and the per-head output magnitude, which flags redundant heads for removal. Policy continuity at growth events and a quantitative bound at prune events are established analytically. On a two- link manipulator with Stribeck friction, the proposed mechanism attains full success across all memory regimes, eliminating the long-horizon failure mode and removing the need for offline tuning of the head count.

Closest papers

Nearest neighbours by meaning across the whole index.

Differentiable Iterative Learning Control for Black-Box Position-Commanded Manipulators

cs.RO eess.SY ·2026-10-31 ·Mohammed Al-Ghaili, Yakoop Qasim, Hassan Almubarak
similarity 0.62
Abstract

High-accuracy repetitive tracking is challenging for position-commanded manipulators whose low-level controllers are inaccessible, while constrained model-based iterative learning control (ILC) can require substantial manual parameter tuning. We present Differentiable Iterative Learning Control (Diff-ILC), an online bilevel framework that adapts the parameters of a constrained ILC update between physical robot executions. A closed-loop differential model is identified once from paired smooth perturbations and subsequently kept fixed. At each trial, a constrained quadratic program computes a feasible command correction, while a real-anchored next-trial prediction defines a task-level objective. Hypergradients are obtained by implicitly differentiating the locally active KKT system, enabling multiple computational parameter updates without additional robot executions. Experiments on a 6-DoF DOBOT CR3 across unseen planar and three-dimensional trajectories show substantially faster physical-trial convergence than fixed-parameter model-based ILC. Across the four primary trajectories, after only two trial-to-trial updates, Diff-ILC achieves 78.8% and 76.4% lower mean joint-vector and Cartesian RMSE, respectively, than fixed-parameter model-based ILC.

0.19

Value as Memory: Online Value-Guided Post-training for General Robot Policies

cs.RO ·2026-10-15 ·Wenye Yu, Yijin Chen, Kaiyi Xiong, Wenbo Han, Wendi Chen +5 more
similarity 0.60
Abstract

Real-world post-training of generalist robot policies faces two challenges: partial observability makes action selection ambiguous, while offline post-training methods delay feedback from deployment experience. We present Value as Memory (VaM), a framework that addresses these challenges through value caching and online real-world reinforcement learning (RL). Value caching conditions actions on recent value predictions as task-progress memory, while online updates enable timely policy improvement by combining offline demonstrations, off-policy interventions, and on-policy rollouts for sample-efficient adaptation. As a result, VaM transitions from offline RL warmup to online autonomous learning with sparse rewards. Across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM yields a mean absolute success-rate gain of $29.4\%$ with value caching, while achieving a $32.8\%$ gain with online autonomous learning over matched offline training. The evaluated online schedule performs $8\times$ as many policy iterations under the same gradient-step budget, with more than $2\times$ lower recorded time per gradient step with concurrent data collection and training.

0.17

ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models

cs.RO cs.AIcs.LG ·2026-10-13 ·Changyeon Kim, Kimin Lee, Yashraj Narang, Yijie Guo
similarity 0.60
Abstract · Project website: https://accrue-rl.github.io

Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors. Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks. We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL. The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes. Fixed visual features keep the inherited learner and its replay compatible with base-policy updates. Across contact-rich assembly sequences in simulation and on real-robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-robot training time by 31.0\% to 68.5\%.

0.34

Skills in Weights, Memory in Code: Hybrid Learning for Memory-Dependent Robot Manipulation

cs.RO ·2026-08-10 ·Yunhao Zhao, Zhenyang Ni, Haoyang Chen, Ruohan Zhang, Qi Zhu
similarity 0.59
Abstract · 9 pages, 4 figures, and 3 tables

Modern vision-language-action (VLA) policies have acquired broad manipulation skills, but typically generate each action chunk from the current observation or a short fixed-length history. However, real-world manipulation is often non-Markovian, requiring robots to retain and reason over task-relevant information from long-horizon interaction histories to determine the next action. To address this challenge, we propose HyMeS, a hybrid learning framework that leverages the reasoning and memory-management capabilities of coding agents to steer a Markovian VLA for memory-dependent manipulation. Specifically, HyMeS learns low-level motor skills through gradient-based imitation learning, while a coding agent acquires high-level memory-management strategies through heuristic learning by iteratively updating an executable heuristic system from rollout feedback. Furthermore, we close the loop between steering and execution through multimodal stage-completion verification, which updates memory using proprioceptive signals and multi-frame VLM judgments. Compared with end-to-end memory-augmented VLAs, HyMeS requires demonstrations only for reusable motor skills rather than for every history-dependent task configuration, enabling data-efficient compositional generalization. On RoboMemArena, HyMeS improves mean cumulative success from 52.5% to 66.2% and mean task success from 41.3% to 60.1% over pi0.5, while outperforming PrediMem by 4.5 points in cumulative success and 14.5 points in task success.

0.46

ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning

cs.RO cs.AI ·2026-08-19 ·Jayjun Lee, Jessica Yin, Asif Rana, Nicholas Blauch, Octi Zhang +6 more
publishedcode/data similarity 0.59
Abstract · Accepted at CoRL 2026 | Project page: https://adept-dexterity.github.io/

We introduce Accelerating Dexterity via Pre-Training (ADEPT), a large-scale reinforcement learning (RL) framework for learning sim-to-real transferable dexterity across high degree-of-freedom (DoF) robot embodiments that can solve long-horizon tasks directly from raw visuo-tactile perception. ADEPT pretrains a dexterous policy on a generic object reposing task, then post-trains downstream policies with this pretrained behavior as a prior. ADEPT enables learning new behaviors that are otherwise difficult to discover from scratch on multi-fingered robots and avoids learning the same set of skills over again for every new downstream task. The pretrained policy zero-shots the reposing phase of downstream tasks, but naïve RL fine-tuning rapidly degrades this capability during transfer. We address this with a stable post-training recipe combining behavior-cloning distillation, critic warm-up, and conservative on-policy updates. To safely exploit the full kinematic dexterity, we introduce a joint-space Geometric Fabric that mediates between the RL policy and the robot. We distill post-trained teachers into perceptive students that zero-shot sim-to-real transfer on two embodiments: a 23 DoF Kuka-Allegro with two RGB cameras, and a 29 DoF Flexiv-Sharpa with two RGB cameras and five vision-based tactile sensors, and can solve long-horizon tasks from challenging initial states with dexterity at human-level speed.

0.60

ResMerge: Residual Policy Learning and Merging for Continual Adaptation of Pre-trained Robot Policies

cs.RO ·2026-10-30 ·Kefei Zhu, Lingyun Xu, Yue Xin, Xiaoqi Li, Caixin Guo +4 more
code/data similarity 0.58
Abstract · 9 pages. Code: https://github.com/ResMerge/resmerge-official

Continual adaptation to new tasks is essential for robots deployed in the real world, but adapting pre-trained policies remains challenging due to parameter growth and the need for continual demonstration collection. Parameter-Efficient Continual Fine-Tuning (PECFT) isolates task-specific updates, but adapter-per-task designs require demonstrations and grow with the task set. We propose ResMerge, a two-stage framework that learns mergeable residual policies atop a frozen pre-trained policy: residual reinforcement learning (RL) acquires task-specific corrections without additional expert demonstrations during adaptation, and parameter-space merging consolidates compatible residuals to reduce stored parameters. We introduce Residual-Field Consistency (RFC), a simple, bounded, and interpretable metric that assesses compatibility in residual action-field space by comparing correction direction and magnitude. Experiments on LIBERO across two backbones and on a real-world Franka robot show improved continual learning performance over strong PECFT baselines, while selective merging reduces the residual-parameter footprint. Project website: https://resmerge.github.io/.

0.26

Rapid On-Robot Learning for Dynamic Manipulation Skills: Robot Juggling

cs.RO ·2026-08-27 ·Taeyoon Lee, Chunpeng Wang, Christopher G. Atkeson, Alfred A. Rizzi, Nicolas Rojas
similarity 0.58
Abstract

We present an online learning framework that enables a bimanual robot to acquire diverse juggling patterns directly on physical hardware within minutes, even with a significant sim2real gap. One of the most important lessons from this work is that a model, even when far from reality, can be extremely useful for learning. This motivates a central philosophy of our approach: learning should build upon the robot's current knowledge rather than replace it. Our regularized memory-based learning puts this principle into practice by learning a local model from accumulated experience while retaining the global prior model to extrapolate where experience is sparse. This enables efficient and stable online learning from each new experience without resorting to uninformed exploration over a vast space of possible behaviors. Equally important to continual on-robot learning is safety, allowing the robot to repeatedly practice and improve in the real world. We construct a mutually reachable set that allows safe transitions between successive throws and catches, without driving either arm into a state from which its next action would require violating the robot's joint or actuator limits. Together, these ideas enable a bimanual robot with multi-fingered hands and onboard vision to safely learn and compose five canonical three-ball juggling patterns, including cascade, tennis, half-shower, shower, and box, within less than 5 minutes of real-world interaction. More broadly, this work points toward robots that build upon imperfect prior knowledge and continually refine their behavior through their own real-world experience.

0.14

Three High Performance Global Tracking Composite Adaptive Controllers for Fully Actuated Euler-Lagrange Systems: Experimental Validation

eess.SY ·2026-09-22 ·Luis Cervantes-Pérez, Jose Guadalupe Romero, Romeo Ortega, Víctor Santibáñez, Jesús Sandoval
similarity 0.58
Abstract

Three adaptive global tracking controllers for fully actuated Euler-Lagrange systems, with verifiable performance improvement over existing designs, are reported in this letter. Two of these controllers ensure global exponential convergence under a weak interval excitation condition. Besides, one of the proposed controllers features a simple adaptive PID-like structure that-unlike classical solutions-avoids the need for additional filtering. We adopt a composite adaptation architecture, invoke a novel parameterization of the system dynamics and use a high performance estimation scheme recently introduced in the literature. Real-time experiments and a comparative study with a learning-based adaptive controller on a two-degrees-of-freedom manipulator arm illustrate the effectiveness of the proposed controllers.

0.21