CS Paper Radar

AQUAACT: A Joystick-Level Underwater Teleoperation Dataset and Closed-Loop Benchmark

Zhenqi Wu, Trung Tien Dong, Aditya Penumarti, Mohamed A. Fouad, Menna A. Fouad, Mingyang Mao, Jane Jaejeong Shin, Xiaomin Lin

cs.RO ·submitted 2026-10-18 21:07 UTC · v1
benchmark

Abstract

Underwater robots support a growing range of work, from inspecting offshore structures and pipelines to surveying reefs and accompanying divers. Most of these tasks, however, are still flown by hand by a pilot, which raises the question of whether they can be carried out autonomously instead. On land, vision-language-action (VLA) models already let robots carry out tasks autonomously from language instructions, which makes them a natural candidate for taking the pilot's place. Underwater, however, there is little data to train such models. We take inspiration from how these tasks are flown today: the pilot watches the camera image and steers with a joystick, whose commands the flight controller turns into motion. We therefore record the pilots and introduce AQUAACT, to our knowledge the first publicly available language-annotated underwater dataset pairing camera images with human joystick commands. It comprises 1.9 hours of real-world and 17.9 hours of simulated demonstrations, both flown by human pilots, together with the vehicle's inertial measurements. Experiments with AQUAACT show that its simulated demonstrations transfer to real data and greatly reduce the real demonstrations a policy needs. A closed-loop benchmark of seven policies in simulation, from imitation-learning baselines to VLA models, each trained with its default recipe, shows that these tasks remain far from solved: no policy completes more than a fifth of the episodes. Pretraining on land data gives large VLA models no advantage over small policies trained from scratch, a gap that land-trained models have yet to close underwater.

Closest papers

Nearest neighbours by meaning across the whole index.

AquaCap: A Training-Free Underwater Embodied Agent with Code-as-Policy

cs.RO ·2026-09-19 ·Xiaoshi Li, Yule Xu, Chunghiu Kong, Yizhou Zhou, Yang Liu +3 more
similarity 0.78
Abstract · 8 pages, 6 figures, 4 tables, 1 algorithm

Recent advances in vision-language-action models have stimulated growing interest in underwater embodied intelligence. However, their reliance on large-scale interaction data limits their applicability underwater, where data collection is costly and scarce. To address this challenge, we present AquaCap, a training-free Code-as-Policy framework for autonomous underwater navigation and manipulation. AquaCap employs a dual-layer agent that translates task instructions and environmental observations into condition-aware plans and executable control programs. Structured perception then provides the agent with semantic, geometric, and reliability-aware observations under degraded underwater conditions. A failure-aware memory diagnoses unsuccessful actions and supports closed-loop replanning and code revision. This design enables online adaptation without task-specific training or parameter updates. AquaCap achieves a 66.43% success rate in simulation. Real-world experiments further demonstrate autonomous grasping and object transport with an ROV, including the manipulation of targets displaced by hydrodynamic disturbances.

0.46

AquaWAM: A Dynamics-aware World Action Model for Underwater Embodied Agents

cs.RO cs.AI ·2026-09-27 ·Cunhao Zhu, Yifeng Wang, Dongliang Xu, Yunzhong Hou, Yue Yao +1 more
benchmark similarity 0.76
Abstract

World Action Models (WAMs) are becoming increasingly important and useful for embodied intelligence, as they enable robots to anticipate the consequences of candidate actions before interacting with the physical environment. However, underwater robots are usually subject to passive dynamics, such as inertia, buoyancy, hydrodynamic drag, and persistent drift, which can continue to affect the vehicle even after an action is completed. Existing WAMs, which primarily predict action-conditioned visual observations, are not explicitly designed to capture such passive motion dynamics. In this paper, we present AquaWAM, the first World Action Model designed for underwater embodied agents. Instead of predicting future images, AquaWAM models both action-conditioned and passive physical dynamics, including the thruster dead band, the inertial glide that outlasts each command, and ambient currents. Specifically, it senses through the DVL, IMU, pressure sensor and joint encoders, while cameras supply only semantics for understanding goals and target pose. By modeling compact navigation states rather than high-dimensional visual observations, AquaWAM substantially reduces the model size and computational cost compared with conventional WAMs. Experimentally, AquaWAM achieves a 72.6% task success rate across 20 underwater tasks on the USIM benchmark, outperforming existing methods while making action decisions 2.7x faster than U0 on an NVIDIA Jetson AGX Orin. Our model also remains effective when some onboard sensor measurements are unavailable. For example, without DVL velocity measurements, our method still achieves a 61.6% success rate, compared with 39.4% for U0.

0.16

ULOHA: An Underwater Bimanual Robot System for Robot Learning

cs.RO ·2026-09-16 ·Masato Kobayashi, Takeru Tsunoori
similarity 0.74
Abstract

Underwater visuomotor policy learning has focused primarily on single manipulators, while bimanual imitation learning has been studied largely in air. We present ULOHA, an underwater bimanual robot learning platform that combines custom-designed leader--follower hardware with software extensions to LeRobot, integrating teleoperation, multi-view sensing, demonstration collection, policy training, and autonomous deployment. Real-robot experiments demonstrate a range of coordinated underwater bimanual behaviors, including inter-arm transfer, shared-object manipulation, and buoyancy-driven interception. We evaluate ACT, Diffusion Policy, and the vision--language--action model SmolVLA on the platform. We investigate how learning methods and execution strategies developed for manipulation in air perform underwater, examining bubble disturbances, buoyancy-driven object motion, action-execution horizons, and real-time chunking. A separate single-arm study examines policy transfer between air and water and shows that demonstrations spanning both media support execution in both under the tested conditions. ULOHA provides a unified experimental platform for studying underwater bimanual robot learning under the coupled perceptual and physical effects of underwater environments. Additional material: https://mertcookimg.github.io/uloha/

0.61

Embodied at Sea: Imitation Learning for Multi-Task Underwater Manipulation

cs.RO cs.LG ·2026-10-28 ·Greta Perez-Haiek, Luca Macesanu, Kevin Angers, Mohammed Imam, Timothy Gambin +1 more
code/data similarity 0.73
Abstract · 8 pages, 8 figures. For associated files, see https://lair-research-nyu.github.io/Embodied-at-Sea/

Behavior Cloning has demonstrated promising generalization capabilities for robotic manipulation in terrestrial environments. However, marine visual distortions and dynamic currents create a severe domain shift that remains a key bottleneck in adapting generalist architectures to underwater field manipulation. To address this, 1616 episodes of underwater manipulation data were collected in calm and turbulent waters and were used to train 14 generalist Vision-Language-Action (VLA) models (DynamicVLA) with varying proprioception observation configurations. Across three tasks on 1260 gradable on-hardware rollouts in a pool, 10 models outperformed the baseline, with the highest-performing model achieving a 93.3% overall success rate, including partial successes. Results are also introduced from a case study evaluating the performance of a single-task Action Chunking Transformer (ACT) and a generalist VLA (SmolVLA) policy for two tasks in marine archaeology and conservation: grasping and mooring. Experiments are performed on an ROV on and around the WWII wreck Coralita X127 and at Manoel Island Marina, Malta. Across 748 collected episodes and 508 gradable on-hardware rollouts, the ACT policy substantially outperformed SmolVLA (47.9% vs. 7.9% pooled success rate), earning a 66.7% success rate on the most successful task at the marine archaeological site. To enable further contributions to embodied intelligence in marine field robotics, the BLUEMBODIED KIT used to collect data, train models, and deploy policies on the ROV has been open sourced.

0.22

WasserMan: Benchmark for Underwater Manipulation Policy Learning

cs.RO ·2026-10-03 ·Danil Belov, Artem Erkhov, Sergei Parsegov, Pavel Osinenko
benchmark similarity 0.71
Abstract · 13 pages, 8 figures

Underwater contact manipulation couples visual decisions with vehicle stabilization and mechanism interaction. To our knowledge, WasserMan is the first multi-task simulation benchmark focused on visuomotor policy learning for manipulating articulated underwater mechanisms. The suite supports demonstration-based policy learning, alternative vehicle--arm platforms and bimanual manipulation. We compare ACT, diffusion policies and behavioral cloning under matched sample budgets, and evaluate adaptation of a pretrained SmolVLA baseline. Fixed-policy studies vary water appearance, lighting, contact parameters, actuator delay, arm properties and mechanism pose. RGB feedback controls and matched observation-history and expert-trajectory ablations examine observation and training choices. Expert references establish task feasibility under the same physical success tests. The experiments reveal task-dependent visual transfer and pose sensitivity, while contact, tracking and actuator measurements distinguish execution responses that completion alone hides. Complete expert candidate streams make the initial states associated with demonstration selection inspectable. Versioned code, demonstrations, checkpoints and per-episode physical evidence support reproducible comparisons within the simulation.

0.20

A Sim-to-Real Integration Pipeline for Training and Deployment of Chunk-Based VLA Manipulation Policies

cs.RO ·2026-09-18 ·Mathilde Kappel, Clémence Grislain, Mohamed Chetouani, Olivier Sigaud, Louis Annabi +3 more
publishedcode/data similarity 0.71
Abstract · Accepted at IROS 2026 Workshop on Sim2Real and Classical Control: From Rigorous Theory to Data-Driven Robotics

Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions. Most state-of-the-art models predict actions in the end-effector pose space as sequences of action chunks. Training and evaluating these models requires large-scale collections of real-world demonstrations, pairing robot actions with the corresponding visual and proprioceptive observations. Collecting such data on real hardware typically relies on human teleoperation, making the process costly, time-consuming, and difficult to scale. We present an open-source sim-to-real experimental protocol that addresses this bottleneck: expert trajectories generated in simulation are replayed open-loop on a real Franka FR3 setup, where the corresponding real visual and proprioceptive observations are recorded and converted into a format compatible with VLA training. The same deployment stack is then reused, in closed-loop, to evaluate a trained policy on that setup, so that data collection and evaluation share an identical hardware configuration. Because each real recording is paired with the simulated trajectory that produced it, the protocol also yields a direct measurement of the sim-to-real gap. We release the collected datasets on Hugging Face together with the pipeline source code https://gitlab.isir.upmc.fr/kappel/sim2real_public_chunk_control.

0.34

Towards Reliable Underwater Diver-Robot Interaction: Gesture Design, Interaction Logic, and Real-World Evaluation

cs.RO ·2026-09-20 ·Yingqi Liu, Kanzhong Yao, Zimeng Peng, Yuanbo Bi, Anran Li +2 more
hot direction similarity 0.69
Abstract · This work has been submitted to IEEE for possible publication

Underwater human--robot interaction requires gesture commands that are both easy for divers to use and reliable for robots to recognize. We investigate these aspects through a closed-loop diver--robot interaction framework integrating a compact seven-gesture vocabulary, lightweight landmark-based recognition, and command-level interaction logic. We evaluate the framework through a user study and underwater robot experiments in a laboratory tank and a swimming pool. The user study supported the reproducibility of the gestures after brief learning. Recognition analysis further showed that visual similarity was associated with gesture confusion, while intermediate poses during gesture formation introduced temporal ambiguity. Command-level processing mitigated the effects of transient recognition errors on robot execution, reducing unintended triggers and premature task interruptions. These findings show that reliable underwater gesture interaction depends on human usability, gesture recognizability, and execution reliability in underwater interaction.

0.30

DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptation

cs.RO ·2026-08-29 ·Makoto Sato, Tatsuya Matsushima, Yutaka Matsuo, Yusuke Iwasawa
similarity 0.69
Abstract · 8 pages, 5 figures

Vision-language-action (VLA) models have made strong progress in language-conditioned robot manipulation, but improving their performance in a new workspace still often requires action-labeled data from that environment. Collecting such data by human teleoperation is costly, especially when each workspace, object arrangement, or task may require new demonstrations. We present DREAM, a framework that generates fine-tuning data for a pretrained VLA from a captured workspace and a language instruction, without requiring a task-specific human demonstration. DREAM reconstructs the workspace, automatically translates the instruction into symbolic task goals and success criteria using a large language model, and uses task-and-motion planning to generate feasible robot trajectories. The planned trajectories are augmented across randomized object configurations, verified by the generated success criteria, and rendered into image-action examples for VLA fine-tuning. Through real-robot experiments on language-conditioned manipulation tasks, we study whether DREAM can serve as a scalable data-collection system for the deployment workspace by examining whether fine-tuning on its automatically generated data improves success over direct deployment and how its data-collection cost compares with human teleoperation when adapting a VLA to a new workspace.

0.40