CS Paper Radar

Location-Independent Robot-Assisted Finishing Using Digital Twins and Extended Reality

Jose Outeiro, Jia Holt, Tero Kaarlela, Khalil Chakal

cs.HC ·submitted 2026-09-08 17:16 UTC · v1

Abstract

This paper presents a cyber-physical system (CPS) for location-independent programming, supervision, training, and teleoperation of a Robot-Assisted Finishing (RAF) system used to post-process metal additive-manufactured (AM) components. A digital twin (DT) built in Unity is delivered to the operator as a WebGL application that supports both desktop and immersive modes through WebXR-compatible devices. Moreover, it exchanges robot state and pose commands with a collaborative robot through a Message Queuing Telemetry Transport (MQTT) broker. The DT enforces kinematic and collision constraints before a pose is released to the physical robot, and augments the virtual component with a color map of the surface topography that supports operator decisions on part repositioning or process termination. The architecture was validated on a specially designed physical RAF system. A steady-state joint synchronization error of 0.12 deg and a mean round-trip latency of 563 ms were measured, which is adequate for supervisory programming and intermittent teleoperation.

Comment
6 pages, 5 figures, 2027 IEEE/SICE International Symposium on System Integration

Closest papers

Nearest neighbours by meaning across the whole index.

Cyber-Physical Digital Factory Architecture as the Enabler of Disembodied Work

cs.HC ·2026-08-31 ·Tero Kaarlela, Ivan Ruchkin, Jose Outeiro, Souradeep Dutta
similarity 0.72
Abstract

Digital Twins (DTs), Artificial Intelligence (AI), and Industrial Internet of Things (IIoT) technologies have significantly advanced manufacturing digitalization. However, these technologies are typically applied to individual manufacturing processes rather than integrated into a unified cyber-physical manufacturing environment. This paper proposes a cyber-physical digital factory architecture that enables disembodied work, where manufacturing systems can be supervised and operated remotely through eXtended Reality (XR) user interfaces in collaboration between AI-based control and human operators. The architecture integrates synchronized DTs, hierarchical cloud-edge AI, IIoT, and XR teleoperation interfaces into a cyber-physical manufacturing environment. The proposed approach is validated through representative manufacturing operations, including CNC machining, robotic-assisted abrasive finishing, and robotized disassembly. The results demonstrate the feasibility of the proposed architecture for disembodied manufacturing work and provide a reusable cyber-physical framework for future human-AI-controlled digital factories.

0.06

A Cyber-Physical Machine Tool Framework with a Real-Time Machining Process Digital Twin

cs.HC ·2026-08-30 ·Khalil Chakal, Tero Kaarlela, Jose Outeiro, Carlos Andrade
similarity 0.66
Abstract

Digital Twins (DTs) have emerged as a key technology for improving the monitoring, optimization, and automation of manufacturing systems. However, existing Cyber-Physical Machine Tool (CPMT) implementations primarily represent the machine tool, while the machining process remains only partially synchronized with its physical counterpart. This paper extends a previously presented CPMT framework by introducing a hierarchical DT framework that simultaneously maintains DTs of both the machine tool and the machining process. The proposed framework integrates real-time CNC operational data, a voxel-based workpiece representation, synchronized process vibration measurements, and a persistent part DT repository for process replay, traceability, and future synthetic data generation. Experimental evaluation demonstrated real-time operation at a 20 Hz machining-state update rate, interactive visualization exceeding 100 frames per second, and a mean depth reconstruction error of 0.16 mm. The implementation provides a foundation for AI-assisted machining applications while preserving the machine tool monitoring and teleoperation capabilities.

0.23

GHOST in the Robots: Real-Time Exocentric Dual-Robot VR Teleoperation from Onboard Cameras

cs.RO ·2026-08-29 ·Yichen Wei, Faisal Zaghloul, Soujanya C Aryal, Aanya K. Agrawal, Chengfan Li +3 more
top venuecode/data similarity 0.65
Abstract

Teleoperating multiple robots simultaneously enables additional views and coordinated control. Yet, it poses fundamental challenges: the system must present sensor data cohesively and allow operators to manage multiple robot bases, arms, and cameras while maintaining low latency. Current multi-robot teleoperation systems require multiple operators, rely on autonomy, or restrict operators to high-level commands. We present GHOST: an open-source VR teleoperation system that enables single operator control of two mobile manipulators via direct lowlevel commands using only onboard sensing. GHOST creates an exocentric 3D workspace by aligning real-time point clouds from the robots' RGB-D cameras, where scene coverage is improved through learning-based completion to aid operator spatial awareness. For control, the operator uses a mode-switching architecture to command either robot individually or both robots simultaneously. Experiments with 15 novice participants demonstrate 1.6-4x the success rate of an off-the-shelf tablet interface. For experts across nine challenging dual-robot tasks, our system enabled completion of two tasks that were infeasible with the tablet, and was 1.47x faster on average than the tablet. Website and code: https://h2r.github.io/GHOST/.

0.38

Now You Feel It, Now You See Me: Digital-Twin-based Teleoperation Interface for Dexterous Manipulation

cs.RO ·2026-10-04 ·Youngchan Shim, Kyutae Lee, JooYun Kim, Jaeseong Hwang, Harim Ji +1 more
similarity 0.65
Abstract · 8 pages, 5 figures, 4 tables

Teleoperation is becoming increasingly important for collecting high-quality demonstrations to teach robots dexterous manipulation skills. For dexterous manipulation, bare-hand tracking provides a practical way to control robotic hands and demonstrate coordinated finger movements without gloves or exoskeletons. However, this type of teleoperation faces two key feedback limitations: a lack of force feedback and visual feedback of occluded region. The absence of force feedback hinders precise and safe manipulation, as operators must infer contact force visually rather than feel them directly. Occlusion by objects or other robot parts impairs the assessment of hand positions, approach distances, and pre-grasp configurations suited to the object's shape. To address these limitations, we propose a digital-twin-based augmented reality (AR) teleoperation interface that integrates bare-hand tracking, bimanual robotic hands, and virtual representations of the robot and task objects. The interface renders tactile measurements on robot hand meshes as visuo-force feedback. Furthermore, it offers a user-controlled virtual view panel or occlusion-aware transparency rendering to provide occlusion-mitigating visual feedback. We evaluated the interface through user studies involving Task 1 and Task 2, examining how force visualization and visual assistance support demonstration collection in tasks requiring careful force regulation and manipulation under occlusion.

0.12

Leveraging Human-In-The-Loop Demonstrations in Reinforcement Learning for Digital Twin-Driven Robot Flexibility

cs.RO eess.SY ·2026-10-08 ·Yuzhu Sun, Mien Van, Nguyen Minh Nhat, Stephen McIlvanna, Sean McLoone
similarity 0.62
Abstract

Growing automation makes collaborative robots work in more variable environments, increasing the need for adaptation. We propose a human-in-the-loop online training framework combining a digital twin (DT), reinforcement learning (RL), and human demonstrations. Unlike DTs used mainly to generate synthetic data before task execution, our DT is synchronized with the physical system in real time through camera feeds, allowing the virtual robot to update its observations and policy from real-world feedback. A dual actor framework integrates imitation learning (IL) without adding a direct imitation loss to the RL actor, so demonstrations can guide adaptation instead of manual reprogramming. The proposed framework is demonstrated on the Ufactory Xarm5 collaborative robot, where the robot's end-effector aims to reach the target position while avoiding obstacles. The experiments show that the framework can resume training after a change in the physical workspace and that, with a fixed set of non-optimal demonstrations, the dual actor framework achieves a much higher final success rate than two methods that add an imitation loss to the actor. The same pattern holds with real human demonstrations collected in virtual reality (VR): with demonstrations that never reach the goal, the dual actor framework reached 83-100% mean deterministic evaluation success, against 0-17% for the two imitation-loss methods.

0.26

Design and Evaluation of a Collaborative Teleoperation Interface for Multi-Operator Multi-Robot Systems in Challenging Environments

cs.RO cs.HC ·2026-10-10 ·Alperen Kenan, Juan José García Cárdenas, Adriana Tapus, Paul Bremner, Manuel Giuliani
published similarity 0.62
Abstract · 9 pages, 8 figures, accepted for presentation at the 2026 IEEE Conference on Telepresence in Bristol, United Kingdom, November 9-13, 2026

Multi-robot systems that combine the complementary capabilities of specialised platforms are critical for complex missions in challenging, inaccessible environments. Current deployments assign an operator or operator team to each robot, with collaboration relying on operator-to-operator communication to manually coordinate heterogeneous platforms with incompatible interfaces. This operator-dependent approach increases cognitive load and limits scalability across robots from different manufacturers. This study designs and evaluates a shared-visualisation collaborative teleoperation interface to improve operator coordination, situational awareness, and the task performance in multi-operator, multi-robot systems. A comparative user study with 16 participants compared three teleoperation conditions: co-located operators with a shared interface, remote operators with a shared interface, and co-located operators with individual interfaces. Participants controlled two simulated robots in a nuclear power plant digital twin to complete disaster response tasks. Performance metrics, physiological data, and behavioural measures were recorded to assess task performance, cognitive load, and operator awareness across conditions. The co-located shared-interface condition achieved the fastest mission completion (292.8 s), outperforming remote operation (351.6 s) and individual interfaces (385.1 s), with the fewest collisions (0.9 per trial) and highest situational awareness accuracy (91.1 %). Physical co-presence was associated with reduced cognitive load, with NASA-TLX scores increasing from 38.2 in co-located to 57.7 in remote operation, supported by consistent physiological indicators. These results demonstrate that co-presence and shared interfaces provide complementary benefits, supporting workload regulation and mutual awareness respectively in multi-operator, multi-robot teleoperation.

0.36

PUPPET: Physical-Unified Platform for Presence, Embodiment, and Teleoperation

cs.RO ·2026-11-19 ·Mahika Vats, Milan Groshev, Carlos Gomez Cubero, Eduardo Castelló Ferrer
similarity 0.61
Abstract · Submitted to IEEE ICRA 27

Remote access to robotics hardware does not by itself guarantee usable experimental data. A teleoperated trial can complete the task and still produce an incomplete or inconsistent record. PUPPET aims to solve this problem by presenting a cyber-physical framework that opens a physical robotics laboratory through a continuously synchronized VR replica of the robot, stranding environment, and task objects. In addition, PUPPET integrates a trial validation pipeline built into the experimental workflow. We evaluated PUPPET in a within-subject study with 30 participants, who performed the same table-to-shelf object-transfer task in person, through 2D teleoperation, and through PUPPET. VR reduced post-learning completion time by 27.7% over 2D (68.7,s vs. 95.0,s), statistically indistinguishable from in-person execution (72.1,s). It also produced more validation-ready (85.6% vs. 66.0%) and intervention-free (89.7% vs. 67.2%) trials. With PUPPET we show that modality shapes the remote task performance and the reusability of the resulting experimental record.

0.35

Single-Controller Teleoperation and Multimodal Dataset Generation for a Mobile Manipulator

cs.RO ·2026-11-18 ·Fardien Azizi, Sitor Sla, Christiaan Peppelman, Dion van Santen, Andrei-Carlo Papuc +3 more
similarity 0.61
Abstract

Robot learning systems require synchronized, high-quality demonstrations that combine visual, proprioceptive, and action data. However, collecting such datasets with mobile manipulators is often slowed down by fragmented teleoperation interfaces and manual post-processing workflows. This paper presents an integrated framework for teleoperated multimodal data collection on a Clearpath Dingo-O mobile base equipped with a Kinova Gen3 Lite manipulator and an Intel RealSense RGB-D camera. The framework combines single-controller operation of the base, arm, and recording process with an automated ROS-based pipeline for data capture, image-quality filtering, temporal synchronization, visualization, and export to machine-learning-ready Zarr datasets. We validate the system on a pick-and-place mobile-manipulation task in which the robot transports a bottle between two surfaces. The results show that the unified controller supports complete task execution and recording without external commands, while the data pipeline produces synchronized datasets for short demonstrations and identifies corrupted visual frames. The proposed framework provides a practical basis for producing structured demonstrations for robot-learning research.

0.20