CS Paper Radar

Evaluating the Impact of Adaptive Extended Reality on Human-Robot Interaction Across the Reality-Virtuality Continuum

Carl Tornberg, Alicia Torck, Lotfi El Hafi, Tadahiro Taniguchi

cs.RO cs.HC ·submitted 2026-09-25 11:27 UTC · v1
code/data

Abstract

As populations in developed countries age and labor shortages intensify, Cybernetic Avatars (CAs) are proposed to extend human capabilities through robotic embodiments, requiring effective Human-Robot Interaction (HRI) frameworks. Extended Reality (XR), an umbrella term for Augmented Reality (AR), Augmented Virtuality (AV), and Virtual Reality (VR), offers such interfaces, but prior research typically fixes the XR modality without evaluating its effect on task outcomes. This study examines whether the XR modality impacts HRI performance and whether an adaptive interface adjusting the level of virtuality along the Reality-Virtuality Continuum (RVC) at runtime improves it. A custom XR application interfaced with a mobile manipulator supports immersive control and runtime modality switching. In a within-participant multi-room pick-and-place experiment comparing fixed AR, AV, and VR with dynamic RVC through task metrics, the NASA-TLX, and the System Usability Scale (SUS), this study demonstrates that 1) the fixed reality modality affects HRI results, and 2) dynamically changing the modality along the RVC improves them. AR yielded significantly lower mental demand, effort, and frustration than AV and VR, while the dynamic RVC condition achieved the highest throughput and lowest workload, highlighting the value of adaptive XR interfaces for human-robot symbiosis. The implementation is available at https://github.com/CarlTornberg/XR-HRI.

Comment
Submitted to Advanced Robotics, Special Issue on "Next Generation Cognitive Robotics: Nurturing Embodied Intelligence for a Symbiotic Future with Humans and AI"

Closest papers

Nearest neighbours by meaning across the whole index.

Comparative Evaluation of an XR Pen-based Control Interface for Semi-Autonomous Mobile Robot Navigation in Service Environments

cs.RO cs.HC ·2026-09-25 ·Alicia Torc, Carl Tornberg, Eric Piette, Renaud Ronsse, Benoit Macq +3 more
hot direction similarity 0.71
Abstract · Submitted for presentation at the 2027 IEEE/SICE International Symposium on System Integration (SII), Kobe, Japan

Service robots remain difficult to deploy in domestic environments, partly because fully autonomous operation is not yet reliable in unpredictable surroundings, and partly because conventional control methods remain inaccessible to novice users. Extended Reality (XR) enables operators to visualize robot information overlaid onto the real world and to interact with augmented elements. Yet, common XR control methods, such as motion controllers and hand gestures, are still perceived as unintuitive. This paper presents a control interface that uses a commercial XR pen to command a semi-autonomous mobile robot in Augmented Reality (AR): the operator points at a position in the room, selects it, and drags an augmented arrow to set the desired orientation of the robot at this destination. Two additional interfaces, based on the XR motion controllers and hand gestures, were developed within the same framework. To assess the performance and users' perception of these interfaces, and of the XR pen in particular, a study with 10 participants compared four control methods, i.e., the XR pen, the XR motion controllers, hand gestures, and a computer-based baseline RViz, in navigation tasks performed in a home-like environment. Results show that the XR pen significantly outperforms the other methods in task selection time with the most consistent selections, and that the XR motion controllers obtain the best perceived workload and usability scores, ahead of the computer-based baseline, supporting XR-based control as an intuitive alternative for novice users. However, technical limitations in the integration of the recently released XR pen currently hold back its user experience.

0.36

Beyond the 1D Reality-Virtuality Continuum: A 2D Framework for Seven Extended Reality Paradigms

cs.HC ·2026-10-27 ·Rajashekhar Vachiravelu Saminathan
published similarity 0.63
Abstract · Accepted for IndiaHCI 2026 Conference in the Student Research Consortium held at Indian Institute of Technology - Hyderabad, India; 7 pages,…

The Reality-Virtuality (RV) continuum helps in describing the relationship between physical and computer-generated environments. However, the increasing number of Extended Reality (XR) paradigms introduces new technologies. Their characteristics cannot be completely represented by a single real-virtual dimension. It can be noted that XR systems not only vary in their proportion of real and virtual content but also adds, suppresses, or transforms information available to the user. This work proposes a framework for characterizing seven XR realities, namely Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), Augmented Virtuality (AV), Phenomenal Reality (PR), Annexing Reality (AnR), and Diminished Reality (DR). It uses a two-dimensional extension of the conventional 1D RV continuum. The first dimension helps in identification of the position of an XR technology between real and virtual environments. The second dimension captures information addition and subtraction. The proposed approach gives a qualitative basis for comparing existing and emerging XR paradigms within a unified XR framework.

0.25

Exploring Mixed Reality for Engagement in Robotics: A Comparison of Robot Control Methods

cs.HC ·2026-10-12 ·Samantha Edvell, Dinh Tung Le, Gavin Paul
similarity 0.62
Abstract · 10 pages. Submitted to ACRA 2026

This paper presents a Mixed Reality (MR) robotics experience that provides an accessible introduction to interacting with a physical robot. The application combines a TurtleBot with a virtual animal-tending activity and three control methods: Joystick, Waypoint, and Path-Drawing. A mixed-methods user study with 10 participants investigated changes in robotics interest and confidence following the experience and how the control methods influenced engagement, workload, and preference. Confidence increased following the experience, with the exploratory analysis indicating a significant pre--post difference, while the increase in interest was not statistically significant. The control methods produced clear differences in perceived workload and ease of use, with Path-Drawing generally more demanding and difficult to use. Joystick was preferred by eight of ten participants and was associated with better perceived performance and lower frustration. However, no statistically significant differences were detected in the other engagement dimensions, suggesting that the most preferred method was not necessarily more engaging across all aspects of the experience. These findings highlight the importance of considering both usability and engagement when designing how users interact with robots in introductory MR experiences.

0.23

Beyond Task Performance: Lessons Learned from Evaluating an Exploratory VR Interaction Technique

cs.HC ·2026-09-21 ·Nevzat Umut Demirseren, Corey Pittman, Isayas Berhe Adhanom, Karthikeyan Umapathy, Kevin Pfeil
similarity 0.61
Abstract · 6 pages, 10 figures

Dense virtual environments present significant challenges for object selection and manipulation, motivating the development of novel interaction techniques. This paper presents RodCast as an exploratory case study to investigate explicit trajectory visualization for interaction in dense virtual environments. We conducted a within-subjects user study comparing RodCast with Go-Go Hand and FlowerCone across three representative interaction tasks using objective performance measures, subjective evaluations, and qualitative feedback. The results revealed that the proposed implementation incurred performance costs on more demanding manipulation tasks, while qualitative feedback suggested that participants attributed these challenges to the complexity of the bimanual control scheme. Additionally, subjective evaluations and qualitative feedback identified benefits in spatial awareness and target accessibility that were not fully reflected by conventional performance measures. Together, these findings highlight the importance of control simplicity in interaction technique design and emphasize that incorporating qualitative feedback into the evaluation process is essential for distinguishing implementation limitations from the potential of the interaction concept. Collectively, these lessons provide guidance for the design and evaluation of future interaction techniques.

0.26

Beyond the Mirror: Balancing Interaction Modality and Avatar Fidelity in Public 3D Virtual Try-On Systems

cs.HC ·2026-08-24 ·Yueqian Guo, Tianzhao Li, Xin Lv
similarity 0.61
Abstract

Virtual Try-On (VTON) systems deployed on large public displays face a dual barrier: the physical strain of mid-air interaction and the social inhibition caused by public self-consciousness. This paper presents a real-time 3D avatar system integrating markerless motion capture with dynamic visual fidelity control to investigate and mitigate both barriers. Through a dual-study empirical evaluation, we first decoupled physical fatigue from gesture interaction ($N=20$), demonstrating that interaction fatigue is primarily driven by visuomotor latency rather than the physical act of gesturing; our optimized low-latency gesture pipeline achieved usability comparable to touchscreens while delivering superior immersion and hygiene. Building on these insights, our second study ($N=25$) investigated the "avatar fidelity paradox" via a $2 \times 2$ factorial design manipulating interaction modality (gestures vs. touch) and visual fidelity (photorealistic MetaHuman vs. stylized mannequin). Results reveal that while high fidelity and mid-air gestures independently maximize virtual embodiment ($p < .05$), their combination elicits the highest social awkwardness. Crucially, low-fidelity avatars serve as a "psychological mask" that alleviates public embarrassment during expressive gestures, while mid-air gestures simultaneously act as a compensatory mechanism to preserve perceived try-on trust despite reduced visual realism. Finally, we propose a context-aware fidelity framework to balance privacy, immersion, and commercial trust in public spatial interactions.

0.08

Multimodal Socially Aware Framework for Human-Robot Interaction in Virtual Reality

cs.RO ·2026-10-25 ·Richard Xu, Dakarai Crowder, Alexis E. Block, Wenzhen Yuan
similarity 0.61
Abstract

Social robots must coordinate speech and movement while interpreting multimodal human input. We present a multimodal framework that integrates speech, vision, and touch to generate contextually appropriate robot speech and movement. We evaluated the framework in virtual reality with 18 participants across five social interaction scenarios, comparing its performance against a human-teleoperated baseline. Although participants rated the autonomous system above the neutral midpoint for both naturalness and social appropriateness, the teleoperated baseline received significantly higher ratings on both measures. Behavioral coding and qualitative feedback revealed that conversational timing, clarification of ambiguous requests, and spatial coordination were important factors contributing to this performance gap. These findings highlight key challenges in developing autonomous robots capable of natural, multimodal social interaction.

0.18

Better Situational Awareness in AR-HRC? A Comparative Study of Augmented Reality and Mobile Interfaces for Human-Robot Collaboration

cs.HC ·2026-09-01 ·Zhehan Qu, Christian Fronk, Jaewoong Jeong, Pavel Manakhov, Maria Gorlatova
similarity 0.61
Abstract · 11 pages, 9 figures, 1 table. Zhehan Qu and Christian Fronk contributed equally

Augmented reality (AR) facilitates human-robot collaboration (HRC) by enabling in-situ spatial visualizations of the robot and the joint task. However, in safety-critical HRC scenarios such as search-and-rescue, spatial visualizations may also reshape visual attention in ways that create competing situational awareness (SA) demands, potentially introducing new safety concerns. While prior AR-HRC work suggests potential benefits for SA, rigorous evaluations that jointly consider robot and environmental awareness across multiple levels of SA remain limited. We address this through a between-subjects study with 30 participants comparing custom AR and mobile interfaces presenting equivalent information, measuring robot and environmental SA with the Situation Awareness Global Assessment Technique (SAGAT) across all three levels, with concurrent eye tracking to identify the attentional mechanisms underlying any SA differences. Both interfaces achieved high usability; relative to the mobile baseline, AR improved perception-level awareness of the robot but yielded no gains in higher-level robot awareness or in environmental awareness at any level. Gaze analysis explained this: AR freed attention from the map, but that attention was re-invested in the conformal visuals rather than the physical environment. Freeing the eyes from a screen is not the same as directing them to the world, a distinction AR interfaces for safety-critical HRC must design around.

0.30

CognitiveReality: Robot-Agnostic Semantic Gaussian Mapping with an LLM Agent for Immersive Collaborative VR Teleoperation

cs.RO ·2026-09-25 ·Timofei Kozlov, Dmitrii Maliukov, Andrey Marchenko, Dmitrii Plotnikov, Miguel Altamirano Cabrera +1 more
similarity 0.61
Abstract

A photorealistic 3D view tells a teleoperator where a robot is, but not what the scene contains, how well each object has been observed, or how to turn pointing and speech into robot action. CognitiveReality turns a robot's RGB-D stream into a live, semantically indexed Gaussian-TSDF map shared by an operator in virtual reality and a tool-using language agent. One mapper binary serves any platform through configuration alone: it ingests poses from robot SLAM, joint kinematics, motion capture or an inline visual tracker, bridges localization outages with a shadow tracker and keyframe-anchored PnP, and maintains open-vocabulary instance identities with per-object quality at 2 Hz. Speech and controller rays are grounded against persistent scene objects through validated typed tools and operator-confirmed robot actions. In the controlled agent evaluation, the deployed local Qwen3-VL-8B router reaches 81.24\% tool exact match, while merge-aware replay correctly redirects 101 absorbed object identifiers. On robot data CognitiveReality exceeds a Gaussian-plus-SDF baseline by 2-8 dB; pose error through 5-40 s SLAM outages stays within 1-8 cm. Deployed live on two quadrupeds, the agent executed 26 of 30 navigation requests and 20 of 20 re-observation requests, raising object quality by 2-5 dB.

0.25