CS Paper Radar

TransCAVE-E: A distributed virtual reality testbed for adaptive external human-machine interfaces

Yun Ye, Zexuan Li, Haoyang Liang, Boya Sun, Jian Sun, Haotian Shi

cs.HC ·submitted 2026-08-30 22:13 UTC · v1

Abstract

External human-machine interfaces (eHMIs) are evolving from predefined displays toward adaptive communication strategies that respond to changing traffic and road-user states. This transition requires experimental infrastructure that supports human-in-the-loop (HIL) interaction, software-in-the-loop (SIL) algorithm execution, reusable experiment orchestration, and synchronized multimodal human-factors evaluation. This paper presents TransCAVE-E, a distributed virtual-reality testbed for developing and validating adaptive and intelligent eHMIs. The platform integrates three coupled modules: a Scenario Design Center for configurable traffic environments and experimental conditions; an eHMI Algorithm Module for bidirectional real-time coupling between simulation and external algorithms; and a Data Management System that synchronizes trajectories, eye-tracking, physiological, system-log, and subjective data. A distributed multi-agent architecture supports synchronous interaction among pedestrians, human drivers, automated vehicles, and other traffic entities. Two use cases demonstrate the platform. In an AV-pedestrian experiment, an intent-recognition-based eHMI improved decision efficiency by 12.8% and 13.0% in yielding and non-yielding scenarios, reduced gaze distraction by 17.1% in the yielding scenario, and reduced unnecessary prompts by 40% in the non-yielding scenario while maintaining interaction safety. An HV-AV study further demonstrated real-time SIL validation of a game-theoretic information-disclosure strategy under active driver interaction. TransCAVE-E provides an extensible and reproducible infrastructure for closed-loop evaluation and iterative refinement of intelligent human-vehicle communication strategies.

Closest papers

Nearest neighbours by meaning across the whole index.

CARLAverse: A Highly Modular, Distributed, and Multimodal Framework for Human-in-the-Loop Simulation

cs.RO cs.HC ·2026-09-10 ·Patrick Rebling, Philipp Nenninger, Reiner Kriesten
code/data similarity 0.68
Abstract · 19 pages, 13 figures, Code available at https://git.ieem-ka.de/simulator-environments/carlaverse

The development of autonomous driving demands comprehensive testing in mixed-traffic scenarios involving vulnerable road users (VRUs), where purely artificial agents often fail to capture authentic human social negotiations. While human-in-the-loop (HITL) simulators enable safe investigation of these interactions, existing multi-agent platforms struggle with the network latency and synchronization constraints required for high-fidelity haptic feedback. To resolve this, we present CARLAverse, an open-source, multimodal simulation ecosystem. Extending modular hardware abstraction, CARLAverse integrates driving (DrivoCARLA), cycling (CycloCARLA), and pedestrian (WalkoCARLA) simulators into a shared virtual environment. Its core methodological contribution is a distributed physics architecture: latency-critical ego dynamics and high-frequency force feedback are computed locally on client nodes, while a central CARLA server orchestrates non-player character (NPC) physics and global traffic. By decoupling haptic control loops from network bottlenecks, CARLAverse enables scalable, cross-institutional HITL experiments without compromising physical immersion. Code and documentation: https://git.ieem-ka.de/simulator-environments/carlaverse

0.24

InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

cs.HC cs.CV ·2026-10-25 ·Qinsi Wang, Wei Wen, Chong Zhou, Saksham Suri, Zechun Liu +5 more
benchmarkcommunity pick similarity 0.62
Abstract

People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered by multimodal large language models (MLLMs), egocentric AI assistants on wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user's physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered: Can today's MLLMs truly help people in physical world? To answer this question, we propose InteractiveWorld, the first verifiable environment for closed-loop interaction between human and AI assistant. Unlike existing robot-centric simulators, InteractiveWorld lets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they see next. Building on it, we introduce InteractiveBench, the first dynamic benchmark for egocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable, state-based rewards. Experiments with 12 frontier MLLMs show they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistants that truly help people in the physical world.

0.08

Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

cs.CV ·2026-11-05 ·Chenhao Zheng, Zixian Ma, Kai Li, Kanchana Ranasinghe, Jieyu Zhang +9 more
benchmark similarity 0.61
Abstract · 39 pages, including references and appendices

Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Claude Opus 5.5, completes only 53.5% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment.

0.27

InterSim: Innovative Intersection Digital Twins for Human-in-the-Loop Driving Simulation

cs.CE cs.CYcs.ET ·2026-11-04 ·Luyang Jiang, Chuheng Wei, Yaguang Li, Shaoqi Wang, Yupeng Zhou +4 more
code/data similarity 0.60
Abstract

Innovative intersection designs can improve roadway safety and operational efficiency, but their uncommon geometries may create challenges for driver comprehension during early deployment. Evaluating these designs through full-scale field implementation can also be costly. This paper presents InterSim, a Unity game engine-based human-in-the-loop (HITL) driving simulator that constructs digital twins of site-specific innovative intersections in a virtual environment. InterSim links the real world and the simulation world through two closed cycles: a digital twin cycle that builds each intersection twin from agency design and traffic data and returns evidence from simulation to the agency, and an HITL cycle in which human drivers or automated driving algorithms operate the ego vehicle under identical scenario conditions. The research capabilities of InterSim are demonstrated through two case studies: a surrogate safety study compares post-encroachment time distributions between a Reduced Conflict Intersection and a standard intersection under different traffic volumes; an eco-driving study compares human drivers with a rule-based agent using vehicle-to-infrastructure information and reveals differences in emission and driving behavior. These studies demonstrate that InterSim provides a reusable digital twin platform for investigating innovative intersection designs from diverse perspectives. Demo video and open-source code can be accessed at https://purduedigitaltwin.github.io/InterSim/

0.27

MIVAIS: A Study Environment for Multi-Agent Mixed-Initiative Visual Analytics Applications

cs.HC cs.MA ·2026-09-04 ·Tobias Stähle, Simon Schneider, Rita Sevastjanova, Mennatallah El-Assady
similarity 0.60
Abstract

Mixed-initiative Visual Analytics (VA) systems empower human users by interleaving human intuition with software agents and their machine intelligence. However, the development and rigorous evaluation of such systems remain constrained by engineering overhead. Developers must, e.g., implement complex, low-level state synchronization to manage asynchronous agent behaviors, while researchers struggle to capture the multimodal provenance required to study and evaluate human-AI collaboration. We present MIVAIS, a dual-layered research platform designed to abstract the structural complexities of mixed-initiative VA. First, it contributes a computational Infrastructure that standardizes human-software agent interaction, state synchronization, and communication between the agents. Second, it provides a declarative Study Environment that automatically logs multimodal human-AI telemetry - including application/system state, screen capture, audio, and additional sensor data - enabling seamless, in-situ user studies and post-session analysis. We technically validate our infrastructure by replicating three state-of-the-art systems (Podium, Voyager 2, and ProactiveVA). Furthermore, we evaluate the framework's expressiveness and efficiency through expert case studies with HCI and VA researchers, demonstrating how MIVAIS effectively lowers the barrier to prototyping and evaluating intelligent, co-adaptive interfaces.

0.19

Evaluating the Impact of Adaptive Extended Reality on Human-Robot Interaction Across the Reality-Virtuality Continuum

cs.RO cs.HC ·2026-09-25 ·Carl Tornberg, Alicia Torck, Lotfi El Hafi, Tadahiro Taniguchi
code/data similarity 0.59
Abstract · Submitted to Advanced Robotics, Special Issue on "Next Generation Cognitive Robotics: Nurturing Embodied Intelligence for a Symbiotic Future…

As populations in developed countries age and labor shortages intensify, Cybernetic Avatars (CAs) are proposed to extend human capabilities through robotic embodiments, requiring effective Human-Robot Interaction (HRI) frameworks. Extended Reality (XR), an umbrella term for Augmented Reality (AR), Augmented Virtuality (AV), and Virtual Reality (VR), offers such interfaces, but prior research typically fixes the XR modality without evaluating its effect on task outcomes. This study examines whether the XR modality impacts HRI performance and whether an adaptive interface adjusting the level of virtuality along the Reality-Virtuality Continuum (RVC) at runtime improves it. A custom XR application interfaced with a mobile manipulator supports immersive control and runtime modality switching. In a within-participant multi-room pick-and-place experiment comparing fixed AR, AV, and VR with dynamic RVC through task metrics, the NASA-TLX, and the System Usability Scale (SUS), this study demonstrates that 1) the fixed reality modality affects HRI results, and 2) dynamically changing the modality along the RVC improves them. AR yielded significantly lower mental demand, effort, and frustration than AV and VR, while the dynamic RVC condition achieved the highest throughput and lowest workload, highlighting the value of adaptive XR interfaces for human-robot symbiosis. The implementation is available at https://github.com/CarlTornberg/XR-HRI.

0.58

Occlusion-induced risk and interventions in pedestrian-autonomous truck interactions on multi-lane roads: A virtual reality study

cs.HC ·2026-08-30 ·Yun Ye, Yuan Che, S. C. Wong, Stergios-Aristoteles Mitoulis, Haoyang Liang
similarity 0.57
Abstract

Autonomous trucks (ATs) may introduce distinct pedestrian-safety risks because of their large physical dimensions, constrained braking capability, limited driver-based communication cues, and potential to occlude surrounding traffic. This study employed a controlled virtual reality experiment with 54 participants to investigate pedestrian-AT interaction risk in an unsignalized multi-lane crossing scenario and to evaluate occlusion-targeted risk mitigation strategies. The experiment examined the effects of near-side vehicle type, weather condition, and far-side vehicle yielding strategy on pedestrian behavior, perceived risk, and objective safety. Based on a representative high-risk scenario, three targeted interventions were designed and tested: an environment-aware external human-machine interface (eHMI), a projected eHMI, and an auditory warning. The results showed that ATs increased perceived risk and encouraged more cautious crossing behavior, suggesting a risk-compensation effect. However, this compensation was weakened under rainy conditions, where braking-related safety margins were reduced. AT-induced occlusion further increased far-side interaction risk by limiting pedestrians' recognition of hidden vehicles. Among the three interventions, the projected eHMI showed the best overall performance, improving objective safety margins, enhancing risk awareness, and supporting behavioral adjustment. These findings highlight the need for AT-specific interface and warning strategies that address both intention communication and risk localization.

0.23

A Low-Latency Interactive System for Real-Time Video Understanding Based on VLMs

cs.MM cs.MA ·2026-09-12 ·Punan Dai, Jun Xu, Bingcong Lu, Zhengxue Cheng, Hongwei Hu +2 more
hot direction similarity 0.57
Abstract · 12 pages. Submitted to IBC 2026

Vision-language models are extending video understanding from offline clip analysis to continuous interactive streaming, but most research still emphasizes model capability rather than deployable low-latency interaction. This paper presents a unified edge-cloud system for real-time video VLM applications. Lightweight phone, smart glasses, PC, and pseudo-replay clients publish video and speech to a server runtime that provides shared ASR/TTS, session orchestration, backend adaptation, response delivery, and archive-backed measurement. The system integrates six representative video VLM backends with streaming or interaction-oriented capabilities and evaluates them across backend runtime, media transport, client-observed latency, and interaction behavior. With suitable backend selection and the WebRTC path, the tested system reaches approximately 0.9 to 1.0 s to first VLM text and 1.3 to 1.5 s to first non-silent TTS audio, while exposing backend adaptation costs and differences in real-time interaction behavior.

0.42