CS Paper Radar

CARLAverse: A Highly Modular, Distributed, and Multimodal Framework for Human-in-the-Loop Simulation

Patrick Rebling, Philipp Nenninger, Reiner Kriesten

cs.RO cs.HC ·submitted 2026-09-10 12:44 UTC · v1
code/data

Abstract

The development of autonomous driving demands comprehensive testing in mixed-traffic scenarios involving vulnerable road users (VRUs), where purely artificial agents often fail to capture authentic human social negotiations. While human-in-the-loop (HITL) simulators enable safe investigation of these interactions, existing multi-agent platforms struggle with the network latency and synchronization constraints required for high-fidelity haptic feedback. To resolve this, we present CARLAverse, an open-source, multimodal simulation ecosystem. Extending modular hardware abstraction, CARLAverse integrates driving (DrivoCARLA), cycling (CycloCARLA), and pedestrian (WalkoCARLA) simulators into a shared virtual environment. Its core methodological contribution is a distributed physics architecture: latency-critical ego dynamics and high-frequency force feedback are computed locally on client nodes, while a central CARLA server orchestrates non-player character (NPC) physics and global traffic. By decoupling haptic control loops from network bottlenecks, CARLAverse enables scalable, cross-institutional HITL experiments without compromising physical immersion. Code and documentation: https://git.ieem-ka.de/simulator-environments/carlaverse

Comment
19 pages, 13 figures, Code available at https://git.ieem-ka.de/simulator-environments/carlaverse

Closest papers

Nearest neighbours by meaning across the whole index.

TransCAVE-E: A distributed virtual reality testbed for adaptive external human-machine interfaces

cs.HC ·2026-08-30 ·Yun Ye, Zexuan Li, Haoyang Liang, Boya Sun, Jian Sun +1 more
similarity 0.68
Abstract

External human-machine interfaces (eHMIs) are evolving from predefined displays toward adaptive communication strategies that respond to changing traffic and road-user states. This transition requires experimental infrastructure that supports human-in-the-loop (HIL) interaction, software-in-the-loop (SIL) algorithm execution, reusable experiment orchestration, and synchronized multimodal human-factors evaluation. This paper presents TransCAVE-E, a distributed virtual-reality testbed for developing and validating adaptive and intelligent eHMIs. The platform integrates three coupled modules: a Scenario Design Center for configurable traffic environments and experimental conditions; an eHMI Algorithm Module for bidirectional real-time coupling between simulation and external algorithms; and a Data Management System that synchronizes trajectories, eye-tracking, physiological, system-log, and subjective data. A distributed multi-agent architecture supports synchronous interaction among pedestrians, human drivers, automated vehicles, and other traffic entities. Two use cases demonstrate the platform. In an AV-pedestrian experiment, an intent-recognition-based eHMI improved decision efficiency by 12.8% and 13.0% in yielding and non-yielding scenarios, reduced gaze distraction by 17.1% in the yielding scenario, and reduced unnecessary prompts by 40% in the non-yielding scenario while maintaining interaction safety. An HV-AV study further demonstrated real-time SIL validation of a game-theoretic information-disclosure strategy under active driver interaction. TransCAVE-E provides an extensible and reproducible infrastructure for closed-loop evaluation and iterative refinement of intelligent human-vehicle communication strategies.

0.32

PCLA-Bridge: A Framework for Cross-Domain Deployment of CARLA Agents

cs.SE cs.RO ·2026-10-20 ·Masoud Jamshidiyan Tehrani, Jinhan Kim, Paolo Tonella
benchmark similarity 0.66
Abstract

Most high-performing end-to-end autonomous driving agents (ADAs) are developed for the CARLA Leaderboard, tightly coupling them to CARLA's APIs, sensor layouts, coordinate systems, and internal clock. Reusing these models on real datasets, alternative simulators, or physical platforms necessitates substantial manual re-engineering. We present PCLA-Bridge, a deployable tool that runs unmodified CARLA Leaderboard agents from any sensor source to any control destination without an active CARLA server. With PCLA-Bridge handling dependency isolation, tensor marshalling, routing, and synthetic timing, integrators only need to provide a minimal interface. We validate the tool across two operational paradigms: offline replay on in-domain CARLA recordings and real-world nuScenes logs, and live closed-loop control across three diverse simulation engines: MetaDrive, AWSIM, and AlpaSim. We analyze the lightweight integration requirements for each platform, highlighting output translation (such as converting agent actions into target trajectory formats) and coordinate-frame transformations, showing that integration requires minimal glue code rather than structural changes. All eight evaluated agents execute across all three simulators without modification, confirming the reliability of PCLA-Bridge and establishing a foundation for systematic sim-to-real, cross-sim, and security testing.

0.16

VehicleArena: A Realistic Urban Environment for Multi-Agent Driving

cs.MA cs.CLcs.CV ·2026-09-28 ·Jie Yang, Jiajun Chen, Jiazheng Zhou, Mianqiu Huang, Yining Zheng +2 more
benchmark similarity 0.64
Abstract

Real-world embodied agents often pursue independent objectives within a shared physical environment, where their actions can alter the conditions faced by others. Existing benchmarks, however, typically assume shared goals or explicitly prescribed interaction protocols, leaving such emergent physical coupling underexplored. We introduce VehicleArena, a 3D urban-driving benchmark for studying independently operating agents in a dynamic shared world. In VehicleArena, LLM-controlled agents must fulfill evolving passenger requests while navigating complex traffic, and each agent's driving decisions can reshape traffic flow, delays, risks, and subsequent observations for surrounding agents. The benchmark provides 112 evaluation tasks spanning single-agent and multi-agent driving. Across nine evaluated models, the highest arrival rates reach only 65.0% on single-agent tasks and 65.6% on multi-agent tasks, while strong passenger-request or cabin scores do not reliably translate into successful trip completion. Moreover, in matched multi-agent runs, every tested focal policy reduces the arrival rate of surrounding vehicles relative to the simulator's native traffic controller, revealing measurable externalities beyond the focal vehicle itself.

0.14

Complete Street Design informed by Multi-Agent Simulation

cs.MA ·2026-10-20 ·Sharika J. Hegde, Amirmohammad Khakpour, Pedram Beigi, Yu, Nie +1 more
similarity 0.63
Abstract · Under review for publication in Transportation Research Record (TRR)

Objectives: Complete Street design is currently guided largely by qualitative principles despite the increasing complexity of multimodal urban traffic. This paper develops a microscopic simulation framework for quantitatively evaluating alternative street designs by explicitly representing heterogeneous road users, including Autonomous Vehicles (AVs), Connected Vehicles (CVs), Human-Driven Vehicles (HDVs), pick-up and drop-off (PUDO) services, and micromobility users. The framework jointly evaluates mobility and safety within a unified testbed. Methods:The framework enriches microscopic simulation with behavioral models for heterogeneous road users. It explicitly captures lateral movements, parking maneuvers, pedestrian crossings and jaywalking, and shared-lane bicycle and scooter interactions. Safety is quantified using a conflict severity measure grounded in collision physics, while candidate street layouts are optimized using a genetic algorithm. Findings: Applied to a segment of Belmont Avenue in Chicago, Illinois, the framework shows that dedicated bicycle lanes reduce conflict severity at the expense of vehicular throughput. Increasing AV penetration mitigates congestion by reducing traffic shockwaves while improving interactions with pedestrians relative to CVs and HDVs. The optimization framework efficiently identifies Pareto-efficient street layouts that balance mobility and safety. Novelty: The proposed framework integrates heterogeneous behavioral modeling, physics-based conflict assessment, stochastic performance evaluation, and design optimization within a single quantitative framework for Complete Street design. Practical Applications: The framework provides transportation planners with a predictive virtual testbed for evaluating street designs and quantifying the tradeoff between mobility and safety before implementation.

0.39

SHRED: Self-Play for Heterogeneous Road-Users without Expert Demonstrations

cs.LG cs.AIcs.RO ·2026-10-14 ·Wei-Jer Chang, Zhouchonghao Wu, Akshay Rangesh, Weixin Li, Masayoshi Tomizuka +1 more
benchmark similarity 0.62
Abstract · Self-play for Autonomous Driving

Realistic traffic simulation and autonomous planning both depend on modeling interactions among heterogeneous road users whose dynamics, constraints, and behaviors differ fundamentally. Existing approaches predominantly learn realistic interaction behavior from large-scale human driving demonstrations. In this work, we ask whether such behavior can instead emerge purely through self-play. We present SHRED, a framework that jointly trains vehicles, pedestrians, and cyclists through large-scale heterogeneous self-play from random initialization, without demonstrations or logged-trajectory replay. SHRED combines type-specific dynamics and objectives with procedural scenario generation and per-agent randomization, allowing diverse behaviors to emerge through closed-loop interaction. On the Waymo Open Sim Agents Challenge (WOSAC), SHRED jointly controls all three road-user types and outperforms both the closest demonstration-free self-play baseline and prior self-play methods that use demonstration-trained reference policies. Policies trained only on nuPlan map geometry retain comparable realism under zero-shot evaluation on Waymo. For planning, SHRED outperforms all compared planners on the interaction-heavy InterPlan benchmark. Controlled ablations show that heterogeneity in self-play improves both traffic realism and planning performance, with particularly large planning gains in interaction-heavy scenarios. Together, these results establish heterogeneous procedural self-play as a scalable, demonstration-free paradigm for learning realistic and interactive traffic behavior.

0.15

VirTooS: A ROS 2 - Unity Virtualization Toolkit for Fleet Management of Autonomous Mobile Robots

cs.RO ·2026-08-26 ·Andrea Drudi, Lorenzo Pichierri, Andrea Testa, Giuseppe Notarstefano
code/data similarity 0.61
Abstract

In this paper, we present VirTooS, a Python/C# toolkit designed to implement fleet-management tasks on teams of Autonomous Mobile Robots (AMRs). VirTooS leverages the Robot Operating System (ROS) 2 and Unity game engine to provide realistic, scalable virtual experiments in a mixed-reality environment. The toolbox allows users to easily generate and customize virtual scenarios for realistic simulations. Virtual and real sensors as, e.g., LiDARs, can be exploited to map and safely navigate in the mixed-reality environment. To enable distributed robotics experiments, we propose a set of tailored routines leveraging the ChoiRbot framework. As a motivating example, we show a set of experiments for task assignment problems in a virtual environment, allowing seamless interaction among real and virtual robots. Moreover, the package comes with a containerized suite to easily deploy it on different machines. The source code will be made publicly available on GitHub.

0.38

HODAgent: Towards On-Demand, Responsive Humanoids for Physical World Human Interaction

cs.RO ·2026-08-18 ·Wang Warren Chen, Jiahao Zhang, Zhenjiang Li, Mingxu Wang, Lei Yi +4 more
similarity 0.60
Abstract · we have received a formal directive from our company requiring all company assets to undergo a mandatory internal review process before any …

We propose HODAgent, a System-2 embodied agent for humanoid robots in service settings, addressing situated intent, responsive execution, task revision, and outcome verification. Its semi-duplex architecture integrates an Env-Interactor, Planner, Executor, and hierarchical Memory to maintain coherent interaction, planning, and task state during service episodes. This allows handling new requests during motion, retaining progress, revising actions, and grounding closure in execution outcomes. A shared interface connects simulation and physical robots (Unitree G1), isolating platform-specific control. In an interactive simulation with 164 cases, HODAgent achieves 84.8% and 91.5% Joint Success under two VLM backbones, outperforming baselines by 9.8 and 18.9 points. On physical robots, pass rates are 92% (atomic), 72% (composite), and 63.3% (complete tasks). On multiple embodied benchmarks, it improves over baselines by 0.7-9.0 points. Results show a unified System-2 agent enables adaptive humanoid service across simulation and reality.

0.19

Interactive World Simulation with Cosmos for Physical AI

cs.AI cs.CVcs.LGcs.RO ·2026-11-04 ·NVIDIA, :, Anis Ahmad, Jon Allen, Malik Aqeel Anwar +85 more
code/data similarity 0.60
Abstract · Project website: https://research.nvidia.com/labs/cosmos-lab/cosmos-simulation

We present a shared framework that adapts Cosmos world foundation models for real-time interactive world simulation in Physical AI. By streaming visual observations in response to actions and scene controls, the framework enables users and embodied agents to interact with evolving virtual environments. It supports both predicting the visual consequences of actions and generating observations from physical state maintained by an external simulator. We demonstrate this approach across camera navigation and robot manipulation, controllable video transfer, and multi-view autonomous vehicle (AV) simulation. These examples provide a shared foundation for developing interactive virtual environments for Physical AI training and evaluation. Our code and model checkpoints are available at https://github.com/nvidia/cosmos and https://huggingface.co/collections/nvidia/cosmos3-examples. The project website is available at https://research.nvidia.com/labs/cosmos-lab/cosmos-simulation.

0.25