CS Paper Radar

ROSTrans: Empirically Benchmarking LLMs for ROS1-to-ROS2 Migration

Yichen Zhao, Minghao Shao, Zhuoyan Yu, Yijun Shen, Boyuan Chen, Yik-Cheung Tam, Muhammad Shafique

cs.SE cs.AI ·submitted 2026-10-30 02:44 UTC · v1
benchmarkcode/data

Abstract

Although large language models have shown strong capabilities in domain-specific code generation, their performance in robotics software development, especially for the Robot Operating System (ROS), remains largely unexplored. ROS code generation presents distinctive challenges, including complex inter-node coordination, middleware configuration, and lifecycle semantics that go far beyond standard coding benchmarks, and many important correctness properties become observable only once migrated nodes are launched and communicate. We present ROSTrans, a benchmark for evaluating LLMs on ROS1-to-ROS2 code migration, a representative and practically important setting for ROS code generation. ROSTrans comprises 108 scaffolded completion tasks drawn from a diverse set of 70+ open-source ROS1 repositories and organized into a hierarchical taxonomy assessing interface-level, behavioral-level and system-level correctness. For every task we establish an executable reference: a verified ROS2 implementation, a runtime test suite that launches the nodes and exercises their topics, services, actions and parameters, and a container ensuring reproducibility. Candidates are scored by injecting only their target files into an otherwise fixed copy of the reference package, so a candidate can neither supply nor weaken the tests it is judged by. Evaluating six frontier LLMs over five stochastic generations per task, we find pass@1 below 10.9%, and that 56.5% of all samples build successfully and then fail at runtime,and 32.0% fail runtime assertions on the migrated interfaces, a class of error no compile-only oracle can detect. Execution-guided repair further improves task success, reaching 49.1% and 47.2% for Claude Opus 4.6 and GPT-5.5 after three rounds, respectively.

Closest papers

Nearest neighbours by meaning across the whole index.

A SysML v2 Domain Library for ROS2 Robot Architectures

eess.SY ·2026-10-23 ·Sai Sandeep Damera, John S. Baras
similarity 0.74
Abstract

Systems engineers who model a ROS2 robot in SysML v2 must first model ROS2 itself. Its nodes, communication endpoints, message types, lifecycle states, quality-of-service policies, and coordinate frames have no ready definitions in the language, so each project reconstructs them. We present ros2-sysmlv2, a native SysML v2 domain library that supplies these definitions for reuse. A robot model imports the library and specializes its definitions to declare its own nodes, ports, connections, frames, and parameters. The library maps ROS2 concepts to SysML v2 definitions and usages and organizes them in Primitives, Infrastructure, and Application layers. We check its declarations for parity with the ROS2 Jazzy sources they model. Because every ROS2 element of a model specializes a library definition, a tool can select those elements by type. We state what this selection is and use it for a round trip. A ROS2 package is generated from the selection, and a monitor generated with the package compares the running graph with the same selection and reports every departure against the model. The generation-gap workflow derives topic endpoints and parameter declarations while preserving developer-owned node algorithms. In a case study on a generated ground robot, the generated conformance monitor reported each of the three departures that we introduced in the implementation. The library is distributed as a versioned package that any SysML v2 model can import.

0.27

Robo-SWE: An Executable Issue-Resolution Dataset Mined from Robot Software Repositories

cs.SE cs.RO ·2026-11-11 ·Vahid Haratian, Vahid Ansari, Mohammadhossein Malmir, Alois Knoll
benchmarkdatasetcode/data similarity 0.69
Abstract · 5 pages, 1 figure, 2 tables. Submitted to the MSR 2027 Data and Tool Showcase Track. Dataset: https://huggingface.co/datasets/vharatian/Robo…

Coding agents are evaluated mostly on general-purpose software, but domain-specific systems require domain-aware evaluation. Robot software is a particularly challenging case: Robot Operating System 2 (ROS 2) packages have their own build tools, simulators and learning frameworks are built on physics engines, and behavior that depends on timing and hardware is hard to test. We present Robo-SWE, a dataset of 236 executable issue-resolution tasks mined from merged pull requests in 28 public robotics repositories in four domains: ROS 2 systems, simulation and physics engines, robotics algorithm libraries, and robot-learning frameworks. Each task contains the original issue text, a container recipe for the repository at the commit before the fix, the developers' fix, and the developers' tests, which fail before the fix and pass after it in repeated offline runs. The dataset also keeps a provenance ledger of all 1,567 candidate pull requests, with the stage each reached and the reason for every rejection, and, as a first use, 3,540 graded attempts by five coding-agent setups with recorded patches and 3,529 available trajectories. Across three runs per primary setup, resolve rates range from 45.8% for Claude Sonnet 5.5 to 56.5% for Claude Opus 5.5; GPT-6.1 Sol reaches 54.1% at maximum effort and 52.5% at medium effort, and GPT-6 Astra 51.4%. GLM-5.3 resolves 46.2%, and Kimi K3.1 Pro 43.9%. The tasks are validated by execution; verification of their fairness by human reviewers is future work. The dataset and public leaderboard are available online.

0.53

ROS2SmolVLA: Enabling Small Vision-Language-Action Models for Integration into Industrial-Grade Lightweight Robots

cs.RO ·2026-08-24 ·Nils Mandischer, Noah Böckmann, Ludwig Holl, Lars Mikelsons
publishedcode/data similarity 0.67
Abstract · Accepted at 8th International Conference on Industry of the Future and Smart Manufacturing, Padua & Venice, 2026

Industrial demand changes the paradigms of production. Due to smaller batch sizes and more variations in products, companies face a growing challenge to adopt more adaptive production systems. In particular, robot-based automation is usually static and fails to respond to constantly changing processes. Vision-Language-Action (VLA) Models are a promising opportunity to mitigate this challenge by generating robot actions based on the observed system state. However, current research either focuses on large models that cannot be computed on premise, creating compliance and security challenges, or use lab-grade robot hardware that obscures exploitation in real industrial settings. In this work, we adapt Hugging Face's SmolVLA for Universal Robots lightweight robots. Further, we release the open-source repository ROS2SmolVLA that implements an interface for ROS 2 to SmolVLA, and makes it applicable for industrial-grade hardware. By this, we allow a lenient adoption into lab and industrial environments. We validate the functionality of SmolVLA for a Universal Robots UR10e using a pick-and-place task and give implementation guidelines. Our findings support that SmolVLA is a well-suited option for small-sized tasks that need to be computed on premise.

0.56

Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework

cs.RO cs.AI ·2026-09-11 ·Kazi Abrar Mahmud, Nilotpaul Kundu Dhurubo, Tamal Kirttonia, Sabbir Hossain Ujjal, Mohammad Ariful Haque
code/data similarity 0.67
Abstract · 8 pages, 6 figures, Submitted to IEEE Transactions on Cognitive and Developmental Systems (TCDS)

Large Language Models (LLMs) have enabled more natural human-robot interaction, but open-source models often exhibit unstable long-horizon reasoning and inefficient action execution when deployed in agentic robotic frameworks. This paper presents an enhanced ROS-Agent based architecture that improves task reliability and execution efficiency for agentic robotic systems using open-source LLMs. The proposed system introduces a novel intermediate mechanism, termed the MetaTool, which enforces structured planning prior to action execution. Given a natural-language command, the MetaTool induces the LLM to generate a pseudo-code plan of intended tool invocations, which is stored in the ROS-Agent's scratchpad and persists throughout execution. By explicitly separating planning from execution, the proposed approach reduces execution loops and improves deterministic behavior. The architecture is validated on a custom mobile robotic platform with multimodal perception and motion control capabilities. Experimental results on real-world interactive tasks demonstrate improved task completion and contextual consistency, with up to ~24% gains on complex tasks compared to the baseline framework.

0.22

ROSMonitoring in Action: Runtime Monitoring and Enforcement for ROS 2 Actions

cs.AI cs.LOcs.RO ·2026-11-17 ·Giacomo Paltrinieri, Angelo Ferrando
published similarity 0.66
Abstract · In Proceedings FMAS 2026, arXiv:2611.21375

Robotic applications often rely on long-running operations, such as navigation, manipulation, and inspection, which in ROS~2 are commonly implemented as actions. Unlike topics and services, actions involve a structured lifecycle comprising goal submission and acceptance, feedback, status updates, result retrieval, and cancellation. Existing versions of ROSMonitoring support topics and services, but do not reconstruct this lifecycle or expose action-level events to a runtime-verification oracle. In this paper, we extend ROSMonitoring with runtime monitoring and enforcement for ROS~2 actions. Actions are declared directly in the monitoring configuration and automatically instrumented through the underlying ROS~2 communication endpoints. The generated monitor reconstructs action-level events, correlates them through goal identifiers, maintains the state of active goals, and translates violations into protocol-aware enforcement operations. These include goal rejection, cancellation, message suppression, and protocol-compliant responses generated directly by the monitor when the original communication cannot be forwarded. We evaluate the implementation through controlled stress tests that vary the number of action invocations, workload size, and client–server topology. The results show approximately linear growth in monitoring cost with repeated action invocations under a fixed workload, while larger action workloads and a greater number of configured client–server pairs introduce additional overhead. These results characterise the scaling behaviour of the current implementation under the tested sequential workloads while highlighting the main performance limitations of the current synchronous implementation.

0.35

OpenHarmony Bench: Evaluating LLMs and Coding Agents on OpenHarmony App Development

cs.SE ·2026-08-17 ·Li Li, Han Hu, Tianjian Zhang, Xin Peng, Fangzhu Mao +25 more
benchmark similarity 0.65
Abstract

We present OPENHARMONY BENCH, an app-level coding benchmark for evaluating LLM-based coding agents on OpenHarmony ArkTS applications. Unlike function-level benchmarks, it evaluates complete app-level changes: each task requires an agent to modify a buildable ArkTS project so that a requested behavior works end to end, involving UI state, data persistence, build configuration, and platform APIs. The benchmark installs and drives the delivered application on a device to check whether the behavior is observable. It covers three input sources: natural-language feature requests (new-feature), structured scenario specifications (spec-driven), and bug descriptions (bug-fix). The benchmark contains 153 top-level tasks and 242 Feature points (F-points), where an F-point is one executable behavior check. The snapshot includes 32 new-feature tasks, 50 spec-driven tasks with 139 F-points, and 71 bug-fix tasks. The main leaderboard is scored over top-level tasks rather than independently weighted F-points. We describe the benchmark construction, statistics, and build-and-test evaluation pipeline, and evaluate DevEco Code with eight LLMs across three independent full-suite runs per configuration. Three findings emerge. First, newer generations complete more tasks than their predecessors within evaluated model-family pairs. Second, buildability is close to saturated while behavioral correctness is not: mean Final Build Success Rate is 94.77% to 100.00%, whereas mean Task Completion is 48.36% to 58.39%. Third, spec-driven tasks have the lowest Task Completion under all-checks task scoring, with no configuration exceeding 35%. The code, data, tasks, reference solutions, tests, evaluation scripts, and leaderboard are released through the official OPENHARMONY BENCH website at https://bench.matrix.openharmony.cn/.

0.24

RoboBugBench: Benchmarking Automated Repair of Real Robot Software Under an Audited Behavioral Oracle

cs.RO ·2026-10-12 ·Alexey Tuzikov
publishedbenchmark similarity 0.65
Abstract · 14 pages: 9 main + references + two appendices. 1 figure, 4 tables. Accepted non-archival workshop paper at the Verify-Agents workshop ('Who…

Language agents increasingly modify robot software, but whether they can repair it is unmeasured, because the domain supplies no oracle to inherit: a robot bug manifests as behavior, and robot repositories ship almost no tests that capture it. We present RoboBugBench, a benchmark of 33 real, execution-validated defects in a pinned ROS 2 navigation stack, in which a candidate patch is graded by compiling it and driving the robot through a physics simulation rather than by running a unit test. Three frontier models, each given 1 attempt per instance, repair 24.2%-36.4%, with no pairwise difference significant; difficulty tracks the oracle rather than the domain, at 66.7% on instances graded by process survival against 22.2% on those graded by what the robot did (instance-level p = 0.0027). Because the oracle is constructed rather than inherited we audit it adversarially, and report the decomposition that audit yields: counterfeit patches which suppress a failure without repairing the defect are accepted by the shipped rule 11/11, while a distributional rule over the same data rejects 4 at no cost in false rejects. The residual is an observability bound (what the observation channel discards, no predicate over it can recover), so discrimination must be bought with instrumentation rather than stricter criteria, and our repair rates are bounds: inflated by what the oracle cannot distinguish, deflated by unanimity's rejection of correct fixes under run-to-run noise. We release the corpus, criterion batteries, counterfeit suite and cached telemetry.

0.57

XPolicyLab: A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment

cs.RO ·2026-08-10 ·XPolicyLab Community, Tianxing Chen, Yue Chen, Tian Nian, Zijian Cai +65 more
code/datahighly cited similarity 0.65
Abstract · Website: xpolicylab.github.io, Code: https://github.com/XPolicyLab/XPolicyLab

Robot policy evaluation and deployment remain fragmented by model-specific software dependencies, data representations, and runtime interfaces, so that connecting N policies to M evaluation environments requires O(NM) separate integrations. We present XPolicyLab, a unified standard and open ecosystem that reduces this cost to O(N+M). XPolicyLab specifies common observation, action, and trajectory schemas together with a minimal adapter interface for observation updates, action prediction, batched execution, and episode reset, while a dependency-isolated client/server architecture separates policy inference from environment execution, so that each side retains its native software stack and may run locally or remotely. The ecosystem integrates 42 robot policies and standardizes their installation, debugging, serving, and evaluation workflows. Across these adapters, model-specific code varies by an order of magnitude while the environment-facing loop stays within a few lines of a fixed reference, confirming that the contract confines heterogeneity to the policy side. In a controlled study, conforming to the standard reduces the integration effort of a representative policy from over five hours to two hours, and packaged agent skills reduce it further to thirty minutes. The same adapters serve RoboTwin, RoboDojo simulation, and standardized real-robot evaluation through one interface. XPolicyLab is released as shared infrastructure for reproducible policy comparison and standardized deployment across simulation and physical platforms. Project website: https://xpolicylab.github.io/.

0.63