CS Paper Radar

Embodied-BenchForge: A Closed-Loop Agentic Workflow for Embodied Benchmark Construction

Baoyang Jiang, Fengchun Zhang, Leyuan Wang, Haotian Li, Yida Wang, Zhe Ji, Jinshan Lai, Xi Ren, Danyang Li, Zheng Yang, Jianwei Hu, Qiang Ma

cs.AI ·submitted 2026-09-11 17:16 UTC · v1
benchmark

Abstract

Agentic systems offer a promising way to automate embodied benchmark construction, but existing approaches typically cover isolated stages or remain specialized to predefined environments and task families. More importantly, multi-step construction produces dependent intermediate artifacts that are often passed downstream without artifact-specific verification, allowing local defects to propagate into the final benchmark. We present Embodied-BenchForge, an agentic framework that transforms user-specified evaluation intents into complete embodied benchmark artifacts. It formulates construction as Closed-Loop Benchmark Synthesis, integrating forward artifact synthesis with backward verification and repair. Skill-Orchestrated Artifact Synthesis composes typed and reusable skills into executable workflows, while an artifact dependency graph records intermediate outputs and their dependencies. Requirement-Guided Verification and Repair applies artifact-specific contracts throughout construction and uses provenance to trigger local re-execution or upstream rollback when verification fails. Embodied-BenchForge constructs six benchmarks covering diverse embodied scenarios in the Offline EQA Track, together with one interactive benchmark containing 220 executable tasks in the Interactive Embodied Track. Evaluations of representative MLLMs and embodied agents show that the benchmarks distinguish model capabilities in both observation-based understanding and closed-loop execution. Quality assessment and ablations validate benchmark quality and the effectiveness of verification and repair, while repair and skill-reuse analyses demonstrate efficient localized recovery and cross-benchmark reusability.

Closest papers

Nearest neighbours by meaning across the whole index.

WorkBorne: Discovering Agentic Tasks from Real-World Artifact Transitions

cs.AI cs.CL ·2026-10-10 ·Junyi Zhou, Yufei Wang, Qiyuan Zhang, Chuhan Wu, Zhengxi Li +5 more
similarity 0.76
Abstract · 27 pages, including references and appendices

Developing terminal-based agents for long-horizon execution demands synthetic tasks that tightly couple complex instructions with authentic, interactive environments. However, current automated synthesis suffers from a fundamental asymmetry: instruction-oriented methods fabricate low-fidelity, synthetic environments, whereas resource-oriented methods treat real artifacts merely as static inputs, degrading task instructions into shallow, trivial operations. To address this limitation, we introduce WorkBorne, a synthesis framework that uses artifact transitions as the central primitive for modeling the full life-cycle of agentic tasks. In WorkBorne, each transition specifies who performs the task, which artifacts anchor the initial state and intended outcome, and the corresponding relationship between them. Specifically, WorkBorne first constructs coherent and diverse artifact corpora from iterative web retrieval, then discovers non-trivial transitions by identifying meaningful input-output artifact assemblies and the relations that connect them. Each discovered transition is finally instantiated into grounded agentic tasks. We construct 1,061 artifact corpora and 7,882 task environments, yielding 15,485 high-quality trajectories, with 53.6% exceeding 30 assistant turns. Experiments on Qwen3.5-27B and Qwen3.5-35B-A3B show that WorkBorne substantially improves over base models (i.e., improving Terminal-Bench 2.1 from 43.1% to 61.8% and ClawEval General score from 75.0 to 80.2) and achieves the strongest overall performance among other synthesis baselines (e.g., TMax (Ivison et al., 2026), RST (Li et al., 2026a), Terminal-Universe (Wu et al., 2026a)). Further analysis shows that transition is strongly associated with execution depth, while corpus breadth and transition depth provide complementary scaling gains.

0.25

VISTA: Active Exploration for Self-Improvement in Embodied Agents

cs.RO ·2026-10-26 ·Zhang Zhang, Shengjun Zhang, Simin Huang, Min Chen, Chong Xia +2 more
code/data similarity 0.76
Abstract · code: https://github.com/AgenticEI/VISTA, project page: https://agenticei.github.io/VISTA-page/

Agentic embodied intelligence plays a foundational role in open-ended, dynamic physical environments. Rather than treating policy learning as an end-to-end mapping from instructions to actions, recent studies reveal that embodied systems depend not only on pre-trained foundation models, but also on harness. Prior approaches largely treat rollouts as static logs for post-hoc reflection, producing harness updates without autonomously testing and refining them through subsequent embodied interaction. To address this challenge, we propose VISTA, a meta-agent framework that transforms post-hoc rollout reflection into active self-exploration for embodied harness improvement. Specifically, VISTA maintains a harness graph that records the lineage of proposed harnesses, which enables the meta-agent to retrieve prior hypotheses, build on effective mechanisms, and avoid repeatedly exploring failed directions. Each iteration begins from a retained harness version and its execution evidence. Rather than directly committing an update from these historical trajectories, the meta-agent actively explores in the environment. It autonomously generates new rollouts, probes execution failures and recovery behaviors, and iteratively modifies the harness based on the resulting evidence. The refined harness is then evaluated afresh and added to the harness graph. We evaluate VISTA on the LIBERO-Pro, RoboCasa, and RoboTwin benchmarks. Experimental results demonstrate that VISTA progressively improves task success and is more rollout-efficient than static reflection, achieving higher success rates with fewer simulator rollouts.

0.21

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents

cs.AI ·2026-09-25 ·Maokai Qin, Chuan Qin, Qi Zhang, Dianyu Liu, Zirui Liu +3 more
benchmarkcode/data similarity 0.75
Abstract

Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systematic evaluation environments. Existing simulation-based laboratory benchmarks rely heavily on manual task engineering, making it challenging to systematically compile diverse scientific protocols into executable and verifiable embodied tasks at scale. To address this challenge, we introduce SciHorizon-eLab, an agentic protocol-to-task compiler that formulates scientific embodied task construction as a compilation problem. Given a natural-language protocol of scientific experiments, SciHorizon-eLab progressively compiles laboratory protocols into semantic-preserving embodied tasks through semantic grounding, executable task synthesis, and multi-stage simulation-based certification. The system generates semantically grounded environments, executable manipulation programs, and step-level success specifications, while enabling reproducible generation of expert demonstrations and execution traces. Using this pipeline, we further construct \BenchName, a ready-to-use benchmark comprising 300 certified tasks across diverse laboratory operations. It supports HIL task execution, reproducible expert-demonstration generation, and ordered step-level evaluation. Across representative tasks, the strongest policy attains an average success rate of only 49.7%, with further evaluations revealing pronounced weaknesses in human and embodied agent coordination. We publicly release the code, benchmark data, and evaluation toolkit at https://github.com/SciHorizon-elab/SciHorizon-elab.

0.48

EMHO: EMbodied Agent Harness Optimization via Experience Traces

cs.AI ·2026-10-06 ·Hyun Jung Lee, Jungtaek Kim, Jongwon Jeong, Tae-Eui Kam, Donghyun Kim +1 more
similarity 0.75
Abstract

Improving embodied agents often focuses on optimizing the underlying model through training, while the surrounding agent harness that controls planning, context, and tool use is typically engineered. We ask whether this harness can instead improve itself directly from experience traces under sparse environmental feedback. We propose EMbodied Agent Harness Optimization (EMHO), a self-evolving framework that keeps the embodied model frozen and iteratively revises its harness by analyzing execution trajectories and prior harness history. EMHO optimizes beyond skills or recovery prompts, modifying how the agent monitors progress, uses vision tools, grounds observations, and responds to failures. To support multiple subtasks with a single harness, we introduce EMHO-Merge, which addresses trade-offs in jointly optimizing a single shared harness across subtasks by using episode-level gains and losses to guide evidence-supported refinement of when and how revised behaviors are applied. We evaluate EMHO on EmbodiedBench across navigation and manipulation tasks, and EMHO consistently improves task success for both Qwen 9B and 27B models. Qualitative analysis shows that EMHO goes beyond recovering from failures and unproductive actions to reshape how the embodied agent interprets and interacts with its environment.

0.44

Towards a Systematic Study of Embodied System 2 over Full Task Execution Cycles

cs.RO ·2026-10-13 ·Yuxi Wei, Ye Fang, Shuai Yang, Tong Wu, Tai Wang +1 more
benchmarkcode/data similarity 0.74
Abstract

An embodied brain, or System 2, must turn an abstract goal into actions while discovering relevant scene conditions and checking whether executed skills achieved their intended effects. Open-loop evaluations cannot test observations shaped by an agent's decisions, while some closed-loop evaluations provide limited evidence during execution or assume idealized skill outcomes. We introduce SCOPER, a suite for studying observation-grounded planning, execution, and recovery. SCOPER-bench combines high-level goals, ongoing visual observations, a shared rule-based System 1 for low-level actions, and controlled execution failures. Its construction and validation pipeline yields 870 tasks across BEHAVIOR and ProcTHOR, complemented by about 3,500 situated questions tied to the evaluated agent's execution. Benchmark results reveal difficulties in retaining scene evidence, monitoring progress, and recovering from unexpected outcomes. To address these difficulties, SCOPER-harness combines Evidence Memory and Progress Verification with a dynamic task agenda and bounded exploration, without additional model training. Across two evaluated open-source backbones and two simulators, the harness improves task success by 11.1-28.6%; a harnessed 9B model outperforms a directly executed 27B model on both simulators. In an offline next-action evaluation using observations from 200 real-robot demonstrations, the harness also improves full-call accuracy by 14.7% and 15.3%. SCOPER provides a setting for diagnosing and improving embodied reasoning throughout a complete task execution cycle.

0.49

RoboBridge: A Self-Evolving Embodied Agent Framework for Sim-to-Real Transfer

cs.RO ·2026-10-02 ·Chenxi Li, Zhangrui Zhao, Rui Li, Yuan Gao, Kehui Liu +6 more
similarity 0.73
Abstract

A key challenge in bringing embodied intelligence into the real world is transferring capabilities from simulation to reality and enabling agents to continually adapt after deployment. End-to-end vision-language-action policies provide strong manipulation capabilities, but their transfer to physical environments typically relies on calibrating simulated visual and dynamical conditions, collecting additional target-domain demonstrations, and optimizing the policy through further training. Tool-using embodied agents offer flexible task orchestration, yet existing systems primarily emphasize task execution and experience reuse within a given environment, with limited support for transferring procedural knowledge and continuously adapting it across simulation and reality. We propose RoboBridge, a framework that treats sim-to-real transfer as the continued adaptation of executable task skills. The agent represents task knowledge as procedures connecting task intent, observations, tool operations, and outcome verification. Interaction feedback is used to generate candidate skill revisions, which are evaluated before being persisted or rejected. A pretrained vision-language-action policy is exposed as a reusable action tool and enhanced with inference-time guidance, enabling fine-grained execution without retraining the underlying policy. RoboBridge grounds transferable skills in task semantics and interaction interfaces shared across simulation and reality. This representation preserves reusable task structure while allowing environment-dependent operations to be selectively revised through real-world execution feedback. We evaluate the framework on LIBERO-PRO and corresponding physical tasks, studying both skill evolution and post-transfer adaptation. Our framework provides a route from one-shot policy deployment to continual procedural learning across environments.

0.47

EnactOpt: Diagnosis-Guided Harness Evolution for Embodied Agents

cs.RO ·2026-10-10 ·ZhiYuan Feng, Zhuoshi Pan, Zhaolu Kang, Zhongxia Chen, Shuo Yang +3 more
hot direction similarity 0.73
Abstract · 30 pages, 5 figures, 5 tables; includes appendices

Embodied agents built on foundation models rely on an execution harness to organize observations, guide actions, and incorporate execution feedback into subsequent decisions. Failures can arise not only from limited model capabilities, but also from how the harness presents evidence and guides its use. These model-external mechanisms provide an adaptation target without updating model parameters, but task-specific interaction experience can be costly to collect. Task-level outcomes alone do not reveal whether relevant evidence was missing from the model input or was available but inadequately used. Moreover, these two aspects are coupled: effective action guidance may require evidence that the current context omits, while exposing that evidence alone does not ensure an appropriate response. We introduce EnactOpt, which uses limited execution feedback to jointly evolve reusable textual skills and context-construction code while keeping the underlying models and runtime fixed. It diagnoses adjacent decisions by relating action feedback to the actual model input and subsequent action. Structured episode summaries guide joint revisions through prompt-level edit operators, and candidate harnesses undergo compatibility checks and validation-based retention. Across VLABench, ALFWorld, and Embodied Agent Interface (EAI), EnactOpt improves task success by approximately 10-27 percentage points over the initial harnesses and outperforms the evaluated GEPA, TextGrad, and Trace2Skill baselines. These gains extend to task conditions not used for harness evolution or candidate selection. The results support a complementary route to embodied adaptation: turning limited execution experience into reusable behavioral improvements by jointly refining how evidence is presented and acted upon.

0.26

EmbodiExp: Compositional Experience Transfer for Embodied Agents

cs.RO cs.AIcs.CVcs.LG ·2026-10-29 ·Zhifeng Gu, Ying Chen, Rui Jiang, Lei Yi, Bing Wang
similarity 0.71
Abstract

Experience can help embodied agents turn high-level plans into grounded actions, but transferring it across tasks remains challenging: previously effective strategies may be inapplicable to the current state or fail to produce their intended outcomes. We present EmbodiExp, a framework for constructing transferable experiences and composing them online during embodied task execution. Our key observation is that tasks often share subgoals whose completion can be assessed independently of overall task success. We therefore introduce an outcome-grounded subgoal experience that specifies a subgoal, applicability conditions, an adaptable policy, and a verifiable outcome. During execution, applicable experiences guide action generation, while verified outcomes can establish the conditions required by subsequent experiences, enabling online composition. Subgoal-level outcome attribution further enables failure utilization: failed trajectories can still contribute verified local successes and condition-specific guidance on actions to avoid. Experiments on multiple embodied benchmarks under a fold-based protocol with frozen experience banks demonstrate the effectiveness of EmbodiExp.

0.09