CS Paper Radar

ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows

Kaiyuan Hou, Yang Ping Kuo, Indranil Bhattacharya, Rohith Nama, Arushi Gupta, Carlos D. Castillo, Prathap Ramachandra, Xiaofan Jiang, Francesco Carbone

cs.AI cs.LGcs.MA ·submitted 2026-11-10 18:49 UTC · v1

Abstract

Business process automation runs the same Standard Operating Procedure (SOP) again and again on new inputs. Large Language Model (LLM) agents can follow an SOP written in natural language, but they call an LLM to choose every step of every run, which adds latency and cost at scale. We present ReTrace, which turns an SOP and a few logged agent runs into a reusable workflow in which code, not an LLM, selects the next step. The agent's reasoning, the explanation it writes with each tool call, often states why it took an action; the tool calls alone do not. ReTrace extracts the condition behind each step, merges these conditions across runs into a compact rule document, and generates an executable workflow from it. Only this last stage depends on the platform, so rule documents can be compiled for different orchestration platforms. On $79$ procedures from our production platform, ReTrace workflows reach $93.8\%$ completion, against $96.8\%$ for the agent, at $24\%$ lower cost per task. On $τ^2$-Bench, they reach $75.8\%$ task success (agent: $80.6\%$) and run in $1.3$s instead of $170$s; three rounds of repair on training failures lift $k{=}2$ workflows from $66.7\%$ to $81.2\%$. On eight SOP-Bench procedures, ReTrace without repair trails few-shot generation ($80.6\%$ vs. $83.6\%$), but repair from labeled training failures lifts it to $87.8\%$ at $0.40$s per task, compared to $85.5\%$ in $46.6$s for the agent.

Closest papers

Nearest neighbours by meaning across the whole index.

Natural-Language Workflows Are Not Software Yet: Artifact-Driven Compilation for Reliable Agent Execution

cs.SE ·2026-08-21 ·Xiangzhe Xu, Hanxi Guo, Guangyu Shen, Siyuan Cheng, Xiangyu Zhang
similarity 0.73
Abstract · The first two authors contributed equally

Natural-language workflows offer a software-like interface for agents: domain experts can write reusable procedures, and agents can execute them as instructions. This promise is not yet reliable. Workflow descriptions often leave data dependencies implicit, so the executor must infer which prior results a step should use; agents can also fail to follow long or branching instructions under context pressure. We propose Artic, an artifact-driven workflow compiler that transforms a natural-language workflow into an artifact-driven workflow in which each step declares the artifacts it reads and writes, constraints gate produced artifacts, and explicit control transfers route execution. This representation exposes the enforcement burden placed on agent execution, allowing the compiler to identify steps that depend on too much state or contain difficult control logic and refine them through constrained optimization. To validate the LLM-assisted transformation, Artic decomposes faithfulness checking into local obligations and uses scenario-based dry runs to test whether compiled workflow regions conform to the source workflow. We evaluate Artic on 488 problem instances from 11 real-world domain workflows; it improves task resolve rate by 28 percentage points over the original text workflow. We also show that workflows compiled by Artic are 32 and 56 percentage points more consistent in cross-model and repeated-execution setups, respectively.

0.34

AgentOrch: How Close Can Global Orchestration Come to Step-by-Step Interaction?

cs.AI ·2026-10-24 ·Zijie Liu, Qi Yang Huang, Jinhao Duan, Yueh Che Lee, Thomas A. Joseph +4 more
benchmark similarity 0.72
Abstract

Enterprise workflow agents typically rely on step-by-step interaction (Step-by-Step): the agent executes an action, observes its result, and invokes a large language model (LLM) again to decide the next step. Global orchestration (Global) instead generates a complete executable workflow in advance and executes it without further LLM calls, substantially reducing inference overhead, but it remains less accurate than Step-by-Step. We study how close Global can come to Step-by-Step and what narrows the remaining gap. We introduce AgentOrch, a benchmark that directly compares global orchestration with step-by-step interaction on 650 enterprise SOPs and 3,220 runtime variants under matched tools, environments, and evaluation. Step-by-Step remains more accurate for every evaluated model, but the gap has shrunk by about 79% from early models to the strongest recent model. We identify three routes to further narrowing this gap. Incremental workflow construction improves Global accuracy by an average of 9.98 percentage points (pp) without runtime execution feedback. Revising complete workflows with execution feedback improves all evaluated models, including 6.13 pp on held-out runtime variants. Targeted training on complete workflows improves accuracy by an average of 12.18 pp across all five trained models. Although a gap remains, global orchestration offers a promising path toward enterprise agents that deliver higher accuracy with fewer LLM calls.

0.16

ToolSpace: Scaling Tool-Centric Environment Synthesis for Long-Horizon Agents

cs.CL ·2026-11-03 ·Guanting Dong, Zhichao Sun, Xinyu Yang, Rui Wang, Xin Deng +20 more
community pick similarity 0.69
Abstract · Working in progress

Large language models are deployed as long-horizon agents that compose diverse tools across MCP services into executable workflows. Existing MCP environment-scaling methods are data-centric: they first synthesize a database and scenario, then derive tools and tasks from them. As a result, tool interfaces may not match real ones, tool compositions are not grounded in execution, and tasks rarely span more than one environment. We propose ToolSpace, a tool-centric framework that reverses this order. ToolSpace mines data, state, and retry dependencies from adjacent calls in execution traces of real MCP services and open agent corpora, using deterministic rules rather than LLM judgment, and clusters the resulting weighted tool-composition graph into environments. Databases are synthesized only for state-mutating environments, while query and stateless tools replay recorded returns. ToolSpace then performs state-aware random walks with rollback and decompiles each chain into a natural-language task with assertion-based checks, keeping only samples that replay successfully from the initial state. Applied to AutomationBench and tau3-Bench Banking, the pipeline produces harder Pro versions on which 7 frontier models drop by up to 38.2 points. Across seven long-horizon tool-use benchmarks, ToolSpace raises Qwen3.6-35B-A3B from 23.0 to 41.0 average score (43.2 with RL) and GLM-5.2 from 44.6 to 56.6, on par with Claude Opus 5. Ablations show that execution-witnessed edges clearly outperform LLM-judged ones, and performance keeps improving with data and environment scale.

0.20

AgentPRO: Coordinating Compiler Optimizations for LLM Agent Workflows

cs.SE ·2026-10-31 ·Jie Ren, Zheng Wang
similarity 0.68
Abstract

LLM agents repeatedly return to a model to generate actions and interpret tool results, incurring repeated context processing and generation overhead. Yet a value needed to execute an action need not be observed by the model to generate that action. This distinction creates an optimization opportunity: reconsider where model decisions are necessary, together with the work performed between them. We present AgentPRO, a compiler framework for optimizing agent workflows as execution information becomes available. AgentPRO organizes planned duties, generation regions, and concrete operations in a hierarchical representation, distinguishing execution dependencies from observations required for generation. Region transformations combine compatible duties into fewer model requests, while operation transformations reduce redundant context and tool work. These scopes interact: eliminating a redundant read can remove an observation requirement that otherwise separates generation regions. AgentPRO coordinates transformations using initial plans, current observations, and historical execution profiles through just-in-time, adaptive, and profile-guided optimization. Runtime guards check state-dependent assumptions and retain fallback paths. We implement guarded workflow transformations for AppWorld agents. On 115 tasks selected by successful historical baseline executions, automatic AgentPRO reduces all-attempt mean token consumption by 41.85% and mean component time by 6.61% relative to those baselines, successfully completing 110 of 115 tasks.

0.06

How Much Skill Orchestration Can LLM Agents Compile Away? Execution-Grounded Specialization for Token-Efficient Skill Use

cs.SE cs.AIcs.CLcs.LG ·2026-11-05 ·Jiayi Ni, Zac Wang, Alex Zhang, Zheng Liu, Roopa Prabhu
published similarity 0.67
Abstract · 35 pages, 20 figures, 16 tables. An earlier version has been selected for an oral/spotlight presentation at the AgenticOS workshop at NeurIP…

Reusable skills guide large language model (LLM) agents through recurring requests, but each invocation still requires the model to interpret the skill and reconstruct its orchestration. Can agents automatically compile this orchestration into an executable fast path without human intervention? We first study zero-shot skill specialization, which compiles without execution evidence, and find that despite the skill's procedural guidance, incorrect assumptions about runtime bindings can make generated plans brittle and recovery costly. This motivates SkillCache, a one-trace skill specialization method that compiles a completed invocation into a parameterized capsule of programs and natural-language steps, replays it before reuse, and drives it with a plan executor that returns control to the model only when needed, keeping the source skill for recovery. With the Claude Code harness, warm execution of the same request reduces tokens on 87 SkillsBench tasks by 81% with Claude Sonnet 4.6 and by 74% with Claude Opus 4.8 at comparable reward, and capsules stay effective under controlled request changes. Capsules also transfer across models, raising three open-weight models' mean reward by 12-25 percentage points while reducing their tokens by 72-81%, suggesting that execution-grounded compilation is a promising approach to efficient repeated skill use and cross-model transfer.

0.27

WorkflowArena: Benchmarking Executable Workflow Orchestration in LLM-Based Multi-Agent Systems

cs.CL cs.SE ·2026-11-09 ·Zheng Jiang, Fang Niu, Bo Peng, Houde Qian, Wenyi Li +8 more
benchmarkcode/data similarity 0.67
Abstract · Project page: https://lab.einsia.ai/workflowarena

Large language model agents are increasingly deployed as multi-agent systems rather than as a standalone model operating within a single context. Recent agent harnesses allow the model to instantiate worker agents and to author the executable workflow that coordinates them, shifting the design of system structure from developers to the model at run time. The quality of this organization therefore constitutes a distinct capability, yet existing benchmarks either measure end-to-end task completion or evaluate coordination under structures specified in advance. We formalize this capability as workflow orchestration and introduce WorkflowArena, a benchmark of 145 long-horizon tasks spanning 8 professional domains, from software engineering and formal mathematics to legal analysis and 3D production. Each task specifies an initial workspace and an integrated deliverable composed of interdependent components, which is assessed by a hidden verifier. Agents determine autonomously whether and how to delegate, and all delegation is recorded, enabling joint evaluation of deliverable quality and the organization of the underlying work. Experiments with nine models show that no single configuration performs best across all domains and that systems exhibit distinct delegation strategies: at high effort, Claude models in Claude Code invoke workflows less frequently but with larger worker groups, whereas GPT models in Codex delegate more frequently in smaller units. When the executing model is held fixed on a challenging subset of tasks, workflows authored by Claude Opus 5.5 substantially improve weaker executors, indicating that workflow organization is itself a measurable determinant of task performance.

0.24

Mining the Design Space of Multi-Agent Workflows

cs.SE cs.AI ·2026-10-22 ·Xavier Cadet, Junyan Cheng, Peter Chin, Kyle Richardson
similarity 0.66
Abstract · 21 pages, including appendices. Equal contribution: Xavier Cadet and Junyan Cheng. Website: https://junyan.ch/apex-paper/

Designing large language model (LLM) multi-agent systems (MAS) still relies on trial and error and informal "folk wisdom." Manual studies test only a few communication structures or topologies (e.g., sequential, parallel, shared-memory, or debate), while automated optimizers discard search trajectories after selecting a task-specific workflow. We present a two-phase pipeline for mining reproducible design principles from workflow code and execution traces. We repurpose AFlow's MCTS optimizer across six composable operators and 26 deterministically verified Enigmata tasks, retaining all intermediate rounds to build a corpus of 28,568 executable workflows. A sandboxed AutoResearch loop then mines orchestration code and prompt templates under an $L_0$-penalized validation objective and tests candidate principles on held-out searches. Across 300 feature-mining trials, we identify effects that persist on held-out searches across three levels of workflow design: workflows beginning with independent solvers and one-shot aggregation (Star-first) perform worse ($Δ= -0.045$); persistent shared documents (SharedMemory) outperform debate ($Δ= +0.135$); and prompt specificity and placeholder-based context routing yield gains comparable to communication-structure selection ($Δ= +0.118$). More broadly, our pipeline and results show the feasibility of turning discarded search trajectories into a statistical testbed for discovering and validating multi-agent workflow design principles.

0.30

FlowScout: From Execution Feedback to Reliable Tool-Using Agent Workflows

cs.LG ·2026-08-10 ·Shuo Hao, You Lu, Bihuan Chen, Xin Peng
similarity 0.66
Abstract

Agentic workflows have become an important abstraction for building reliable LLM-based automation systems by organizing large language models (LLMs), tools, and control logic into explicit execution structures. However, constructing high-quality agentic workflows remains largely manual and requires substantial domain expertise. Recent studies have explored automatic agentic workflow generation from historical task-solving records, but they mainly produce LLM-centric workflows, where real tool executions are abstracted and simulated by LLM nodes, limiting the usability and stability of generated workflows. To address these limitations, we propose FlowScout, an execution-guided framework for generating tool-integrated agentic workflows from historical task-solving records. Specifically, FlowScout represents an agentic workflow as a directed graph composed of LLM nodes, tool-calling nodes, and dependency edges. It first mines a common tool coordination skeleton from historical records to construct an initial workflow, and then refines the workflow topology through Monte Carlo tree search guided by execution feedback. We evaluate FlowScout on four representative task domains and compare it with three baselines, i.e., PM4Py, ReAct and AFlow. Experimental results show that agentic workflows generated by FlowScout improve tool invocation correctness by at least 92.69% and execution quality by at least 17.66% over the baselines, while achieving lower performance variation across repeated runs.

0.34