CS Paper Radar

Domain-Grounded Tool Orchestration for LLM-Guided Scientific Analysis

Jeff Lee, Sebastien Jourdain, Cory Quammen, Patrick O'Leary, Berk Geveci

cs.CE cs.HC ·submitted 2026-08-31 12:32 UTC · v2

Abstract

Scientific analysis workflows encode deep domain knowledge through sequences of tightly coupled operations where correctness depends on tool selection, execution order, and parameterization. A CFD engineer investigating flow separation must extract wall shear stress, identify zero-crossings in skin friction, and confirm with boundary-layer profiles: a chain that requires both domain expertise and proficiency with visualization tools. Current approaches to LLM-assisted scientific visualization generate scripts that encode this knowledge implicitly, and often incorrectly, producing code that executes but yields wrong results. We present an architecture that separates intent interpretation (LLM) from execution (deterministic domain tools) from explanation (LLM), connected by the Model Context Protocol (MCP) and grounded by domain ontologies that constrain planning to valid analysis chains. We instantiate the architecture in two domains on the same ParaView server infrastructure: computational fluid dynamics post-processing and topological data analysis via the Topology ToolKit (TTK). Adding the second domain required only an ontology and tool wrappers around existing filters, with no change to the architecture, protocol, or deployment. By construction the design removes whole classes of failure that affect script generation (such as API hallucination and missing pipeline stages) and narrows the strategic errors that remain. An ablation across both domains locates the ontology's empirical effect: it does not change which tools the planner selects, which is already reliable, but corrects how the model interprets results, raising interpretation accuracy from 0.41 to 0.91, and only when the relevant fact is retrieved in scoped rather than bulk form. ParaView's client-server model carries analysis to production-scale datasets through a thin browser client.

Comment
27 pages, 12 figures

Closest papers

Nearest neighbours by meaning across the whole index.

CATVis: A Collaborative Multi-Agent Workflow for Turbomachinery Simulation Data Visualization

cs.HC ·2026-09-15 ·Zhe Wang, Zehao Lou, Guanghui Zhao, Yu Dong, Guan Li +4 more
published similarity 0.69
Abstract · The paper is accepted by IEEE VIS 2026 Short Paper

Recent advances in AI for Science have enabled natural language (NL) interfaces for scientific data analysis. In turbomachinery CFD post-processing, translating ambiguous high-level analytical goals (e.g., vortex identification) into precise visualization procedures supporting complex domain-specific analysis is challenging. We present CATVis, a Collaborative multi-agent workflow system that bridges this gap by transforming NL intents into structured middle representation for visualization. Our approach reformulates domain-specific visualization procedures as composable workflow representations, and use multi agent to generate workflow representations via intent planning, template generation, and error-aware refinement, where each stage incrementally updates a shared structured representation. We evaluate the impact of external knowledge and workflow structuring on generation accuracy, demonstrating that the proposed approach significantly improves complex workflow generation correctness while reducing prompt complexity.

0.14

MCPGen: Benchmarking LLMs on Executable MCPWorkflow Development

cs.SE cs.AI ·2026-09-20 ·Yingxuan Yang, Jiaqi Liu, Lirui Guan, Jiaye Gao, Weiwen Liu +2 more
top venuebenchmark similarity 0.65
Abstract

We study whether LLMs can produce executable workflow artifacts that remain consistent across graph structure, tool implementation, schema bindings, and runtime wiring. In this setting, correctness depends on cross-layer consistency: a workflow may be structurally plausible, yet still fail because tool implementations, schema bindings, or runtime execution do not align. Existing benchmarks largely evaluate these capabilities in isolation or rely on trajectory-level proxies, leaving open whether generated workflow artifacts execute end-to-end. We introduce \textbf{MCPGen}, an executable benchmark for Model Context Protocol (MCP) workflow development. MCPGen contains 100 self-contained MCP projects across 16 application domains and evaluates three diagnostic tasks: workflow reconstruction, tool creation, and backward-compatible workflow extension. We evaluate 11 representative LLMs in a single-turn foundation-model setting, assessing generated artifacts through static analysis, unit and integration tests, and process-isolated end-to-end execution. Models reach 88.5\% on workflow reconstruction, but no model exceeds 57\% end-to-end execution success. Per-tool unit-test pass rates reach 63.8\%, while project-level integration success does not exceed 45\%, suggesting that integration remains a major bottleneck even when isolated tool tests pass.

0.31

Self-Supervised Scaling of Terminal Environments for Scientific Domains

cs.SE cs.AI ·2026-10-02 ·Zhongzhi Li, Yucheng Shi, Zongxia Li, Junyao Yang, Ruhan Wang +6 more
similarity 0.64
Abstract

Terminal agents are increasingly deployed beyond software engineering in science and other specialized domains. Constructing training environments requires executable reference behavior and a domain-specific verifier that distinguishes semantic correctness from superficially plausible artifacts. Authoring these components for each task requires repeated engineering and limits reuse. We introduce software-in-the-loop reconstruction, a self-supervised framework that obtains reference outputs and verification targets from existing software workflows, executable programs mapping structured inputs to outputs. For each workflow, we execute multiple input configurations and partition cases into public observations and hidden evaluations. Given the instruction, input schema, and public input--output observations, an agent constructs an editable program without access to the source workflow. The candidate is evaluated on hidden configurations against workflow outputs. A hierarchical verifier combines domain-specific semantic comparison, structural validity, and anti-shortcut checks, while public feedback supports iterative revision. The construction admits additional workflows and configurations without authoring a reference solution for each task. We instantiate SWR with 500 workflows and 46 software families across six domains. Across three attempts per task, Qwen3.8-Max solves 838 tasks and produces 1,422 verified trajectories, which we oversample to 3,000 reconstruction-only training examples. Supervised fine-tuning of Qwen3.8-27B improves mean Terminal-Bench 2 performance from 47.94% to 53.56% across three seeds and achieves the highest mean among four matched-token corpus controls on all four reported evaluations. These results indicate that existing scientific software can provide scalable, behaviorally verified supervision for terminal agents.

0.46

Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives

cs.SE cs.AIcs.CLcs.LG+1 ·2026-09-01 ·Haibo Jin, Suijin Wang, Xucheng Yu, Haojing Luo, Haohan Wang
similarity 0.63
Abstract · 21 pages

Large language models (LLMs) augmented with external tools have demonstrated remarkable capability in solving complex real-world tasks. However, existing approaches suffer from two key challenges: brittle multi-step and multi-turn reasoning caused by incompatible tool output types and API schemas, and performance degradation under large tool catalogues. To address these, we introduce \textbf{Tool Primitives}, a design that replaces rigid API schema-based invocation with natural language as the interface for tool calling, where each tool is wrapped with an LLM interface that handles schema resolution and execution internally, enabling natural inter-tool communication for nested and multi-turn tool calling. Building on Tool Primitives, we host \textbf{ToolFace}, a centralized repository of 25,519 functions from which LLMs dynamically retrieve only the relevant tools at inference time, eliminating the need to enumerate raw API schemas in context. To orchestrate Tool Primitives and ToolFace reliably in complex settings, we further propose \textbf{HEART}, a \textbf{H}arness \textbf{E}ngineering framework via \textbf{A}gent-native, \textbf{R}eusable \textbf{T}ool Primitives, comprising a Planner, Router, and Verifier that jointly support dynamic tool invocation planning, multi-step execution, and feedback-driven recovery. Experiments on five benchmarks demonstrate that HEART outperforms SFT-based models by $10\%$ on average and surpasses GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro by $6\%$ on average while reducing API cost by up to $85\%$. On 50 real-world tasks, HEART achieves $84\%$ task completion, $3.8\times$ the average of three frontier commercial models ($22\%$).

0.45

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?

cs.AI cs.CL ·2026-08-11 ·Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub, Md Tahmid Rahman Laskar, Shafiq Joty +1 more
top venuebenchmarkcode/data similarity 0.63
Abstract

Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.

0.84

Natural-Language Workflows Are Not Software Yet: Artifact-Driven Compilation for Reliable Agent Execution

cs.SE ·2026-08-21 ·Xiangzhe Xu, Hanxi Guo, Guangyu Shen, Siyuan Cheng, Xiangyu Zhang
similarity 0.63
Abstract · The first two authors contributed equally

Natural-language workflows offer a software-like interface for agents: domain experts can write reusable procedures, and agents can execute them as instructions. This promise is not yet reliable. Workflow descriptions often leave data dependencies implicit, so the executor must infer which prior results a step should use; agents can also fail to follow long or branching instructions under context pressure. We propose Artic, an artifact-driven workflow compiler that transforms a natural-language workflow into an artifact-driven workflow in which each step declares the artifacts it reads and writes, constraints gate produced artifacts, and explicit control transfers route execution. This representation exposes the enforcement burden placed on agent execution, allowing the compiler to identify steps that depend on too much state or contain difficult control logic and refine them through constrained optimization. To validate the LLM-assisted transformation, Artic decomposes faithfulness checking into local obligations and uses scenario-based dry runs to test whether compiled workflow regions conform to the source workflow. We evaluate Artic on 488 problem instances from 11 real-world domain workflows; it improves task resolve rate by 28 percentage points over the original text workflow. We also show that workflows compiled by Artic are 32 and 56 percentage points more consistent in cross-model and repeated-execution setups, respectively.

0.35

Designing Collaborative AI-Driven Workflows for Scientific Software Engineering

cs.SE ·2026-10-07 ·Akash Dhruv, Max Knobbe, Pedro Machado, Anshu Dubey
similarity 0.61
Abstract

Agentic artificial intelligence systems can carry out a broad range of tasks in software engineering and scientific research, from writing and translating code to running workflows for data analysis and visualization. In scientific computing, the difficulty is verifying that agent-generated code is both correct and understandable to teams whose members bring different areas of expertise. We therefore argue that these systems are best used within collaborative team structures rather than as full automation. In the workflows we propose, domain experts write the specification and plan, and agents operate under a deterministic orchestration pattern to write the target code. Each stage ends with a numerical comparison against the reference code and requires human review and approval before the next begins. We evaluate these workflows on the translation of a large high-energy physics application from Fortran to C++, running the same task under different orchestrators, design patterns, and models. Across fourteen experiments, a simple author--reviewer loop with enforced limits completed a comparable number of files to a multi-agent workflow at about one-third of the cost per file.

0.44

FlowScout: From Execution Feedback to Reliable Tool-Using Agent Workflows

cs.LG ·2026-08-10 ·Shuo Hao, You Lu, Bihuan Chen, Xin Peng
similarity 0.61
Abstract

Agentic workflows have become an important abstraction for building reliable LLM-based automation systems by organizing large language models (LLMs), tools, and control logic into explicit execution structures. However, constructing high-quality agentic workflows remains largely manual and requires substantial domain expertise. Recent studies have explored automatic agentic workflow generation from historical task-solving records, but they mainly produce LLM-centric workflows, where real tool executions are abstracted and simulated by LLM nodes, limiting the usability and stability of generated workflows. To address these limitations, we propose FlowScout, an execution-guided framework for generating tool-integrated agentic workflows from historical task-solving records. Specifically, FlowScout represents an agentic workflow as a directed graph composed of LLM nodes, tool-calling nodes, and dependency edges. It first mines a common tool coordination skeleton from historical records to construct an initial workflow, and then refines the workflow topology through Monte Carlo tree search guided by execution feedback. We evaluate FlowScout on four representative task domains and compare it with three baselines, i.e., PM4Py, ReAct and AFlow. Experimental results show that agentic workflows generated by FlowScout improve tool invocation correctness by at least 92.69% and execution quality by at least 17.66% over the baselines, while achieving lower performance variation across repeated runs.

0.34