CS Paper Radar

Praxist: From Experimental Artifacts to Solution Lineages

Jin Li, Ahmed Murtadha, Zhiyu Wang, Qiwen Chen, William Chen, Yifei Wu, Guan Wang, Cheng Gong, Andy L. Siy, Jiayi Yang, Mengsha Huang, Wenhao Li, Yixuan Liu, Shuailin Pan, Mingli Yuan, Sen Song, Yuhao Sun

cs.MA cs.SE ·submitted 2026-08-26 16:13 UTC · v2
benchmark

Abstract

Autonomous R&D agents now write, run, and improve executable artifacts under automated evaluation-but largely as laboratory instruments: shown on curated benchmarks, with gains that are hard to trace to a cause and costs well above what sustained engineering practice absorbs. The limitation is structural. Most systems treat each attempt as nearly self-contained, so logs, memories, and search trees record what happened without establishing which design element produced an improvement, whether its evidence survived validation, or how it recombines with others. Long campaigns therefore keep re-learning the same lessons. We introduce Praxist, a lineage-centered generational system that converts reproducible artifacts and evaluator outcomes into a typed evidence graph of findings, lane-structured frontiers, and agendas. Separating local artifact construction from cohort-level evidence synthesis lets later attempts inherit validated mechanisms, unresolved claims, and useful constraints, and leaves results attached to an inspectable lineage. On the standardized 75-task MLE-bench suite, the finalized official-grader results give Praxist 60 medals (80.0%), 49 of them gold, against 55 medals (73.3%) and 34 gold for a Claude Code baseline on Claude Opus 4.8-at a recorded model spend of $3054 versus $38370, roughly a twelfth of the cost. Five case studies-quantitative trading, LiDAR-inertial-visual SLAM, tokamak magnetic control, rocket landing, and job-shop rescheduling-carry the same process into open-ended engineering problems, improving on each task-native baseline in headline accuracy, survival, or resource cost, with the discovery path on record. Stronger artifacts at an order of magnitude less spend, each backed by an auditable lineage, are, to our knowledge, first brought together here: the operating profile production research requires, not the one a benchmark demonstration establishes.

Closest papers

Nearest neighbours by meaning across the whole index.

Artifact-centered Claim-aware Observability for Autonomous Scientific Agents

cs.CL cs.CYcs.HC ·2026-08-18 ·Xiangyu Yin, Ming Du, Michael H. Prince, Mathew J. Cherukara
positionhighly cited similarity 0.70
Abstract

Autonomous scientific agents now increasingly propose ideas, write code, run experiments, analyze results, and even draft papers. Observe and audit those agents are necessary but logging every model call is not enough, scientists also need to inspect the artifacts and claims that the systems produced and their relations. This is driven by the fact that failures in scientific agent systems are often distributed across several objects. A manuscript claim may cite the wrong evidence, a search process may select a degenerate candidate, a laboratory novelty claim may depend on an unstated rule, or a multi-agent plan may change without a visible trigger. Existing tracing, experiment tracking, and archival provenance tools are valuable, but their native objects do not make these scientific audit relations first-class. We argue that autonomous scientific systems should emit portable, claim-aware artifact lineage as a minimum audit layer. We propose a compact observability profile organized around individuals, operators, fitness records, lineage, archives, runs, streams, and steering commands. In this profile, scientific claims are ordinary individuals with explicit evidence bindings and verification records. The profile is intended as a semantic layer that complements current telemetry and provenance standards. Execution details can remain in OpenTelemetry. Final packages can export to PROV-O or RO-Crate standards.

0.52

ReproBridge: Building Reproducibility into ML Research Artifacts

cs.AI ·2026-11-05 ·Yubo Wang, Rui Meng, Jaehyun Nam, Tomas Pfister, Jinsung Yoon
community pick similarity 0.69
Abstract

Reproducing machine learning experiments requires reconstructing procedures from information scattered across papers, released code, and supporting artifacts. For coding agents, reproduction success can therefore depend on their ability to fill in missing details, rather than on the released artifacts alone. We ask whether preserving recovered procedures and decisions improves subsequent independent reproduction. We define a standardized artifact design comprising a refined codebase, environment setup, experiment launchers, and a reproduction guide, and introduce ReproBridge, an agent workflow for constructing these refined reproducible artifacts. ReproBridge-Bench provides 1,029 tasks from 387 experiments across 36 papers, anchored by five human-verified reference scores per task. Agents attempt tasks in fresh sessions with raw code or refined artifacts; a separate Auditor checks execution evidence. Across four evaluated models, refined artifacts raise mean reproduction success from 76.4% to 92.6% and reduce downstream API spending by 19.7% on average. The cross-model success-rate spread narrows from 21.6 to 3.9 percentage points, with all four models achieving mean success rates above 90%. Removing any component reduces success in the ablation study, with the largest losses from code and environment removal. These results suggest that preserving execution knowledge helps even models with lower raw-code success reproduce experiments more reliably and at lower API cost.

0.21

From Final Artifacts to Trajectories: Retrospective Process Supervision for Evidence-Grounded Long-Form Generation

cs.CL ·2026-08-31 ·Junjie Huang, Jiarui Qin, Di Yin, Weiwen Liu, Yong Yu +2 more
top venue similarity 0.69
Abstract

Trajectory data is getting more vital for training large language models for boosting the agentic abilities. Unlike the verifiable domains such as coding or mathematics, scaling trajectory data for open-ended tasks is much more difficult because these tasks lack singular ground truth and are costly to annotate or verify. In this paper, we propose RetroGen, a self-improving framework of retrospective process supervision. Our key observation is that although expert trajectories are scarce, high-quality final artifacts such as literature reviews, analyst reports and legal judgments, are abundant in pre-training data and can be viewed as compressed traces of the evidence-seeking processes that produced them. RetroGen reconstructs candidate latent trajectories from expert artifacts, verifies them against both the artifact and supporting evidence, and trains models on their own successful reconstruction data, without requiring trajectory data from stronger models. Experiments show that RetroGen improves grounding, faithful synthesis, and long-form evidence-seeking agent tasks.

0.47

Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

cs.AI ·2026-10-09 ·Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang +12 more
similarity 0.68
Abstract · 21 pages, 11 figures, 3 tables. Includes appendix

Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its final state passes a binary check. We present Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0), an extension of Long-Horizon-Terminal-Bench that preserves its 46 original tasks while correcting task and evaluation-infrastructure defects and extending 31 tasks. LHTB 2.0 contains 77 tasks across 13 domains and eight task types. The extended tasks are produced through a systematic reconstruction-task pipeline. We start from a working scientific or engineering artifact, then use a parameterized reference implementation to generate multiple scenarios, reveal six as behavioral evidence, hold out three for evaluation, and use dense progress ladders to reward partial progress while preventing memorization of public outputs. We also extend the evaluation horizon from 90 minutes to three hours and record verifier checkpoints every 30 minutes, enabling direct measurement of sustained progress within a run. Across 1,145 scored model-task runs from 15 frontier models, GPT-6-astra leads with dense mean reward 0.69 and 37 tasks at R at least 0.95, followed by GPT-5.6-sol with dense mean reward 0.67 and 32 tasks at the same threshold. On the 30 reconstruction tasks, binary verifier scores average 0.221, whereas dense ladder scores average 0.653, showing that outcome-only evaluation discards substantial intermediate progress. LHTB 2.0 provides a broader, hardened, and process-aware testbed for studying reliable agent execution over hours.

0.48

ResearchFlow-Eval: Tracing the Execution Gap in End-to-End Multi-Claim Autoresearch

cs.AI cs.LG ·2026-10-13 ·Peng Jiang, Dongyang Li, Quanying Liu, Xiaoxuan Jia
similarity 0.67
Abstract · 23 pages, 6 figures

Autoresearch systems increasingly undertake entire research projects. In practice, scientific research requires systems to pursue multiple interdependent claims through a continuous, multi-stage workflow and revise experiments as evidence accumulates. Existing evaluations of autoresearch systems, however, typically score isolated tasks or final artifacts, leaving these process-level capabilities largely unmeasured. We therefore introduce ResearchFlow-Eval, a fine-grained framework for assessing continuous scientific execution and iteration at both the stage and claim levels. Each evaluation is grounded in a published anchor paper, from which we derive stage-specific rubrics that decompose each research stage into dimensions and separately scored rubric items. Using these rubrics, we score each research stage and examine experimental execution in detail, focusing on multi-claim completion and revisions across attempts. We apply the evaluation framework to general-purpose agent harnesses, skill-augmented harnesses, and systems with research-specific orchestration. Across all evaluated systems, experimental execution consistently lags behind design and planning. Within execution, systems struggle to produce high-quality results and meet claim requirements, and repeated attempts rarely yield measurable improvements. Moreover, neither skill augmentation nor research-specific orchestration reliably yields higher scores. ResearchFlow-Eval thus extends autoresearch evaluation beyond isolated, single-objective runs to continuous workflows in which systems pursue multiple claims and refine their work across iterations.

0.22

EVISKILL: Grounding Skill Evolution in Replayable Evidence

cs.AI ·2026-10-04 ·Yan Zhou, Yili Wang, Yiwei Dai, Qinggang Zhang, Xin Wang
highly cited similarity 0.67
Abstract

Continual skill evolution enables LLM agents to accumulate and refine reusable procedural knowledge from interaction experience without updating model parameters. Its effectiveness depends on determining not only what to change, but also why a change is justified and when it should become persistent guidance. However, existing experience-driven methods can lose the behavioral evidence and task contexts supporting edits. Moreover, a global validation outcome provides an incomplete judgment of its constituent changes: locally supported corrections may be discarded with a rejected revision, while evidence may require further experience to inform useful updates. To this end, we introduce EVISKILL, an evidence-driven framework that organizes execution observations into Replayable Evidence Cards and synthesizes edits with explicit links to their supporting contexts. Targeted replay verifies these edits through re-execution and provides feedback for correction. Across epochs, EVISKILL preserves evidence and provisionally retains supported edits for further refinement, while global validation governs their incorporation into the final skill. Experiments on three interactive benchmarks across six LLM backbones demonstrate the effectiveness of this approach.

0.46

Evolutionary Ensemble Search: Council-Guided Program Evolution with Persistent Memory

cs.NE cs.AI ·2026-09-11 ·Juan P. Madrigal-Cianci, Eshan Chordia
similarity 0.66
Abstract

Evolutionary Ensemble Search (EES) constructs machine-learning procedures through expert-guided program evolution. A role-specialized council turns task evidence and experimental results into structured search directions. An orchestrator allocates these directions to execution specialists and an evolutionary engine. The engine selects measured parents, diagnoses their errors, and produces descendants through code mutation, structured pipeline edits, and crossover. Each child must execute and acquire its own validation evidence. Population archives retain useful alternatives, while compatible predictions compete in a validation-gated ensemble stage. Search adapts through parent-relative operator credit, session memory, and problem-indexed lessons retrieved across runs. We specify these mechanisms, distinguish their execution profiles, and define the contracts required to compare candidates as their computations change. A public MLE-bench Lite development ledger records medal-threshold artifacts on 19 of 22 tasks (86.36\%), with best outcomes of 11 gold, five silver, and three bronze. The procedures span text, images, tables, audio, scientific geometry, and deterministic transformations. The campaign includes grade feedback between runs, external-source routes, and mixed confirmation procedures; its aggregate is an achieved development result, not a blind autonomous-agent success rate. The report contributes a concrete architecture for cumulative executable search and a versioned account of its cross-modal development outcomes.

0.08

Beyond FAIR Data: Instrument Traces for Active and Autonomous Scientific Experimentation

cs.DL ·2026-08-24 ·Sergei V. Kalinin, Boris N. Slautin, Yu Liu, Charles Cao
position similarity 0.66
Abstract

Artificial intelligence is turning scientific instruments into active systems in which observations can determine what is measured next. We argue that this creates an additional scientific record, the experimental trajectory, complementing sample provenance, acquired data and metadata, and analysis workflows. Instrument Traces should ultimately be synchronized with Sample Traces describing specimen evolution and Decision Traces recording human or algorithmic choices. We reconstruct an Instrument Trace retrospectively from a longitudinal AFM/PFM archive containing 118,000 timestamped events from 2023-2026. Conventional saved files reveal material campaigns, latent probe and calibration states, session-level complexity, experimental decision grammar, and composite tuning actions. They also expose what is missing, including unsaved tuning and failures, explicit sample/probe identities, complete timing, exogenous state, and decision rationale. We therefore propose a prospective trace architecture that records synchronized sample, instrument, and decision histories, enabling reproducible autonomy, predictive maintenance, counterfactual analysis, operator training, and transfer across facilities.

0.43