CS Paper Radar

AIPOCH Open-Science: An Open-Source AI Research Workbench with Reproducible .science Packages

Wen Ouyang, Bocheng Huang, Qianyu Yao, Fei Sun, Minying Deng, Haoyue Cheng, Wenwen Zhou, Lei Mou, Yusi Wei, Jiang Li, Kun Zhu, Lei Wang, Yi Feng, Jiaheng Dai, Ke Qian, Dan Wang, Yahui Wang, Chenyi Wang, Mingsheng Lyu, Yaqiang Zhang, Ping Zheng, Chunhua Tang, Xinge Ma, Xu Zhang, Youjia Fan, Can Lu, Zhijun Dai, Man Hu, Qiqi Xie, Beinan Liu, Yuanning Guo, Yunze Liu, Jinhong Jiang, Xianjun Fu, Mingwei Zhang, Yuanji Xu, Shilin Xia, Bowen Yang, Mingjun Yu, Jingdun Xie, Xuwei Chen, Xia Zou, Yuzhou Xue, Jianguo Zhou, Chunhao Cao, Donghao Tian, Lei Huang, Jinxiao Li, Shengrong Long, Zefeng Song, Honglin Li, Jiang Luo, Wenying Jiang, Shanshan Li, Lei Dai, Ning Jiang, Ying Cai, Chaohui Li, Chunyan Peng, Hetong Zhao, Angqing Li, Chunyan Yang, Chongyang Sun, Yong Li, Xinyu Shi, Wei Dai, Ming Peng, Chen Sang, Yuanyuan Han, Desong Qiu, Haixin Miao, Jian Xu, Yi Chen, Xiaoliang Shi, Tian Shen, Teng Hu, Shuhao Wang, Yingyong Hou, Huimei Wang

cs.SE cs.AI ·submitted 2026-10-12 17:03 UTC · v1
code/data

Abstract

AI-assisted research leaves instructions, computations, artifact revisions and checks whose relationships matter when another researcher receives the work. We present AIPOCH Open-Science, an open-source research workbench with portable .science session packages, and specify an exchange contract for these recorded relationships. Version and assessment targets remain attributable while receiving-workspace identifiers can be remapped, eligible contents can be omitted explicitly, and imported history remains separate from a writable fork. A controlled session tested an incorrect first version, a reviewed correction, an unreviewed revision and a planned but unexecuted step. We compared complete and partial exports with source records extracted independently of the exporter, then inspected imports into separate projects within the same installation. Both conditions retained the tested version and review associations, kept the unreviewed revision distinct from earlier assessments and represented excluded content as unavailable. A documentation-rich ordinary directory also supplied evidence for all seven static inspection questions. A separate registry case connected a table to its recorded task, execution and review; offline execution of its retained script and fixed input reproduced two saved CSV tables byte for byte. The study contributes an implemented exchange contract and a controlled examination of its preservation rules, content-selection constraints and interpretation boundaries.

Comment
23 pages; 2 main figures, 3 tables, a graphical abstract and Supplementary Figure S1

Closest papers

Nearest neighbours by meaning across the whole index.

OpenAI4S: Code as Action, Science as Sessions

cs.AI cs.CLcs.SE ·2026-09-14 ·Gongbo Zhang, Hao Li, Yu Wang, Mujie Lin, Liuzhenghao Lv +13 more
code/data similarity 0.66
Abstract

AI co-scientists could accelerate computational research, but over a long-running study the workflow also has to stay inspectable, resumable and reproducible, which requires persistent computational state and provenance. Here we present OpenAI4S, an open-source scientific research agent built around the principle of \emph{Code as Action, Science as Sessions}. OpenAI4S combines a persistent computing runtime with research-session management: orchestration is handled through structured tool calls, while scientific actions are represented as complete code cells executed in persistent Python and R kernels. An append-only Action Ledger, per-cell execution records, versioned artifacts, environment records, and workspace checkpoints preserve how results were produced and support session recovery, branching, and extension. Configurable sandboxing, permission controls, and code and trajectory screening provide complementary safeguards. We evaluate OpenAI4S on 36 research scenarios spanning retrosynthesis, molecular dynamics, protein binder design, protein mutation, catalyst screening, and mineral spectroscopy, measuring scientific task accuracy, workflow completeness, and reproducibility of the resulting repositories. OpenAI4S achieves an overall score of 7.83, compared with 5.7--6.4 for a general-purpose coding harness evaluated with three frontier models, with the largest gains on long-horizon and computation-intensive workflows. These results suggest that integrating persistent execution with session-level provenance can improve the reliability of AI-assisted scientific workflows. Environment specification and full rerunnability remain weak for every evaluated system, ours included, so reproducibility is still an open problem for scientific agents. The system is available under the MIT license at \href{https://github.com/PKU-YuanGroup/OpenAI4S}{github.com/PKU-YuanGroup/OpenAI4S}.

0.23

Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

cs.CL cs.AIcs.IRcs.LG+2 ·2026-08-31 ·Xuehai Wang, Haowei Qin, Tongxin Liu, Junkai Li, Buqiang Xu +4 more
code/data similarity 0.66
Abstract · Work in progress

Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address the problem, we present AutoSciRub, an evaluation-first framework that induces a task-specific executable rubric before research execution, and uses it to guide execution, criterion-level verification as well as iterative revision. AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria. The resulting rubric makes implicit experimental and evidential requirements explicit, providing guidance for experiments and analyses. During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report and its supporting artifacts. On ResearchClawBench, AutoSciRub consistently improves all tested configurations, with an average gain of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a randomly sampled 20-task subset of AstaBench E2E Discovery, AutoSciRub further achieves an average improvement of 16.8 points across three agent harnesses, while maintaining or increasing the number of successfully completed tasks. These results demonstrate that evaluation-first guidance provides an effective and generalizable control mechanism for autonomous scientific research (Code: https://github.com/zjunlp/AutoSciRub).

0.36

AI-Research Agents in the Wild. From GitHub and arXiv to Regularities and Gaps

cs.DL ·2026-09-01 ·Aleksey Komissarov, Andrey Ustyuzhanin
similarity 0.66
Abstract · 56 pages, 1 figure

AI-research agents, or autoresearch systems, combine language models with tools, search, evaluation, and iterative modification of research artifacts. Their public software ecology is hard to compare because repositories, papers, benchmarks, libraries, and companion artifacts are often counted as one population. We connect two agent-drafted, human-adjudicated registries frozen on 10 June 2026: 139 canonical public repositories and 101 papers on AI-assisted research systems, each record carrying an evidence card anchored to its primary sources. Nine promoted design lineages contain 59 memberships among 52 repositories, seven of them assigned to two lineages. Against this evidence we examine six time-stamped candidate design regularities. A prospective audit of 25 newly ingested repositories observed zero of six specified trigger events, supplying bounded support for R1-R4; broader corpus evidence narrowed the original scope of R5, and a failed directional prediction leaves R6 exploratory. The paper-repository graph contains 212 curator-assigned relatedness edges: 64 of 101 papers carry at least one edge, the edges reach 23 of 139 repositories, and the six most-linked repositories carry 120 of the 212 edges (56.6%). A full-text check of seven papers shows the relation measures our curation, not the papers' citation behavior. An identity audit adjudicated against commit histories and public contributor records confirms at least 18 paper authors who also author a canonical registry repository (20 of 63 candidate author slots in 13 of 101 papers); a byline-matched tier reaches 58 identities but is reported separately and more weakly. The study contributes a source-grounded theory under prospective test, a curator-assigned map of paper-repository relatedness, and explicit limits on claims about public visibility and community structure.

0.18

Artifact-centered Claim-aware Observability for Autonomous Scientific Agents

cs.CL cs.CYcs.HC ·2026-08-18 ·Xiangyu Yin, Ming Du, Michael H. Prince, Mathew J. Cherukara
positionhighly cited similarity 0.65
Abstract

Autonomous scientific agents now increasingly propose ideas, write code, run experiments, analyze results, and even draft papers. Observe and audit those agents are necessary but logging every model call is not enough, scientists also need to inspect the artifacts and claims that the systems produced and their relations. This is driven by the fact that failures in scientific agent systems are often distributed across several objects. A manuscript claim may cite the wrong evidence, a search process may select a degenerate candidate, a laboratory novelty claim may depend on an unstated rule, or a multi-agent plan may change without a visible trigger. Existing tracing, experiment tracking, and archival provenance tools are valuable, but their native objects do not make these scientific audit relations first-class. We argue that autonomous scientific systems should emit portable, claim-aware artifact lineage as a minimum audit layer. We propose a compact observability profile organized around individuals, operators, fitness records, lineage, archives, runs, streams, and steering commands. In this profile, scientific claims are ordinary individuals with explicit evidence bindings and verification records. The profile is intended as a semantic layer that complements current telemetry and provenance standards. Execution details can remain in OpenTelemetry. Final packages can export to PROV-O or RO-Crate standards.

0.52

PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews

cs.SE cs.AIcs.CL ·2026-09-10 ·Miguel Zabaleta, Baihan Lin
similarity 0.64
Abstract

Large language models (LLMs) and AI-enabled software increasingly participate in systematic-review decisions, yet the information needed to audit these workflows is reported inconsistently. We analyze SciLitBench, a corpus of 888 review-automation papers with 14,726 annotations, to characterize changes in methods, review-stage use, evaluation and reported limitations. Automation has shifted toward LLM- and software-facing workflows, including stages that can alter the evidence base. Since 2023, 38.0% of software/product papers reported no evaluation, compared with 9.3% of LLM papers. Reporting coverage increased with LLM workflow complexity, yet 52% of positive-only LLM evaluations still reported an unmet reliability or performance requirement. From these patterns, we introduce PRISMA-LLM, an empirically grounded framework separating implementation disclosure from consequence-sensitive evaluation and limitation reporting.

0.08

Auditable AI-Assisted Research Writing: An Engineering Discipline with Pre-Registered Process Observation

cs.DL cs.CYcs.HC ·2026-08-11 ·Yang Zhou, Chengqun Yu
similarity 0.64
Abstract · 25 pages, 2 figures

Language models now draft, classify and criticise inside research production, yet the artifacts they help produce carry little accountable history. Rather than detecting machine involvement afterwards, we specify an auditability discipline built at production time: git sealing with an anchor lineage, hash-bound provenance, red-line gates that refuse non-compliant artifacts and log every refusal, cross-model role separation, and programmatic assembly from registered sources. Adherence is instrumented by metric cards, each carrying a pre-registered blind spot and evidential standing, frozen before the prospective case it observes. In that case the observed project's pre-registered confirmatory test was executed under seal and returned No-Go, and that project's frozen stopping rule halted the work, against its own operators. A lower-graded retrospective case covers families whose machinery predates the protocol. Current observations are provisional; we release a package from which a third party can recompute every primary metric.

0.10

From Code Archival to Knowledge Graph: Bridging Software Heritage, COAR Notify and Wikidata

cs.DL cs.AIcs.SE ·2026-09-18 ·Camillo Carlo Pellizzari di San Girolamo, Francesco Tosoni
publishedcode/data similarity 0.64
Abstract · 15 pages, 4 figures. Accepted at the 7th Wikidata Workshop (Wikidata 2026), co-located with ISWC 2026. Open-source pipeline and code availab…

Software is a first-class scientific object, yet validated links between source code and the scholarly record remain largely absent from the Linked Open Data (LOD) cloud, isolating archived artefacts from semantic discovery. This paper presents an end-to-end reconciliation pipeline that harvests, validates, and models publication-to-repository pairs from sources where the link between a paper and its source code is explicit and editorially verified: the software-centric journals JOSS, SoftwareX, and IPOL, together with the reproducibility reports of the SIGMOD Availability and Reproducibility Initiative (ARI). This yields a curated corpus of 4,397 $\langle$DOI, repository-URL$\rangle$ pairs. We design two distinct application profiles grounded in Wikidata classes (one for scholarly articles, one for software instances) aligned with the schema.org and CodeMeta vocabularies. This architectural separation enables rule-based reconciliation at two granularities: lightweight, inline publication references or standalone, first-class Wikidata software nodes equipped with SWHIDs, Software Heritage's content-addressed identifiers. A read-only lookup against Wikidata shows that only 82 of the harvested repositories were already modelled there; human-reviewed batches have since created 4{,}182 new software items cross-linked to their articles. We further show that payloads of the emerging COAR Notify protocol, an external effort we do not develop, map natively onto our input format, so the same backend could later serve a live enrichment stream. Our core contribution is a pair of application profiles that turn Wikidata into a connector between the scholarly record and archived source code; we openly release all code, application profiles, and harvested datasets.

0.53

ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces

cs.CL ·2026-11-10 ·Jiahang Lin, Wanying Ren, Wanghan Xu, Huayu Sha, Yang Shu +22 more
benchmark similarity 0.64
Abstract

Research agents achieve increasingly strong benchmark scores, yet these scores do not necessarily translate into reliable delivery in ongoing research projects. Many research-agent benchmarks emphasize standalone tasks and relatively self-contained workflows in clean or curated workspaces. However, researchers ask agents to complete one step within an ongoing project, where requests depend on earlier session decisions and workspaces contain unfinished files, outdated versions, and failed attempts. We introduce ARE-bench to evaluate whether research agents advance ongoing projects while respecting prior decisions and existing workspace state, grounded in real session context and in-progress workspaces. ARE-bench includes 75 tasks built on agent sessions and full workspaces from recruited researchers' own projects, spanning seven research stages, 33 requirements, and 150 sub-requirements, with 1,173 binary checklist items in total. Each task is anchored at a real user request, preserves its requirements, and summarizes the earlier requests still in effect as background. The complete workspace is replayed to that boundary without cleanup, preserving artifacts from earlier turns. Each checklist is compiled from an intent-aligned rubric and credits any valid solution. Across seven frontier models, the strongest passes every checklist item on only 66.2% of tasks, and Design and Execution rank among the hardest stages for every model. Three models pass about 69% of tasks when only delivery contracts and general norms are checked, but just 28.4% to 40.0% once historical and material or workspace requirements are also checked. Evaluating delivery without these contextual requirements can therefore overestimate research agents and mask differences between them.

0.11