CS Paper Radar

Symposium: Trust via Auditable Records for Communities of AI Scientist Agents

Dexter Pratt

cs.AI ·submitted 2026-08-20 00:08 UTC · v1

Abstract

Symposium is a formal framework and practical implementation to record the operation of AI agents deployed by small scientific research communities. Symposium provides long-term, immutable histories of agent-driven research activity, leaving auditable trails of analyses, hypotheses, data, and scientific discourse. This shared record of published artifacts enables agents to build on prior work and preserves the evidence researchers and agents need to make purpose-dependent trust assessments. Symposium captures scientific argument, including structured claims, fine-grained evidence citations, assumptions, and explicit declarations of what material may and may not be used as evidence. Symposium differs from AI co-scientist agents or integrated AI research environments; it is a framework that separates a scientific community's durable history from the agents and other systems that operate on that history. It assumes that a community will use diverse AI systems in a rapidly evolving environment. A working implementation of the publication infrastructure, agent prompt components, and documentation are provided to enable users to rapidly set up and run their own Symposium community.

Closest papers

Nearest neighbours by meaning across the whole index.

Puzzles to Paradigms: A Survey of Human--Agent Collaboration in Scientific Innovation

cs.HC cs.AI ·2026-08-31 ·Zhenni Liang, Haoxuan Zhang, Haihua Chen, Ting Xiao, Junhua Ding +1 more
top venuesurveycode/data similarity 0.68
Abstract · Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026. 25 pages, 2 figures, 5 tables

Agentic AI systems are increasingly embedded in scientific workflows, including literature synthesis, hypothesis generation, tool use, writing, and peer review. However, existing surveys and evaluations often emphasize task performance while underexamining how humans and agents coordinate under different epistemic conditions of scientific work. This survey introduces CSCW$\times$Kuhn, a coordination-aware framework that connects computer-supported cooperative work dimensions with distinct scientific contexts. Using this framework, we synthesize recent human--agent systems and benchmarks to examine how coordination demands and failure risks may vary across contexts. Our synthesis identifies five recurring failure modes: procedural drift, grounding substitution, evaluative flattening, proxy optimization, and epistemic black-boxing, and argues for evaluations that move beyond task accuracy and toward grounding, provenance, disagreement preservation, and long-horizon scientific reliability. Our corpus and coding artifact are available at https://github.com/jenlia024/CSCWxKuhn-artifact.

0.55

Author-in-the-loop Agentic Reproducibility Auditing for a Machine Learning Conference

cs.AI cs.HC ·2026-10-15 ·Yukun Dong, Linjun Zhang, Tian Zheng
similarity 0.68
Abstract · 23 pages, 8 figures, 4 tables

Reproducibility checks during peer review require a shared understanding of what authors' code is expected to reproduce. We present an author-in-the-loop collaborative audit that makes author declarations explicit inputs to an AI agent's investigation and assessment. A structured checklist records reproduction instructions and limitations; a two-stage workflow on GitHub connects agent checks to author clarification, revision, and agent reassessment. The agent separately assesses whether results have been reproduced and whether authors need to provide clarification or make repairs, while human reviewers approve requests and decide whether concerns are resolved. We deployed the system across 36 conference papers and 133 experimental tables and figures, raising 20 issues after human review. Further checks supported closing all the issues, although full reproduction remained unverified in some cases. We further compared repeated audits with complete and blank checklists to examine how author declarations shaped the agent's investigation and assessment. Our work contributes a collaborative workflow and agent design that use author knowledge to guide reproducibility auditing. The conference deployment provides empirical evidence that this approach can help identify and address reproducibility problems through collaboration among authors, agents, and reviewers.

0.19

VerifiedLab: Harnessing Knowns for Unknowns

cs.AI ·2026-10-27 ·Yu Yao, Jiayi Dong, Yang Yang, Shuiwang Ji, Ju Li
similarity 0.67
Abstract

Across human history, knowledge has undergone several transitions, from orality to writing and then to digital archives. At each transition, humans established new conventions to preserve knowledge for the new era. Nowadays, as research agents are increasingly deployed for laboratory experiments, knowledge is crossing another boundary: from records written for humans to those for machines. In this work, we introduce VerifiedLab, a multi-agent harness framework for this human-to-machine transition of knowledge. VerifiedLab transforms research output from passive artifacts into active systems that accelerate downstream use, adoption and prediction of future trends. In VerifiedLab, a team of AI scientists with distinct roles reasons over records verified field by field: a principal investigator plans, a librarian computes the evidence directly from the records, specialists draft, and a gate and a critic check each claim against the measured reliability of its data before the harness decides whether to release or withhold it. We demonstrate the efficacy of VerifiedLab through extensive experiments on perovskite solar cells: VerifiedLab recovers known physical trends, anticipates devices reported a year later, and lets AI scientists inherit the literature together with a measure of how far it can be trusted. By scaling the inference-time verification compute, VerifiedLab introduces a new pathway for knowledge dissemination and machine harness for AI co-scientists.

0.18

TRACE: Governing AI-Assisted Research with Principles from Computer Science

cs.CY econ.GNcs.HC ·2026-10-16 ·Amirhossein Zohrehvand, Aldona Kapacinskaite
similarity 0.66
Abstract · 30 pages, 7 tables

Artificial intelligence (AI) has made it easier than ever to produce research outputs. Adding an agentic tool to a research team's work potentially extends the team's capabilities but calls for an active approach to organizing the research process. The pressing question, therefore, is how research teams should organize their work with this technology. We contend that the question is not new: software engineering and security research have long studied how teams of people working with machines keep control of projects with many contributors. We distill principles from the practices these literatures report and transpose them to AI-assisted research. The principles form TRACE: Track records, Review against sources, Accept independently, Control transitions, and Evaluate corrections. Together, the five principles give researchers a common basis for remaining in control while integrating AI into their workflows. By bounding units of work and attending to how each is shaped, they can help researchers pace AI-assisted work to human attention rather than to the speed at which AI produces content. We also introduce TRACE-OS, an operating system for AI-assisted research projects. It puts the principles into practice through shared instructions, tools, and records that fit researchers' existing workflows. With TRACE, we aim to join a conversation about the governing principles through which the research community can benefit from AI. We contend that a simple enough and useful standard can contribute to harnessing AI capabilities, allowing teams to delegate parts of their work to AI, while also building trust within teams and the broader research community.

0.25

AI-Native Academia Should Build a Community Verification Layer

cs.CY ·2026-10-22 ·Haotian Zong, Luhao Zhang
similarity 0.66
Abstract

This paper argues that AI-native academia should build a Community Verification Layer: interoperable, policy-governed infrastructure that converts researchers' routine reading and checking into persistent, claim- and version-linked evidence, while routing scarce deeper verification toward uncertain, consequential, and neglected research. AI is lowering the cost of producing plausible research artifacts faster than the cost of accountable evaluation, yet much of the evaluation researchers already perform while reading, citing, implementing, or extending work is never made reusable. We propose a tiered ladder from comprehension notes to claim checks, audits, artifact execution, reproduction, replication, and downstream validation. The layer would elicit judgments before revealing social signals, preserve multidimensional disagreement, privately verify contributors while allowing flexible public disclosure, allocate attention under expertise and coverage constraints, and make verification labor citable, with credit tied to the quality and depth of the evidence. We distinguish this design from prior open and post-publication review and outline governance requirements and potential counterarguments.

0.24

Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

cs.AI cs.MA ·2026-10-06 ·Wenyu Du, Stephen Chung
similarity 0.66
Abstract · preprint

Recent AI systems have made rapid progress in scientific discovery when given well-defined metrics, but whether they can autonomously undertake open-ended scientific discovery remains unclear. We investigate AI's ability to tackle open-ended tasks in Station, an open-world environment in which multiple agents simulate a scientific ecosystem. To tackle challenges specific to open-ended tasks, we propose augmenting Station with two mechanisms: a Supervisor mechanism and periodic Meta Reflection, which encourage persistent exploration even when intermediate metrics are lacking. We construct open-ended tasks from three recent oral papers presented at ICLR. We give agents the main research question studied in each paper while withholding the paper's results and disabling web access. We then measure how many of the original findings-partitioned into individual criteria-agents rediscover. We find that Station rediscovers 62.7% of the criteria on average, compared with 15.4% for Codex Multiagent-v2 and 14.4-20.6% for AI Scientist-v2. Ablation and behavioral analyses indicate that adding the two mechanisms together improves research coverage and continuity. We further evaluate Station on two open-ended tasks without oracle papers and find that some of the discoveries made by the agents closely match discoveries reported by researchers after the knowledge cutoff date. Together, these results indicate that a suitable environment can enable agents to autonomously make meaningful progress in open-ended scientific discovery.

0.16

Research with AI Agents: How Agentic Systems Are Changing Scientific Work

cs.CY ·2026-09-25 ·Johannes Lotz, Markus Wenzel
similarity 0.65
Abstract · 6 pages, 1 figure. English translation followed by the original German version. Submitted to Die Pathologie (Springer Medizin)

Background. Agentic AI systems independently decompose tasks such as literature search, data analysis, and programming into subtasks, search the web, access databases, and execute code. This allows them to perform digital research tasks at high speed. Objectives. Under what conditions does the use of agentic systems produce reliable efficiency gains, and which tasks remain with researchers? Materials and methods. Summary of current studies on literature searches, data analysis, software development, and clinical decision support. Results. For digital activities, work shifts from execution to steering and review. Efficiency gains are greatest when expected behavior can be formalized in advance and tested automatically. In complex agentic systems, recorded sequences of reasoning steps and tool calls can quickly become too extensive for human review. Furthermore, explanations generated by the model do not reliably reflect how an output was produced. One possible step toward more reliable systems is the validation of individual components. The limited reviewability extends beyond research itself; the review of scientific articles and grant proposals is also reaching capacity limits. In pathology, curated and annotated data, researchers' own analytical skills, and institutional exchange of experience are becoming increasingly important. Conclusions. Researchers remain responsible for their results. They must determine what to delegate and how to review the results. The importance of a research question to patients, the field, and society cannot be fully assessed using formalized criteria and remains a matter of expert judgment. Agentic systems can free up time for this.

0.09

Artifact-centered Claim-aware Observability for Autonomous Scientific Agents

cs.CL cs.CYcs.HC ·2026-08-18 ·Xiangyu Yin, Ming Du, Michael H. Prince, Mathew J. Cherukara
positionhighly cited similarity 0.65
Abstract

Autonomous scientific agents now increasingly propose ideas, write code, run experiments, analyze results, and even draft papers. Observe and audit those agents are necessary but logging every model call is not enough, scientists also need to inspect the artifacts and claims that the systems produced and their relations. This is driven by the fact that failures in scientific agent systems are often distributed across several objects. A manuscript claim may cite the wrong evidence, a search process may select a degenerate candidate, a laboratory novelty claim may depend on an unstated rule, or a multi-agent plan may change without a visible trigger. Existing tracing, experiment tracking, and archival provenance tools are valuable, but their native objects do not make these scientific audit relations first-class. We argue that autonomous scientific systems should emit portable, claim-aware artifact lineage as a minimum audit layer. We propose a compact observability profile organized around individuals, operators, fitness records, lineage, archives, runs, streams, and steering commands. In this profile, scientific claims are ordinary individuals with explicit evidence bindings and verification records. The profile is intended as a semantic layer that complements current telemetry and provenance standards. Execution details can remain in OpenTelemetry. Final packages can export to PROV-O or RO-Crate standards.

0.52