CS Paper Radar

From FAIR to AWARE: Data Readiness for Agentic AI in the Earth Sciences

Ivan Kuznetsov, Dmitrii Pantiukhin, Sonja Hänzelmann, Fabian Wachsmann, Jonas Sølvsteen, Daniel Wiesmann, Thomas Jung, Nikolay Koldunov

cs.AI physics.ao-phphysics.geo-ph ·submitted 2026-11-18 21:36 UTC · v1

Abstract

Artificial intelligence in the Earth sciences is beginning to act: prototype agents already search archives, run simulations and draft analyses. Scientific data were curated for researchers who bring expert judgement; an autonomous agent cannot be assumed to supply it. Data can satisfy the FAIR principles and still mislead a machine acting alone. We propose that Earth-science data become AWARE (Actionable, Well-described, Auditable, Recipe-equipped, Epistemically labelled), an agent-specific requirements layer that extends FAIR, within a three-part agenda: AWARE data, bounded tools, layered validation. The ingredients exist across community infrastructure. We outline a staged, measurable path to adoption, intended to reduce silent error.

Comment
34 pages (27 main text, 7 Supplementary Information), 1 figure, 1 table, 2 boxes

Closest papers

Nearest neighbours by meaning across the whole index.

EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards

cs.AI ·2026-08-24 ·Zhiqing Cui, Xinxiang Yin, Yihong Tang, Xinglang Zhang, Yuanzhe Hu +9 more
benchmark similarity 0.71
Abstract

Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality. Natural hazards make this work consequential because incomplete evidence can change estimates of severity, exposure, and mechanism. We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations. Its 405 reproducible tasks are grounded in 199 documented events and 19 hazard families. Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer. We provide executable ground truth that decomposes each task into fine-grained answer units, together with task-specific rubrics that assess the supporting research process while allowing multiple valid paths. We evaluate 25 model and agent systems under a controlled tool-using protocol, then use controlled studies to locate failures in evidence access, tool selection, memory, reasoning, interaction, and scientific execution. Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%. The gap shows that current agents often complete individual steps without maintaining a consistent chain across evidence, scales, units, calculations, and physical interpretation. EarthVerse provides a reproducible basis for measuring end-to-end scientific reliability in dynamic Earth systems.

0.21

From FAIR to AI-FAIR Metadata-First Infrastructure for AI-Enabled Science

cs.AI cs.DBcs.HCcs.IR+1 ·2026-11-02 ·Elizabeth G. Campolongo, Eric R. Sokol, Marty Downs, Gina Glenne, Gregory E. Maurer +7 more
code/data similarity 0.71
Abstract · 8 pages, 3 figures, Code available at: https://github.com/FAIR4AI-Bio/evaluator

Artificial intelligence (AI) technologies and their capabilities have advanced rapidly over the past few years, leading to widespread interest and adoption in the biological and ecological sciences. However, the data infrastructure and computational resources that provide the foundation for AI workflows in ecology and biodiversity science remain fragmented. Moreover, the flow of value remains largely unidirectional: data move from collectors, through repositories and aggregators into models, and finally to research products or management actions. The derived value, such as corrected taxonomies, annotations, quality flags, and insights, rarely returns to source communities. We propose an extension of the original FAIR principles to address the needs of AI-ready infrastructure: the AI-FAIR metadata criteria, which place AI-enabled science in the context of a bidirectional Question-Data-Model triangle. This framework is designed to facilitate large-scale content-based analysis across diverse collections and ensure that the scientists generating the data benefit from the AI-enabled advances built on their data. Alongside these criteria, we introduce an agentic evaluation workflow that both rates existing datasets on the AI-FAIR and traditional FAIR criteria and provides actionable feedback where critical metadata is missing or incomplete. This modular dataset evaluation tool, paired with a domain-appropriate checklist, translates these principles into a reusable assessment of machine-learning readiness, AI-readiness for specified tasks, traceability, and ethical and CARE-aligned stewardship. Through this approach we aim to create a community-informed system to close the model data loop, so that AI also enriches data to further enable scientific discovery.

0.23

Delegating Earth science model calibration to knowledge-guided LLM agents

cs.AI cs.CE ·2026-10-13 ·Zhanliang Zhu, Ziwei Li, Liujun Zhu, Yuchen Liu, Junliang Jin +1 more
similarity 0.69
Abstract

Applying process-based Earth science models to new sites and questions often requires expert calibration decisions. Reliable delegation to language-model agents must preserve scientific competence and reproducibility. Here we develop a general calibration harness (GCH) for end-to-end delegation from a calibration request to assessed results. Constrained by a knowledge infrastructure (KI) and a validation convention, agents define parameter domains, observation mappings and objectives, and generate model-specific code. GCH checks the formulations against the same declarations and freezes them before numerical search and assessment, which require no further agent judgement. Agent-authored workflows executed across six model families, with assessment identifying unmet criteria. In snow-model tests, five of five knowledge-guided specifications executed on the first attempt, compared with none of five given only the model name, achieving validation Nash-Sutcliffe efficiency (NSE) of 0.81-0.94 against 0.57 for defaults. Under a common objective, independent GR4J formulations yielded scores within 0.001 NSE; re-evaluating eleven saved parameter vectors reproduced their metrics. In ten paired VIC tests with matched budgets, fixed search was non-inferior to continuing agent proposals (margin, -0.02 NSE; median difference, -0.003). This separation enables end-to-end calibration delegation with inspectable scientific choices and reproducible numerical evaluation.

0.23

Agentic Economies for Autonomous Scientific Discovery

cs.CY ·2026-09-25 ·Nenad Tomasev, Matija Franklin, Atoosa Kasirzadeh, Vivek Natarajan, Alan Karthikesalingam +2 more
similarity 0.68
Abstract

Recent advances in agentic Artificial Intelligence (AI) systems have marked a shift in AI for Science: moving away from the use of individual AI systems for narrow task execution, toward multi-agent systems capable of orchestrating complex, end-to-end research workflows and performing (semi-)autonomous scientific discovery. The development of multi-agent AI-for-science systems has primarily focused on improving the cognitive capabilities of AI systems, specifically by making advanced reasoning and hypothesis generation more reliable. However, focusing only on cognitive capability improvement could ignore appropriate management of resources, a key bottleneck in scientific discovery. Testing and validating scientific hypotheses and experiments is, physically and economically, resource-intensive and resources are limited. This means that to make significant advancements in autonomous scientific discovery, such as improving human-AI co-scientist complementarity or reaching a truly closed-loop automated process, we must pair ongoing improvements in reasoning capabilities with robust resource management. In this paper, we outline an infrastructure for AI resource management by developing the necessary foundations of scientific agent economies, markets, and institutions. The aim of this infrastructure is to empower AI agents and human scientists to effectively (i) collaborate and establish research priorities, (ii) assign credit, (iii) track accountability and liability, and (iv) safeguard against malicious use and information security risks. Finally, we engage with the macro-level societal implications. of (semi-)autonomous scientific discovery to inform the development of governance policies ensuring an equitable distribution of AI-driven discoveries and derivative future technologies.

0.06

Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

cs.AI cs.MA ·2026-10-06 ·Wenyu Du, Stephen Chung
similarity 0.67
Abstract · preprint

Recent AI systems have made rapid progress in scientific discovery when given well-defined metrics, but whether they can autonomously undertake open-ended scientific discovery remains unclear. We investigate AI's ability to tackle open-ended tasks in Station, an open-world environment in which multiple agents simulate a scientific ecosystem. To tackle challenges specific to open-ended tasks, we propose augmenting Station with two mechanisms: a Supervisor mechanism and periodic Meta Reflection, which encourage persistent exploration even when intermediate metrics are lacking. We construct open-ended tasks from three recent oral papers presented at ICLR. We give agents the main research question studied in each paper while withholding the paper's results and disabling web access. We then measure how many of the original findings-partitioned into individual criteria-agents rediscover. We find that Station rediscovers 62.7% of the criteria on average, compared with 15.4% for Codex Multiagent-v2 and 14.4-20.6% for AI Scientist-v2. Ablation and behavioral analyses indicate that adding the two mechanisms together improves research coverage and continuity. We further evaluate Station on two open-ended tasks without oracle papers and find that some of the discoveries made by the agents closely match discoveries reported by researchers after the knowledge cutoff date. Together, these results indicate that a suitable environment can enable agents to autonomously make meaningful progress in open-ended scientific discovery.

0.16

FOXDEN: FAIR Services for AI-Ready Scientific Datasets

cs.DL ·2026-09-02 ·Valentin Kuznetsov, Werner M. Sun, Keara Soloway, Rolf Verberg, Katherine S. Shanks +3 more
similarity 0.66
Abstract · 21 pages, 7 figures, typos fixed

Scientific datasets are most compatible with AI workflows when they are described by rich, machine-readable metadata and provenance information, following the FAIR guiding principles. The FAIR Open-Science Extensible Data Exchange Network (FOXDEN) is a set of cyberinfrastructure building blocks developed at the Cornell High Energy Synchrotron Source (CHESS) for annotating raw, reduced, and analyzed datasets with both structured and unstructured metadata and provenance records. It also allows researchers to publish these records with Digital Object Identifiers (DOIs) to create AI-ready datasets. We describe FOXDEN's architecture and its deployment at CHESS and at the National High Magnetic Field Laboratory, where it serves as a foundation for future agentic scientific workflows.

0.17

ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments

cs.CL cs.CY ·2026-09-16 ·Hejia Geng, Zesen Huang, Haoyang Li, Wenbin Li, Koutian Wu +58 more
code/datahighly cited similarity 0.66
Abstract · Code: https://github.com/aitofound/ScienceIDE

Scientific code repositories encode decades of human knowledge in executable models, methods, and tools. Yet fragmented toolchains, implicit domain conventions, and specialized correctness criteria make this knowledge difficult to convert into reliable learning experience-a challenge we call the scientific experience bottleneck. We introduce ScienceIDE, infrastructure for turning the world's scientific code into programmable environments for scientific agents. Guided by expert-defined scientific cases and acceptance criteria, agents transform repositories into executable environments that support task generation, execution, and scientific verification. These environments provide a shared foundation for supervised fine-tuning, reinforcement learning, and evaluation. Using verified interaction trajectories, we train PhAI-IDE-72B/9B/4B. The model family shows gains in held-out scientific-code repair and across selected general-purpose benchmarks in code, reasoning, and knowledge, providing evidence of positive transfer from scientific experience to broader capabilities. ScienceIDE lays the foundation for an integrated workspace for agent learning and scientific practice, making humanity's scientific software a shared substrate for developing scientific intelligence. Code: https://github.com/aitofound/ScienceIDE

0.45

DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?

cs.AI ·2026-09-27 ·Nan Huang, Mario Tapia-Pacheco, Kun Zhou, Yiming Huang, Kevin José Barrientos Díaz +2 more
benchmarkcode/data similarity 0.66
Abstract · Project website: https://discern-benchmark.github.io/discern-website/. Updated to include the project website and planned life-science verif…

Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially reducing routine scientific workload while allowing scientists to focus on interpretation and discovery. Existing benchmarks often only assess analytical task completion or hypothesis generation separately rather than testing whether reliable evidence supports valid and novel claims. We introduce DISCERN (Data Integrity and Scientific Capability: Evidence, Reasoning, and Novelty), a controlled benchmark on real, publicly available datasets that evaluates three key levels of an automated research workflow. The first two levels test data integrity and analysis verification under confounds and tool traps, while the third tests hypothesis generation and revision under adversarial review, including counterfactual cases in which evidence consistent with real data and documented scientific phenomena conflicts with established expectations, motivating alternative explanations and testable hypotheses. Across 203 tasks, eight life-science tracks, and eight models, DISCERN shows that strong aggregate performance can mask level-specific weaknesses. Agents earn perfect scores in only 60.8% of Level 1, 34.2% of Level 2, and 0.6% of Level 3 evaluations, with penalties attributed to rejection of sound data, failure to carry recognized limitations into conclusions, and wide variation in hypothesis production. Cross-track rankings by token and code use are substantially more stable than rankings by evidence judgment, suggesting greater consistency in computational effort than in evidence-based reasoning. These profiles identify opportunities for supervised scientific assistance, but current agents do not yet demonstrate reliable autonomous analysis or discovery. Code and data: https://huggingface.co/datasets/discern-bench-anon/discern-benchmark

0.23