CS Paper Radar

SGHA: Evidence-Grounded Research Problem Discovery with Local Language Models

Sarvesh Gharat, Junpei Komiyama

cs.AI cs.LG ·submitted 2026-08-18 08:27 UTC · v1
code/data

Abstract

Recent efforts toward fully automated AI scientists have demonstrated that language-model agents can generate hypotheses, execute experiments, and draft scientific manuscripts. However, during the early stages of research, when research problems are formulated, these AI scientists often rely heavily on proprietary frontier models. Their proposals are shaped by opaque parametric knowledge and by literature searches conditioned on the proposals themselves. Such knowledge is effectively a black box, and this dependence makes the evidential basis and validity of generated research problems difficult to audit and leaves the process vulnerable to model-specific hallucinations and biases. Furthermore, if proprietary research materials are transmitted to external APIs, the use of these models creates confidentiality, privacy, and data-governance concerns. We introduce the Structural Gap Hypothesis Agent (SGHA), a fully automated, corpus-first research-problem discovery system that runs entirely on a local LLM. SGHA structures a scientific literature corpus into evidence-linked paper objects and a typed evidence graph, detects unresolved structural patterns across papers, screens candidate gaps before formulation, and produces traceable research-problem families. In particular, it is able to output assumptions, objectives, success criteria, and remaining ambiguities. All LLM-based components of SGHA are executed using a locally served open-weight 9B language model, without requiring proprietary frontier-model APIs. We compare SGHA with the AI Scientist-v2 idea formulation module in five machine-learning domains. Our results suggest that explicit corpus structure and evidence-constrained reasoning can support promising, inspectable research-problem formulation without relying on frontier models during generation or verification.

Comment
Link to Code and artifacts: https://github.com/SarveshVGharat/structural-gap-hypothesis-agent

Closest papers

Nearest neighbours by meaning across the whole index.

Modelpedia: A Catalog of Model Findings for the Meta-Science of AI

cs.LG ·2026-09-01 ·Franciszek Bernat, Dawid Płudowski, Michał Jan Włodarczyk, Luca Longo, Jianlong Zhou +4 more
code/data similarity 0.71
Abstract · For the website, see: https://credibleai.github.io/modelpedia For the codebase, see: https://github.com/CredibleAI/modelpedia

Scientific knowledge about AI models is produced faster than the community can organize it. Every few months a new foundation model reshapes the field and hundreds of papers, blogs, and technical reports document how each behaves or fails. Yet, these findings remain scattered and effectively unretrievable. To address this gap we present Modelpedia, an automated, LLM-assisted framework that extracts findings about models from published papers, links it to the model, dataset, method, and concept it concerns, and aggregates the result into a searchable public catalog. Applying the prototype to accepted ICLR 2024 and 2025 papers, we extract over a thousand findings and, treating the catalog itself as an object of study, run a meta-analysis of how the community investigates models. Now, we invite the community to explore, contribute to, and build on the open catalog, and to help establish model findings as a shared foundation for the meta-science of AI.

0.22

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

cs.AI cs.DB ·2026-08-19 ·Valentin Romanov, Monique Bax, Steven Niederer
similarity 0.71
Abstract

Accurately extracting nuanced, contextualized data from research articles is laborious and time intensive. Here, we investigate the performance of frontier, browser-based large language models (LLMs) to extract highly contextualized information. We demonstrate four escalating workflows, 1) given an expert curated prompt and research articles, most frontier LLMs perform well at data extraction, however can struggle with interpreting scientific context and nuance, 2) given simple instructions, LLMs can author their own prompts which were almost as eNective as expert-written prompts, 3) autonomous discovery of research literature was diNicult, agents either missed or hallucinated references, and 4) LLMs can create new datasets from published guidelines that closely match human-expert judges, but still require a human-in-the-loop. Together, these findings define an auditable division of labour in which experts specify the evidence standard, models cross-check repeated extractions and researchers resolve disputed cases, providing a practical route to scaling scientific data curation without relinquishing expert oversight.

0.18

HyGRAIL: Cost-Aware and Evidence-Grounded Scientific Hypothesis Discovery over Knowledge Graphs

cs.CL ·2026-09-02 ·Yihang Sun, Zhihan Zhu, Zhiyuan Jiang, Jingyi Ge, Zixuan Li +1 more
similarity 0.70
Abstract

Scientific knowledge graphs organize entities and relations extracted from scientific literature, but they remain inherently incomplete. Missing typed links in such graphs can therefore represent plausible scientific hypotheses, such as unexplored associations between materials and applications. However, scientific hypothesis discovery is challenging because true discoveries are extremely sparse among typed candidate pairs: graph neural networks (GNNs) are efficient but unreliable for ambiguous cases, while large language models (LLMs) are knowledgeable but too costly to apply exhaustively and are not naturally grounded in graph structures. We propose HyGRAIL, a cost-aware and evidence-grounded framework that combines heterogeneous GNN triage with LLM-based hypothesis review. HyGRAIL first uses a GNN to score candidate hypotheses and identify a validation-calibrated ambiguous region, routing only graph-uncertain cases to LLM review. For each routed hypothesis, HyGRAIL retrieves node-level associations and multi-hop relational paths from the knowledge graph (KG), then converts this structured evidence into natural language through template-based or LLM-based naturalization. An LLM review agent finally judges each hard hypothesis using the naturalized evidence and validation-selected decision criteria. On MatKG, HyGRAIL achieves the best F1 score of 0.429, improving over the strongest prior baseline by 0.242 F1 points and over the GNN-only baseline by 0.322. Meanwhile, GNN triage reduces the LLM call rate by 54.36% on average. Ablation studies further show that retrieved graph evidence is crucial for reliable hypothesis verification and that compact, two-sided evidence is more effective than simply increasing retrieval quantity.

0.27

Can LLMs Discover Scientific Laws in Real and Parallel Worlds?

cs.AI cs.LG ·2026-09-01 ·Yiming Huang, Ziche Liu, Junxia Cui, Zhuohang Wu, Yiqian Wang +9 more
code/data similarity 0.69
Abstract · 41 pages, 14 figures

Scientific law discovery has long been central to scientific progress, proceeding through iterative cycles of generating hypotheses, testing them against empirical evidence, and refining them under scientific constraints. As large language models (LLMs) become increasingly involved in scientific research, whether they can discover scientific laws and how to evaluate this ability remain open questions. A central evaluation challenge is to move beyond familiar published equations while keeping discovery tasks grounded in scientific data and constraints. We introduce SciLaws-Bench, a curated collection of scientific task packages grounded in the source literature, each linking a scientific problem, supporting data, published reference equations, and scientific-validity rubrics. Through agent-assisted curation and human verification, we assemble 118 problems spanning six disciplines, drawing on 381 papers, 291 candidate laws, and roughly 8M data points. Each problem supports two complementary evaluation settings. SciLaws-Real uses fixed scientific data to evaluate proposed laws for held-out predictive fit and scientific validity. SciLaws-Parallel evaluates recovery of a newly synthesized structural variant of a published equation through active queries to a simulator calibrated to the source data. Our evaluation reveals three limitations: good predictive fit need not imply scientific validity, recovering a published formula does not establish recovery of its new structural terms, and candidate selection remains a bottleneck in scientific law discovery. Project page: https://yiyihum.github.io/SciLaws-Bench

0.58

IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?

cs.CL ·2026-11-02 ·Shanshan Zhong, Chen Xu, Jingjie Ning, Zhengzhong Liu, Aviral Kumar +1 more
benchmarkcommunity pick similarity 0.68
Abstract

Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26\% of the human level on citation matching and only 18\% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard (https://cxcscmu.github.io/IdeaTaste) that benchmarks contemporary AI proposals against future publication cycles.

0.28

SciLENS: RL-Driven Autonomous Agents for Scientific Localized Evidence Navigation and Synthesis

cs.IR ·2026-09-03 ·Leqi Zheng, Jinbo Su, Yuying Li, Chaokun Wang, Weiping Wang +8 more
top venuecode/data similarity 0.68
Abstract

Scientific literature synthesis agents increasingly rely on proprietary online services, limiting reproducibility, privacy, and offline deployment. To address this challenge, we introduce SciLENS Scientific Localized Evidence Navigation and Synthesis), a fully local autonomous agent framework operating on a dual-tier infrastructure indexing approximately 12 million academic records. SciLENS pioneers the integration of structural visualization as an actionable tool within the reasoning loop, enabling the agent to compress complex citation topologies into validated data-driven charts and thereby mitigate context exhaustion during macro-level synthesis. To train the agent without human annotation, we develop an automated data synthesis pipeline that extracts multi-hop subgraphs from a citation knowledge graph, verified by cross-model consensus among 20 frontier models. The agent is subsequently aligned through a reverse-decomposition rubric strategy that provides fine-grained process rewards for early planning and strict evidence grounding. Evaluations across six scientific benchmarks encompassing standard QA, citation accuracy, factual reasoning, and structural synthesis demonstrate that SciLENS significantly outperforms open-source baselines and achieves performance comparable to GPT-5.2 and Gemini-3.0-pro. Our source code and data are released at https://github.com/LQgdwind/SciLENS.

0.78

A Stateful Research Information Infrastructure for Scientific Discovery

cs.DL cs.IR ·2026-10-10 ·Lingyuan Kong, Zhengran Wu
similarity 0.68
Abstract · Project Website: https://gewu-insight.com

Artificial intelligence (AI) is being applied to an expanding range of scientific activities, including literature analysis, hypothesis generation, software development and automated experimentation. Yet these capabilities rely on a more fundamental condition: scientific information must remain identifiable and traceable across systems and over time. In practice, different versions of a work and their associated records and files are often scattered across services and local directories. This fragmentation forces researchers to identify and contextualize the same work repeatedly, disrupting continuity from discovery to interpretation and reuse. Here we present Gewu Insight, a research information infrastructure that addresses this discontinuity by organizing scientific works as persistent, stateful research objects. It unifies venue-, topic- and author-based discovery records through a shared paper identity. Within a local library, metadata, versioned resources and research context can then accumulate around the same work. These capabilities are integrated into an iterative workflow for scientist-led, AI-assisted research, allowing earlier interpretations to be revisited and refined. A bounded evaluation showed high candidate precision across the three discovery modes, with paper identity, resource distinctions and research state preserved in controlled workflows. By making continuity an explicit property of research infrastructure, Gewu Insight can serve as an inspectable scientific memory layer shared by researchers and AI assistants, supporting human judgment and providing a foundation for future, more epistemically grounded scientific AI.

0.15

From Papers to Mechanisms: An Evidence-Grounded Knowledge Substrate for Scientific Language Models

cs.AI ·2026-10-05 ·Qiuhui Chen, Yibo Liu, Tao Dai, Jiafan Lu, Zhenglei Zhou +1 more
similarity 0.68
Abstract · 8 pages, 5 figures, 2 tables

Scientific language models often access literature through untyped text chunks, which fragment the functional and evidential structure required for mechanism-rich questions. We introduce an evidence-grounded mechanism knowledge substrate that organizes scientific literature into provenance-linked evidence units, role-typed entities, and directed mechanism paths. We instantiate it as MS$^3$, a Material-Sensor-Signal-System schema for conductive-fiber flexible sensors, over 13,689 papers, 131,083 evidence items, and 26,648 mechanism objects. On in-domain and coverage-shift question-answering benchmarks, we compare closed-book generation, Web search, Raw-PDF RAG, and MS$^3$ retrieval across ten language models. MS$^3$ improves macro-averaged scientific correctness. It also improves citation entailment and answer completeness. These results support mechanism substrates as a reliable representation layer for scientific language models and motivate a source-repair workflow in which insufficient MS$^3$ evidence triggers targeted retrieval from its linked papers rather than assuming that a user has already supplied the correct PDFs.

0.19