CS Paper Radar

HexEval: An Evidence-Driven Hexagonal Framework for Multidimensional Scholar Assessment

Xiaokang Qu, Yiting Lin

cs.AI cs.DL ·submitted 2026-08-11 07:11 UTC · v2
position

Abstract

Scholar assessment plays a fundamental role in faculty recruitment, funding allocation, academic promotion, and talent discovery. Existing scholar assessment methods predominantly rely on bibliometric indicators and reputation proxies, while recent large language model (LLM)-based approaches mainly focus on evaluating individual research papers rather than comprehensively assessing scholars. We argue that scholar assessment should be formulated as an evidence-driven reasoning problem that jointly considers intrinsic research quality and externally verifiable scholarly behavior. To this end, we propose HexEval, an evidence-driven hexagonal framework for multidimensional scholar assessment. HexEval explicitly organizes scholar assessment into two complementary evidence layers. The intrinsic layer evaluates anonymized representative works along three dimensions, namely research rigor, methodological innovation, and scientific contribution, whereas the external layer characterizes scholars through knowledge translation, research coherence, and academic impact using heterogeneous evidence collected from GitHub, Lens, OpenAlex, and other publicly verifiable sources. Instead of producing opaque aggregate scores, HexEval preserves intermediate evidence, dimension-specific rationales, and verification signals throughout the evaluation process, enabling interpretable and auditable scholar profiles. Experiments across all six dimensions show dimension-dependent agreement with human or external reference criteria: structured calibration improves absolute agreement for intrinsic quality, while the external modules recover broad trajectory and ordinal impact signals. These results support evidence-driven reasoning over heterogeneous scholarly evidence as a promising paradigm for auditable AI-assisted scholar assessment, while exposing the coverage and attribution limitations of public scholarly data.

Comment
remove copyright information

Closest papers

Nearest neighbours by meaning across the whole index.

What It Costs to Trust a Source: A Reproducible Method for Institutional Assessment of Author Disambiguation

cs.DL ·2026-10-29 ·Simon Porter
similarity 0.63
Abstract

Records of research outputs are curated with care by universities, and the effort is shaped by the external sources from which candidate records are drawn. The curation required before the author identities in such sources can be trusted is examined, and a reproducible method for estimating its cost is set out. Four tests are described; two can be run on public data alone, and two are conducted against an institution's own curated record, taken from Symplectic Elements Discovery portals. The method is demonstrated on OpenAlex and Dimensions across 30 institutions. Of the publications attributed to matched researchers, 50.4\% were confirmed by the institutional record for OpenAlex and 69.6\% for Dimensions, and fewer trusted researcher profiles were retained from OpenAlex at a matched confidence threshold in each of the 30 institutions. Carried through to effort, integrating OpenAlex requires 1.5 times the person-days of review and entry of Dimensions even when each feed is filtered at the threshold that minimises the amount of review required.

0.27

InsightRAG: Aggregating Distributed Scholarly Judgments for Scientific Survey Generation

cs.AI cs.DL ·2026-10-20 ·Yuquan Lin, Wenshuo Zhang, Yilin Ye, Shuchang Xu, Zhao Chen +2 more
similarity 0.62
Abstract

The rapid growth of scientific literature makes survey writing increasingly challenging, motivating automatic survey generation. Yet, existing systems must search an open space of papers and an open space of concepts with no trace of how the community itself has navigated them, leading to incomplete scope, weak taxonomies, and unreliable citations. Our core insight is that author-written Related Work sections contain the human judgments needed to address both limitations: authors curate the citations relevant to their contributions and explain how the cited works relate, yielding paths for scope reconstruction and relational evidence for organization. We propose InsightRAG. Its insight-guided retrieval follows author-supported citation relations to reconstruct a verified research scope and structure the resulting relational evidence. It aggregates these judgments under a minimum-conflict objective into a taxonomy that reflects recurring author-written perspectives in the retrieved literature, guides generation, and bounds each section's evidence. Results on ten topics show gains in scope reconstruction, taxonomy quality, citation support, and reference reliability.

0.21

Data Citation for Large Language Models: A Challenge

cs.IR cs.AIcs.DBcs.DL ·2026-08-26 ·Gianmaria Silvello
similarity 0.60
Abstract · 7 pages, journal paper

Large language models increasingly mediate access to information, and a growing body of work asks whether they cite the sources behind their outputs. That work treats citation as a verification device and applies it to textual documents. Scholarly citation serves two further functions, credit and provenance, and it applies to data as much as to text. This paper argues that data citation for large language models is an open challenge, distinct from document-level citation grounding and harder to solve. We ask how such models should cite data so that outputs stay verifiable, provenance stays traceable, and credit reaches data creators and curators. We set out three research directions. Training data attribution has to turn influence estimates into references for corpora absorbed into model parameters. Data citation at inference time has to identify datasets, subsets, and query results at the right granularity and fixity. Citing knowledge graph facts has to define what a reference to a single triple denotes and how credit propagates along provenance. Progress on all three depends on joint work across the database, information retrieval, knowledge representation, and artificial intelligence communities.

0.14

The conservative turn in science: The changing character of knowledge recombination

cs.SI cs.DL ·2026-09-08 ·Cheng Zhang, Xinkun Liang, Yi Bu, Hongkan Chen
similarity 0.58
Abstract

This study examines how the dominant mode of cross-disciplinary knowledge combination has changed over time. It addresses a limitation of existing studies that document declining disruptiveness at the level of individual works but do not reveal whether the mechanisms of cross-disciplinary knowledge flow have themselves shifted. Drawing on 56 million publications and 816 million citations from OpenAlex (1960-2025), the study classifies cross-disciplinary citations into five types based on the set relationship between the subfield portfolios of citing and cited publications and introduces a semantic-distance measure derived from SPECTER-v2 embeddings of publication abstracts. It subsequently demonstrates that cross-disciplinary citations have shifted markedly away from configurations with no shared disciplinary ground toward types built on partially overlapping knowledge bases, a reorientation further reflected in a steady decline in average semantic distance between citing and cited fields. This shift co-occurs with a diffusion of brokerage positions across the network and a greater reliance on older literature, consistent with a system that has become broader and more decentralized in its interdisciplinary reach. Finally, this study discusses the implications of these findings for science policy and for the design of cross-disciplinary funding mechanisms.

0.08

ScholarStack: Layered Research Asset Orchestration and Cross-Task Reuse for Scientific Agents

cs.AI ·2026-09-20 ·ScholarSeed AI Team, Caoqinwei Gong, Xue Jiang, Wei Luo, Xiaoyu Qiu +15 more
similarity 0.58
Abstract

Scientific agents support a range of literature-based research tasks, such as retrieval, question answering, evidence-grounded generation, and claim assessment. Most existing systems, however, are organized around individual tasks: the same papers are repeatedly retrieved, segmented, and interpreted, and the understanding built in one task is difficult to reuse in the next. We present ScholarStack, a layered research asset framework that compiles a paper collection into reusable, versioned, and provenance-preserving assets at three complementary levels: source-grounded paper-level statements, domain-level organization, and evidence-grounded cross-paper syntheses. A common access interface returns task-specific views at the evidence granularity each task requires, preserving study conditions, source traceability, and verification status. We instantiate the framework on four task families spanning ten task settings, comparing agents that use the compiled assets with task-specific baselines under matched base models. Quality gains concentrate on tasks that require cross-paper evidence, such as multi-paper question answering and literature review generation, and query-time token cost falls on every task where it is measured, with assets compiled once and reused across tasks. These results suggest that layered research assets can serve as shared infrastructure for scientific agents, shifting literature-based assistance from isolated document processing toward cumulative, evidence-grounded workflows.

0.14

The Scientific Research Taste Benchmark (SciTaste-Bench): A Multi-Disciplinary Benchmark for Assessing Large Language Models' Judgment of Research Taste in Scientific Papers

cs.DL ·2026-10-09 ·Xinyi Gao, Xiaogang Li, Jialu Nie, Yuting Fu, Jingye Xu +7 more
benchmarkcode/data similarity 0.58
Abstract

Whether large language models can assess the innovativeness of academic publications matters for literature reviews, research funding screening, and academic recommendations. We introduce the Scientific Research Taste Benchmark (SciTaste), a gold-standard benchmark covering six disciplines: physics, mathematics, chemistry, biology, medicine, and finance. SciTaste measures research taste, that is, the ability to judge a paper's scientific value and potential, of which innovativeness is one of seven dimensions. It integrates metadata from NASA ADS, OpenAlex, and arXiv, aligns papers to field-specific classification systems, and scores each paper with the seven-dimensional Academic Taste Index. Dimension weights come from a bi-objective machine-learning approach (five regressors, including random forests) and are shared across all six disciplines. The benchmark includes 300 selected papers (50 per discipline) and a backup pool of 5,700 papers (950 per field) covering 1996-2023. SciTaste supports four evaluation modes: pure_qa, native_search, our structured tool-calling agent DiM (9 bibliometric tools in ADS/OpenAlex), and dimagent_v3_native. Outputs are evaluated by two complementary schemes: correlation with dimension-level accuracy, and WAPE relative error. Across ten state-of-the-art LLMs, tool-based structured evaluation generally outperformed toolless and raw web search modes, though gains were model-dependent. Physics is easiest; chemistry, biology, and medicine remain more challenging. SciTaste supplies the value-judgment substrate auto-research systems need to screen, rank, and prioritize candidate ideas. Dataset, harness, and scoring code are available on Hugging Face at https://huggingface.co/datasets/skylenage/SciTaste-benchmark.

0.38

Agent-Enhanced Heterogeneous Graph RAG for Academic Question Answering

cs.SI ·2026-09-01 ·Runsong Jia, Mengjia Wu, Ying Ding, Jie Lu, Yi Zhang
publishedhighly cited similarity 0.58
Abstract · Proceedings of the ACM Web Conference 2026

Academic question answering requires reasoning over heterogeneous scholarly graphs, where queries range from simple attribute lookups to multi-hop inference across author--paper--venue structures. Existing retrieval-augmented generation (RAG) systems struggle in this setting due to three limitations: (1) fixed retrieval strategies that do not adapt to varying query complexity, (2) the absence of sufficiency evaluation leading to incomplete or misaligned evidence, and (3) a lack of structured verification against graph facts. To address these issues, we propose an agentic heterogeneous graph RAG method that transforms the three core stages of the RAG pipeline into explicit agentic decision steps. A query-aware retrieval agent analyzes query type and selects an appropriate graph traversal strategy; a sufficiency-aware reranking agent assesses evidence completeness and adaptively expands the retrieved subgraph; and a graph-grounded verification agent checks entity, relation, and attribute correctness before finalizing the answer. Experiments on heterogeneous graphs constructed from OpenAlex and DBLP suggest that our method consistently outperforms strong LLM, graph-augmented RAG, and agent-based baselines.

0.49

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

cs.AI cs.DB ·2026-08-19 ·Valentin Romanov, Monique Bax, Steven Niederer
similarity 0.58
Abstract

Accurately extracting nuanced, contextualized data from research articles is laborious and time intensive. Here, we investigate the performance of frontier, browser-based large language models (LLMs) to extract highly contextualized information. We demonstrate four escalating workflows, 1) given an expert curated prompt and research articles, most frontier LLMs perform well at data extraction, however can struggle with interpreting scientific context and nuance, 2) given simple instructions, LLMs can author their own prompts which were almost as eNective as expert-written prompts, 3) autonomous discovery of research literature was diNicult, agents either missed or hallucinated references, and 4) LLMs can create new datasets from published guidelines that closely match human-expert judges, but still require a human-in-the-loop. Together, these findings define an auditable division of labour in which experts specify the evidence standard, models cross-check repeated extractions and researchers resolve disputed cases, providing a practical route to scaling scientific data curation without relinquishing expert oversight.

0.18