CS Paper Radar

Measuring the Novelty of Biomedical Papers Using the Latent Distances between Knowledge Units

Yi Zhao, Heng Zhang, Yuzhuo Wang, Wenqing Wu, Tong Bao, Chengzhi Zhang

cs.DL cs.CL ·submitted 2026-09-04 14:14 UTC · v1
published

Abstract

Measuring the novelty of scientific papers is a central concern in research evaluation and scientometrics. From a recombination perspective, prior studies have largely focused on the co-occurrence of knowledge units to assess the novelty of scientific papers. However, these studies often overlook other relationships between knowledge units. This narrow view may result in inaccurate or incomplete evaluations of novelty for scientific papers. To fill this gap, this study introduces a comprehensive novelty measurement that incorporates three types of relationships between knowledge units: network, semantic, and hierarchical. These relationships are used to quantify the latent distances among knowledge units. Using a dataset of 142,036 articles published in PLoS ONE and a validation dataset from the H1 Connect platform, our results demonstrate that (1) each relationship type captures distinct latent distances between MeSH terms; (2) compared to the widely used indicators proposed by Uzzi et al. (2013), our measures show stronger alignment with peer judgements; and (3) combining all three distance metrics yields more effective identification of novel papers than using any single perspective alone.

Journal ref
Journal of Information Science, 2026

Closest papers

Nearest neighbours by meaning across the whole index.

NoveltyScope: A Multi-Agent Research Assistant for Evidence-Grounded Novelty Assessment of Scientific Papers

cs.IR cs.AI ·2026-10-15 ·Yueran Tian, Huizhi Liang
similarity 0.67
Abstract · 13 pages, 1 figure

NoveltyScope is a prototype that compares a scientific paper with retrieved prior work and produces a structured report and a five-level novelty score. The system extracts the core task and at most two author-claimed contributions, then searches a closed local corpus using BM25 and dense retrieval with reciprocal rank fusion. Contribution comparisons request paired passages for potential refutations, and quotations are checked against cached source text. Four scoring roles (Prosecutor, Defender, Judge, and Critic) use the resulting evidence, with at most one revision of the Judge's score. Evaluation uses 50 class-balanced RINoBench samples whose idea-level labels are retained for full-paper input. Mean absolute error is 1.48 and 1.26 for single-LLM and multi-agent scoring with Gemini, respectively, and 1.22 and 1.16 with Qwen3. The vector-similarity baseline has an error of 1.68. A HiCLIP case study illustrates the contribution comparisons and paired evidence passages. Overall, the results demonstrate the feasibility of evidence-grounded novelty assessment and show that the multi-agent configurations achieve lower scoring errors than their single-LLM counterparts across both evaluated models.

0.17

Large Language Models Extend Known Research But Do Not Create New Research Directions

cs.AI ·2026-10-12 ·Zhihua Liu, Kurt Butler, Damian Machlanski, Danilo Bueno, Sotirios A. Tsaftaris
similarity 0.63
Abstract

The success of autonomous research systems relies on the quality of the scientific hypotheses they generate. Recent studies report that human reviewers often rate LLM-generated ideas as more novel than those from human experts, raising hope that AI can accelerate scientific discovery. Yet it is unclear whether these models can do more than extend existing knowledge. Can LLMs drive genuine innovation by revealing new research directions, or are they limited to repeating and recombining what science already knows? This unresolved question lies at the heart of discussions about AI and the future of science. To address this, we distinguish two types of novelty: distance, which measures how far a hypothesis lies from existing work, and direction, which captures whether the hypothesis pursues new lines of inquiry beyond those already present. These two measures diverge sharply for LLM-generated work. We represent scientific papers in a shared conceptual space using a pretrained scientific-paper embedding model. Within this space, we embed 26.9 million human-authored papers across 1,325 research topics in six natural-science fields and 524,308 LLM-generated hypotheses from eight models. Distance is measured using existing metrics. To quantify direction, we introduce the orthogonal novelty score (ONS). LLMs consistently generate ideas more distant from existing work than human-authored papers, up to 16\% further when the LLM is given the papers farthest from the centre of the topic. However, LLM hypotheses have a 15--23\% lower median ONS than published papers from the same topic. Thus, LLMs move further from what is already known, but remain within established directions. This pattern persists across changes in various configurations. As a result, LLMs excel at extending the frontier but the creation of new directions is still led by human scientists.

0.29

MUSES: A Benchmark for Prospective Intellectual-Roots Retrieval

cs.IR ·2026-08-31 ·Rohan Pandey, Sunjae Kwon, Hong Yu
benchmark similarity 0.63
Abstract

Scientific discovery depends on finding prior literature that shapes what comes next. Existing retrieval systems optimize for relevance and popularity, often favoring central papers over less familiar works that later prove generative. We introduce \textbf{MUSES}, a million-instance benchmark for prospective intellectual-roots retrieval over a fixed 2.33M-paper corpus, with roughly 140K test instances per familiarity tier. To our knowledge, it is the first prospective benchmark at this scale with a shared retrieval task and author-confirmed paper-level root labels. Alongside it, \textbf{CiteRoots} pairs a scalable rhetorical layer over local citation text (LLM judge $κ= 0.896$ versus human gold) with a paper-level author-endorsed layer ($n = 1{,}518$ generative-inspiration pairs from 753 focal papers). MUSES organizes difficulty along two axes: a \emph{familiarity} axis spanning CiteNext, CiteNew, and CiteNew-Isolated, and a \emph{functional} axis spanning broad citations, rhetorical roots, and author-endorsed roots. Across 9 method classes, a lean multi-centroid retriever built on SPECTER2 is strongest. Hit@100 falls from 0.534 on CiteNext to 0.424 on CiteNew, 0.205 on rhetorical CiteNew, and 0.171 on author-endorsed CiteNew, a $3.1\times$ decline. In a registered eight-lens full-test audit, roughly half of broad-tier test instances remain unsolved at K=1{,}000. Rhetorical role and author endorsement are distinct: the same judge agrees with endorsement at $κ= 0.037$. We release MUSES, both CiteRoots layers, and a distilled open companion judge for future work on prospective retrieval and intellectual roots.

0.36

NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment

cs.AI ·2026-09-10 ·Guoqiang Zhang, Kexin Tan, Ming Zhang, Li Ju, Wenqing Jing +10 more
benchmark similarity 0.62
Abstract

Large language models (LLMs) are increasingly used in peer review at major AI conferences, yet novelty remains a persistent weak point. Existing benchmarks assess novelty as a single holistic score, making it difficult to diagnose which dimension a model misjudges or whether its evidence is faithful. We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis. The benchmark contains 619 paper pairs and 50 multi-paper sets, drawn from two expert sources: ICLR reviewer overlap claims and survey co-citations. Instances are independently labeled along three dimensions: task, problem, and method, capturing application goals, technical challenges, and solution approaches. We propose a cascading diagnostic pipeline that verifies per-dimension correctness, evidence grounding, and logical support. Evaluation of 18 LLMs shows hallucination rates ranging from 0% to 39% across dimensions, and among non-hallucinated correct-positive judgments, over 70% cite evidence fails to logically support the stated reason. The best-performing model, GPT-5.5, achieves 43-72% Verified F1 across dimensions, while most models retain less than half of their raw F1 after faithfulness verification. These results suggest that current LLMs remain far from reliable scientific novelty assessment, particularly when correctness is conditioned on faithful evidence grounding.

0.16

Building Weighted PubMed Citation Graphs at Scale: A Pipeline and an Audit of Edge-Similarity Features

cs.SI ·2026-10-31 ·Hossein Mohasel Arjomandi
code/data similarity 0.62
Abstract · 6 pages, 6 tables. Submitted to the BTSD workshop at IEEE BigData 2026

Studies of scientific literature often combine citation links with article metadata, but the data preparation needed to do so at the scale of millions of papers is laborious and error-prone. This paper describes a pipeline that turns PubMed metadata and a citation network into a graph whose edges carry seven similarity features: bibliographic coupling and co-citation (each as a count and as a Jaccard coefficient), publication-year proximity, MeSH-term similarity, and similarity of MedCPT title-abstract embeddings. Applied to a citation network derived from the cell biology research literature, the pipeline yields 7,445,898 nodes and 57,512,086 edges with complete year, MeSH, title, and abstract metadata. The paper reports how nodes are matched across sources, how duplicate identifiers are resolved, and which intermediate tables grew to hundreds of gigabytes. An audit of the full feature table shows that 47.2% of the edges share neither references nor citing documents, so their topological features are zero, and that the MeSH feature is nonzero on only 0.28% of edges. Where both kinds of signal exist, they agree: edges with higher coupling also have more similar abstracts and closer publication years. In a pilot, replacing the median by a mean or best-match aggregation makes the MeSH feature nonzero on nearly all article pairs, and near-identical texts point to possible duplicate records that identifier checks miss. The parsing, graph-construction, and feature code are public; the paper states which data are released and which are not. A companion paper uses the resulting weighted graph for clustering.

0.59

Multiplex coupling: Connecting the intellectual base and the topics of a research field

cs.DL stat.ME ·2026-10-15 ·Massimo Aria, Corrado Cuccurullo, Maria Spano
similarity 0.62
Abstract · 41 pages, 9 figures, 6 tables, plus 24 pages of Supplementary Material. Submitted to Quantitative Science Studies. Data and code: https://do…

Two documents can be close because they cite the same works or because they write about the same things, and the two kinds of proximity often disagree. Science mapping reads them in separate maps or merges them, as hybrid clustering does; either way the disagreement is lost. We introduce multiplex coupling, a framework that couples the same documents by their references and by their terms, keeps the two layers separate and analyses their relation. Similarities are standardised against a null model, and the agreement between the layers is measured on random pairs of documents. Consensus clustering partitions each layer into roots, groups sharing an intellectual base, and themes, groups sharing a topic. A test on the roots $\times$ themes table identifies branching roots and convergent themes, and pairs of roots are followed over time. Every structure is reported with its persistence across numbers of neighbours, which on synthetic data identifies most false branching. Applications to Web of Science, Scopus and OpenAlex collections show that the two layers carry largely complementary information, and reveal substantive and methodological bases that have grown into several topics. The framework is implemented in bibliometrix.

0.35

Tracing the Influence of Retrieved Literature on Scientific Ideation in LLMs

cs.CL ·2026-10-28 ·Zhipin Wang, Koel Dutta Chowdhury, Christoph Leiter, Steffen Eger
similarity 0.61
Abstract · Updated preprint. Under review

Large language models (LLMs) are increasingly used to generate research ideas from scientific literature. Existing systems combine literature retrieval with search, generation, and refinement, making it difficult to determine how the retrieval strategy itself shapes the ideas generated. We isolate this contribution by comparing five retrieval strategies and a No Literature baseline under a shared generation protocol, using 45,000 ideas across 15 machine learning topics and ten open-weight models. Relative to no literature basline, all five retrieval strategies reduce diversity among ideas generated from the same set of papers by $5.6\%$--$17.9\%$, while most increase diversity by $10.2\%$--$18.1\%$ when ideas from different sets are pooled. Effects on idea quality and novelty are modest on average but vary across topics. We further apply a JSD-based attribution method to trace the influence of individual input papers on each idea's motivation, core idea, approach, and validation. In $79.3\%$ of ideas, only one or two distinct papers receive the highest attribution scores across these four parts. These leading sources often recur across ideas generated from the same set of papers, and groups with more varied source attribution contain more diverse ideas ($r=0.70$). These findings motivate various sets of literature and the use of attribution to guide the choice of sources for subsequent ideas.

0.14

Do Network Mechanisms in Science Vary Across Research Topics? Contextual Relational Hyperevent Models for Large-Scale Bibliometric Data

cs.DL cs.SIstat.ME ·2026-11-19 ·Jürgen Lerner, Mark Wittek, Raphael Heiberger
similarity 0.61
Abstract

Collaboration and citation networks in science are shaped by well-documented mechanisms such as preferential attachment, gender homophily, and self-citation. Whether these mechanisms operate uniformly across research areas, however, is rarely tested, because statistical network models that estimate several mechanisms jointly are difficult to apply to large bibliometric data and to compare across subnetworks. We introduce contextual relational hyperevent models (C-RHEMs), which extend relational hyperevent models by allowing network effects to vary across research topics within a single model. C-RHEMs treat each publication as one event linking its authors to its references, accommodate reference lists spanning several topics, and scale to hundreds of thousands of publications on standard hardware. In a second step, meta-analysis and meta-regression quantify heterogeneity in topic-specific effects and relate it to topic characteristics. We illustrate the approach with about 590,000 oncology publications from the PubMed Knowledge Graph, assigned to 32 OpenAlex research topics. We find that preferential attachment, gender homophily, and self-citation vary substantially across topics, with between-topic standard deviations of 30 to 70 percent of the corresponding pooled effects. Topic age, size, and gender composition account for between a third and two-thirds of this variation: older topics exhibit stronger preferential attachment, self-citation, and gender homophily, and topics with a higher share of male authors exhibit stronger self-citation. The results caution against treating network mechanisms estimated on pooled data as uniform, and C-RHEMs offer a scalable way to study how they differ across research areas.

0.17