CS Paper Radar

On some unresolved issues in aggregate level comparative research evaluation

Paul Donner

cs.DL ·submitted 2026-10-16 11:23 UTC · v1
unusual

Abstract

This short conceptual paper recalls to the attention of the scientometrics community some long-standing open methodological problems and proposes new directions that might be taken in renewed engagements with them. These problems are the differential dependency of the global reference score on the included units' observed scores and the dilution of impact measurement accuracy due to collaboration, which causes a global convergence in country citation impact due to international collaborative research. Advantages and disadvantages of existing and new solutions are discussed.

Closest papers

Nearest neighbours by meaning across the whole index.

Growth of global science: reflections on measurability and scientometric insights

cs.IR ·2026-10-31 ·Robert Tijssen, Erik van Wijk, Alfredo Yegros
similarity 0.61
Abstract · 23 pages, 5 figures, 1 table

Global science has become much larger and more interconnected in recent years. Such a complex and adaptive social system defies any attempt to systematically and comprehensively measure or monitor its growth and evolution. Yet, as it continues to expand and enhance its public value, effective evidence-based assessment has become increasingly critical for understanding its dynamics and managing the system. This paper offers a framework for addressing system-level measurability challenges. Inspired by Derek de Solla Prices pioneering scientometric work in the 1960s, this paper addresses analytical issues related to current measurements of the systems expansion since the turn of the 21st century. Our empirical analysis of global science dynamics presents new macro-level data on several observable megatrends in research publication output worldwide. Acknowledging the limited analytical scope of these trend data to understand the dynamics of global science, we outline currently available options to improve system-level measurability under conditions of ambiguity and uncertainty. Looking forward, we discuss likely future directions to establish integrated, multi-stakeholder measurement infrastructures.

0.30

Where Do Highly Cited Scientists Stand? A Multi-Dimensional Analysis of Positional Distribution in the Top 2% Ranking

cs.DL ·2026-11-17 ·Yuehai Li, Zuguang Gu
similarity 0.57
Abstract

The Top 2% Scientists list is widely used. Yet current applications either only count the number of scientists in different units or criticize the list's validity as a measure of individual impact. We introduce positional enrichment, a scale-free framework that compares observed to expected counts across percentile bins on the list, revealing where a unit's scientists tend to sit along the list rather than how many it contributes. Applying positional enrichment to the 2026 single-year list, we show that countries, institutions, and research subfields each fall into a consistent pattern of three groups with Head-, Middle-, and Tail-enriched profiles. Subfield profiles are globally stable, whereas countries and institutions are heterogeneous across subfields. National and institutional differences are therefore compositional. Therefore, the reason of a country or institution sitting near the head of the list is unlikely its scientists performing better within the same subfields, while it holding more of the subfields that sit near the head. Academic age adds an independent dimension, with stronger enrichment toward the head after 25 years and exceptions in fast-moving subfields such as artificial intelligence. We show the ranking is not a flat ordering of individual influence but a subfield-organized stratification.

0.22

Guidance List for repOrting Bibliometric AnaLyses (GLOBAL): A Reporting Guideline Informed by a Two-Round Modified Delphi Study

cs.DL ·2026-10-28 ·Jeremy Y. Ng, Henry Liu, Mehvish Masood, Niveen Syed, Dalia Zubashev +22 more
similarity 0.52
Abstract

Background: Despite the growth of bibliometric analyses, few studies offer guidance on how to report them, resulting in a lack of research transparency and completeness. To address this gap, we developed the Guidance List for the repOrting of Bibliometric AnaLyses (GLOBAL). Methods: An initial list of items for the GLOBAL was generated through a scoping review and refined through a modified two-round Delphi, as recommended by the EQUATOR Network. The guideline was finalized through a pilot test, where participants assessed their manuscripts using the checklist and Elaboration and Explanation (E&E) document, which outlines item expectations and provides examples. Results: The scoping review resulted in 32 items. In Round 1, 24 items reached consensus and thematic content analysis resulted in one additional item. This item, along with the eight items that did not reach consensus, were discussed in detail in Round 2. During the in-person meeting, one item was split into two, totalling ten voting items. Eight out of ten items reached consensus, five for inclusion and three for exclusion. Conclusions: The finalized 28-item GLOBAL (and its accompanying E&E document) provides users with a framework to report bibliometric analyses, aimed at promoting high-quality reporting of bibliometric analyses in scholarly literature.

0.25

The illusion of neutrality in metric-based research evaluation

cs.DL ·2026-09-21 ·Fengyuan Liu, Hazem Ibrahim, Yasir Zaki, Talal Rahwan
similarity 0.51
Abstract

Venue prestige and citation counts are two widely used, albeit imperfect, signals of research quality. When the two signals conflict, evaluators must decide how much weight to assign each. Yet, it remains unknown how researchers across disciplines trade off these two signals when evaluating research outcome. To fill this gap, we surveyed 869 researchers using paired choices between hypothetical departmental hiring rules that assigned different weights to venue prestige and citation counts, asking which would produce better science. Among 795 respondents whose choices were largely internally consistent, choices placed nearly equal aggregate weight on the two signals. This apparent balance concealed substantial individual heterogeneity: nearly two in five respondents occupied the most venue-heavy or citation-heavy intervals. Moreover, respondents on average chose more citation-heavy rules than they believed their departments used in hiring. By revealing the subjective judgments that arise when research indicators conflict, our findings reinforce calls for greater caution when quantitative indicators are used to evaluate research.

0.27

The conservative turn in science: The changing character of knowledge recombination

cs.SI cs.DL ·2026-09-08 ·Cheng Zhang, Xinkun Liang, Yi Bu, Hongkan Chen
similarity 0.51
Abstract

This study examines how the dominant mode of cross-disciplinary knowledge combination has changed over time. It addresses a limitation of existing studies that document declining disruptiveness at the level of individual works but do not reveal whether the mechanisms of cross-disciplinary knowledge flow have themselves shifted. Drawing on 56 million publications and 816 million citations from OpenAlex (1960-2025), the study classifies cross-disciplinary citations into five types based on the set relationship between the subfield portfolios of citing and cited publications and introduces a semantic-distance measure derived from SPECTER-v2 embeddings of publication abstracts. It subsequently demonstrates that cross-disciplinary citations have shifted markedly away from configurations with no shared disciplinary ground toward types built on partially overlapping knowledge bases, a reorientation further reflected in a steady decline in average semantic distance between citing and cited fields. This shift co-occurs with a diffusion of brokerage positions across the network and a greater reliance on older literature, consistent with a system that has become broader and more decentralized in its interdisciplinary reach. Finally, this study discusses the implications of these findings for science policy and for the design of cross-disciplinary funding mechanisms.

0.08

Large scientific teams are more, not less, disruptive

cs.DL physics.soc-ph ·2026-10-05 ·Junming Huang, Yu Xie
similarity 0.51
Abstract · 34 pages, 14 figures

Modern science has moved decisively toward larger and more collaborative research teams. Yet an influential 2019 study reported that small teams are more disruptive than large teams, a conclusion difficult to reconcile with both the well-known trend toward increasing collaboration and the disproportionately large teams behind widely recognized disruptive research, such as work that has been awarded Nobel Prizes. Here we show that the reported disruptive advantage of small teams is largely an artifact of measurement rather than a fact about teams. Analyzing about 60 million scientific publications, we find that larger teams cite more references and that the standard disruption index declines mechanically as reference lists lengthen. Equalizing reference-count distributions across team sizes reverses the negative gradient within the index's own framework. Evaluating each paper against one reference at a time yields a reference-robust measure that distinguishes recognized breakthroughs from ordinary papers more sharply and more consistently than the standard disruption index across six independently curated benchmarks, including the Nobel Prize. Under this validated measurement, scientific disruption increases with team size across four scientific domains and in every decade from the 1970s to the 2010s. For scientific publications, we conclude that larger teams are more, not less, disruptive.

0.43

Multiplex coupling: Connecting the intellectual base and the topics of a research field

cs.DL stat.ME ·2026-10-15 ·Massimo Aria, Corrado Cuccurullo, Maria Spano
similarity 0.50
Abstract · 41 pages, 9 figures, 6 tables, plus 24 pages of Supplementary Material. Submitted to Quantitative Science Studies. Data and code: https://do…

Two documents can be close because they cite the same works or because they write about the same things, and the two kinds of proximity often disagree. Science mapping reads them in separate maps or merges them, as hybrid clustering does; either way the disagreement is lost. We introduce multiplex coupling, a framework that couples the same documents by their references and by their terms, keeps the two layers separate and analyses their relation. Similarities are standardised against a null model, and the agreement between the layers is measured on random pairs of documents. Consensus clustering partitions each layer into roots, groups sharing an intellectual base, and themes, groups sharing a topic. A test on the roots $\times$ themes table identifies branching roots and convergent themes, and pairs of roots are followed over time. Every structure is reported with its persistence across numbers of neighbours, which on synthetic data identifies most false branching. Applications to Web of Science, Scopus and OpenAlex collections show that the two layers carry largely complementary information, and reveal substantive and methodological bases that have grown into several topics. The framework is implemented in bibliometrix.

0.35

Assessing the Reliability of Longitudinal Bibliometric Rankings under Incomplete Citation Histories

cs.DL ·2026-11-12 ·João Paulo Papa
similarity 0.50
Abstract · 20 pages, 2 figures. Software available at https://doi.org/10.5281/zenodo.24170285

Longitudinal bibliometric analyses rely on annual citation histories, yet missing annual observations are frequently difficult to distinguish from true zero-citation years. We present a reproducible case-study framework for auditing citation-history gaps, tracing retrieval provenance, and quantifying the sensitivity of longitudinal indicators and rankings to explicit missing-data assumptions. The case study comprises 595 consolidated publications derived from 616 Google Scholar article records, and examines the completed calendar years 2024-2025. Three scenarios are compared: strictly observed annual counts (A), hypothetical zeros for internal gaps only (B), and hypothetical zeros for all missing years within a valid publication interval (C). The number of publications with computable growth and acceleration indicators increases from 196 (A) to 257 (B) and 490 (C); the corresponding counts for a revival indicator are 131, 152, and 177. Although the ordering of publications eligible under both compared scenarios is unchanged (Spearman and Kendall correlations equal to 1.0), complete-ranking positions and Top-K membership may change markedly. In particular, acceleration exhibits a Jaccard overlap of 0.563 at K=50 between A and C, while revival exhibits 0.667 at K=10. These findings demonstrate that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes. The zero assignments are hypothetical sensitivity assumptions, not evidence of provider semantics. The results are descriptive and based on a single researcher profile. A reusable Google Scholar Tracker software workflow supports the audit and sensitivity analysis; version 1.8.5 is publicly available on GitHub and archived on Zenodo (DOI: 10.5281/zenodo.24170285).

0.49