CS Paper Radar

The illusion of neutrality in metric-based research evaluation

Fengyuan Liu, Hazem Ibrahim, Yasir Zaki, Talal Rahwan

cs.DL ·submitted 2026-09-21 12:14 UTC · v1

Abstract

Venue prestige and citation counts are two widely used, albeit imperfect, signals of research quality. When the two signals conflict, evaluators must decide how much weight to assign each. Yet, it remains unknown how researchers across disciplines trade off these two signals when evaluating research outcome. To fill this gap, we surveyed 869 researchers using paired choices between hypothetical departmental hiring rules that assigned different weights to venue prestige and citation counts, asking which would produce better science. Among 795 respondents whose choices were largely internally consistent, choices placed nearly equal aggregate weight on the two signals. This apparent balance concealed substantial individual heterogeneity: nearly two in five respondents occupied the most venue-heavy or citation-heavy intervals. Moreover, respondents on average chose more citation-heavy rules than they believed their departments used in hiring. By revealing the subjective judgments that arise when research indicators conflict, our findings reinforce calls for greater caution when quantitative indicators are used to evaluate research.

Closest papers

Nearest neighbours by meaning across the whole index.

Where Do Highly Cited Scientists Stand? A Multi-Dimensional Analysis of Positional Distribution in the Top 2% Ranking

cs.DL ·2026-11-17 ·Yuehai Li, Zuguang Gu
similarity 0.63
Abstract

The Top 2% Scientists list is widely used. Yet current applications either only count the number of scientists in different units or criticize the list's validity as a measure of individual impact. We introduce positional enrichment, a scale-free framework that compares observed to expected counts across percentile bins on the list, revealing where a unit's scientists tend to sit along the list rather than how many it contributes. Applying positional enrichment to the 2026 single-year list, we show that countries, institutions, and research subfields each fall into a consistent pattern of three groups with Head-, Middle-, and Tail-enriched profiles. Subfield profiles are globally stable, whereas countries and institutions are heterogeneous across subfields. National and institutional differences are therefore compositional. Therefore, the reason of a country or institution sitting near the head of the list is unlikely its scientists performing better within the same subfields, while it holding more of the subfields that sit near the head. Academic age adds an independent dimension, with stronger enrichment toward the head after 25 years and exceptions in fast-moving subfields such as artificial intelligence. We show the ranking is not a flat ordering of individual influence but a subfield-organized stratification.

0.18

Quantifying research uniqueness with Unbalanced Optimal Transport: Case studies of ten leading universities in neuroscience

cs.DL cs.SI ·2026-10-13 ·Yu Ota, Takayuki Ogi, Yo Nakawake, Daichi Mochihashi, Amane Koizumi
similarity 0.58
Abstract · 22 pages, 6 figures, 2 tables. Supplementary material included as an ancillary file

Research evaluation has developed around quantity, citation impact and, more recently, the accumulation of research excellence, but these dimensions remain largely dependent on citations, which describe how research is received rather than what it contains. We propose uniqueness as a complementary dimension and measure it with the Research Uniqueness Index (RUI), which asks how far a university's research sits from what is typical in its field, using no citations but the content of the research itself. Each researcher is represented by an embedding of their abstracts, and an unbalanced version of Optimal Transport returns the share of that work which cannot be matched into a random sample of same-field researchers, the standard population; that unmatched share is the score. Because every researcher is scored, a university is a distribution rather than one number, and we treat that distribution as the indicator: it shows whether uniqueness is spread across an institution or concentrated in a few groups. Quantiles summarise it and, being quantiles, do not reward size. Since uniqueness is defined only relative to a field, we demonstrate the index within one, neuroscience (Scopus, 2021-2025; 9,861 reference researchers). All ten universities studied sit well above the standard (RUI_median 0.40-0.53 versus 0.27) and five specialised institutes higher still (0.57-0.73), the most unique work often confined to a narrow niche that the same scores name. RUI is uncorrelated with citation impact, and a controlled test confirms that it tracks displacement from the field rather than topical concentration; results are robust to an alternative embedding and replicate in computer science and economics. The index is not intended as a league table: the gap above the standard and the researcher-level niche structure are stable, whereas the order among adjacent universities is not.

0.29

Authority Bias in Conversational Search Engines for Academic Paper Recommendation

cs.AI ·2026-08-31 ·Uthman Jinadu, Parsa Ghazvinian, Anjila Budathoki, Benjamin M. Ampel, Rajshekhar Sunderraman +1 more
top venue similarity 0.57
Abstract · Accepted at EMNLP 2026 Main Conference

Large Language Models (LLMs) are increasingly used as conversational search engines for academic literature, yet whether they judge papers on content or on authority signals has not been tested causally. We investigate authority bias: systematic preference for papers based on author prestige, venue, and citations rather than content. Holding title and abstract constant, we vary authority metadata across three counterfactual conditions (original, flipped, boosted) over eight LLMs (five open-weight and three frontier closed-weight) in an in-context, single-turn, top-1 recommendation setting. Our experiments show that authority bias is substantial and directional, varies markedly across models, and is only partially addressable through prompt-level debiasing. We further document a say-do gap: debiasing instructions suppress authority mentions far faster than authority-driven flips, so surface auditing systematically underestimates behavioral bias.

0.47

Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation

cs.DL cs.AI ·2026-09-11 ·Liang Zhao, Yong Wang, Jiangzhe Chen
top venuebenchmark similarity 0.57
Abstract · Accepted at Emnlp 2026

LLM-as-a-judge protocols are commonly debiased by instructing judges to ignore presentation cues such as citation formatting, source labels, and evidence-display style. We show that this intervention can suppress bias while damaging the resolution of the measurement instrument. We introduce TraceJudgeBench, a diagnostic benchmark for auditing citation-like artifacts in RAG and agent-workflow evaluation, covering content-equivalent pairs, citation ablations, correctness conflicts, human-validated soft and moderate quality gaps, prompt-strength ladders, decoupled judging, and a controlled workflow-ranking probe. Across GPT-5.5, Claude Sonnet 4.6, and DeepSeek V4-Flash, stronger anti-citation prompts reduce worse-cited wins from up to 50.5% to 0%; yet some operating points already convert validated moderate-gap decisions into Tie before the strict stress-test endpoint, while correctness-conflict accuracy remains at or above 93.0%. A second, 50-pair FinQA moderate-gap construction reproduces the qualitative frontier, and open-weight Qwen2.5-14B-Instruct-AWQ and Gemma-3-12B-IT runs reproduce the central HotpotQA frontier. TRACE-style decoupling recovers 96.5-100.0% better-plain resolution across the reported settings. Human validation separates three meanings of Tie: correct equivalence Tie, calibrated soft-boundary Tie, and resolution-destroying Tie on validated quality gaps. We frame debiasing as a measurement intervention whose bias suppression, resolution retention, Tie cost, and protocol cost must be reported jointly. The supplementary artifact contains benchmark splits, prompts, raw judge outputs, validation summaries, and analysis.

0.37

Has Scientific Talent Shifted from Depth to Breadth?Evidence across Papers, Knowledge Inputs, Careers, and Teams

cs.DL cs.AIcs.HC ·2026-09-13 ·Xiaoshn Nee, Haobo Zhong, Xiaomin Ni
similarity 0.57
Abstract

Generative artificial intelligence raises a central question for scientific training and organization. Is research shifting from deep specialization toward broad individual knowledge? We examine this proposition across papers, cited knowledge, contributor histories, and teams using 47,959 articles from six fields over 2010-2025, 51,736 resolved cited works, and chronologically reconstructed prior publication histories for 1,754 randomly selected index contributors. From 2010 to 2022, team size increased by an estimated 37.3% (95% confidence interval [34.4%, 40.3%]), while paper topic breadth declined by 0.0144 on a 0-1 hierarchical distance scale. Cited knowledge was stable to modestly broader, revealing a divergence between focused outputs and the reach of knowledge inputs. Established contributors' prior breadth increased by 0.0190 [-0.0078, 0.0459] by 2019-2022, within a +/-0.05 equivalence bound assessed in sensitivity analysis. In mature citation windows, one standard deviation of focal depth was associated with 8.2% higher 1 + FWCI [1.9%, 14.9%]; average breadth and interaction associations were smaller under the specified equivalence bounds. Post-2022 deviations from earlier trends were not systematic, and recent changes did not vary clearly with baseline AI intensity across 83 subfields. The findings support a differentiated structure of scientific expertise in which focused individual accumulation coexists with expanding collaboration and sustained access to diverse knowledge inputs.

0.27

CITECHOICE: A Causal Audit of How Document Presentation Redistributes Citation Credit in Agentic Search

cs.CL ·2026-09-14 ·Sriram Selvam, Anneswa Ghosh
similarity 0.57
Abstract

When several retrieved sources support the same claim, an answer engine cites some but not others. We call this decision citation allocation and introduce CITECHOICE, a causal audit of authentic multi-turn agentic search. From 129 everyday-query transcripts, CITECHOICE selects 113 same-call document pairs with independently verified support for the same pre-specified fact, without observing ranks or answer outcomes; blinded human review confirms 103. It runs a hash-verified 2-by-2 replay crossing pair order with jointly generated, fidelity-checked structured and prose renderings of one target while the rest of the transcript remains fixed. Three results emerge. First, and most importantly, structured rendering concentrates citation credit rather than clearly increasing source admission. It raises target citation count by +0.50 citations per answer (95 percent CI [+0.20, +0.84]; Holm-adjusted p=.033), without increasing total citations or reducing competitor credit. The pre-specified incidence effect (whether the target is cited at all) is +4.5 percentage points and inconclusive (95 percent CI [-1.4, +10.4]; p=.168). Second, observational position differences exceed controlled reordering effects: the citation-rate gap between rank 1 and rank 5 is 42.3 percentage points, compared with +7.9 percentage points in the main replay and 0.0 percentage points held out. Third, citation evaluation has a measurable noise floor. Although the aggregate count effect repeats under fresh decoding of 30 frozen families, 15 percent of binary decisions change and decoding accounts for an estimated 45 percent of single-generation family-effect variance. Together, these findings isolate what survives control: presentation can causally redistribute visible citation credit within frozen transcripts. They do not establish reliable source admission, a pure formatting mechanism, or a general rank advantage.

0.18

Eliciting Research Taste in LLMs through Future Research Direction Choice

cs.LG cs.AIcs.CL ·2026-11-15 ·Dhruv Trehan, Paras Chopra
code/data similarity 0.57
Abstract · Accepted at the LM4Sci Workshop at COLM 2026. 23 pages, 7 figures, 18 tables. Code and data: https://github.com/Lossfunk/research-taste-eval…

Research taste is increasingly discussed in the context of AI scientists, yet remains little studied or measured in LLMs. This judgment matters especially in open-ended research, where a direction must be chosen before a verifiable objective is available. We operationalize research taste through one task of the scientific workflow: choosing which future research direction to pursue given a research goal and current results. Rather than predicting downstream scientometric metrics or treating author choices as objectively correct, we evaluate models against human-authored future-work directions and report author alignment, stable divergence, and unstable choice as behavioral measures of research-direction preference. We construct an order-controlled evaluation of 80 pairwise comparisons from 19 ML papers published between December 2025 and February 2026 across four subdomains. Each query presents a research goal, hypotheses tested with key results, and two candidate future directions: one author-recommended direction and one plausible distractor sampled from a similar paper. Across eight LLMs, author alignment ranges from 45.0% to 67.5%, while stable divergence ranges from 15.0% to 36.2%. These divergences largely persist under similarity controls, prompt ablations, and an author-preference framing check, and inter-model agreement suggests that some non-author preferences are shared across models. We characterize what models diverge toward using transition-category labels and selected reasoning examples, finding recurring movement toward scaling/generalization and trade-offs between methodological and infrastructure/data extensions. We view this as an early descriptive evaluation for eliciting research taste profiles in LLMs, and as a step toward understanding and steering their research preferences as they take on more autonomous roles in novel open-ended scientific discovery.

0.29

Funding the runners-up beats a golden ticket

cs.CY cs.DL ·2026-09-17 ·Haining Wang
benchmarkcode/data similarity 0.57
Abstract · 29 pages, 2 main figures, 3 supplementary figures; includes Supplementary Appendix. Data and code: https://github.com/Wang-Haining/wonka

A golden ticket allows one reviewer to advance a proposal that the rest of a panel would reject. We assess its score-based rationale by comparing minority-support ranking with other rules for allocating the same number of places. We do so on 10,625 rejected submissions to a computer-science conference between 2017 and 2024, using later citation percentiles as the outcome. Minority-support ranking measured the gap between the two highest scores after adjustment for the panel mean and reviewer count. At a budget of 10% of the year's accepted count, panel-mean ranking selected submissions with a mean citation percentile 7.75 points higher than minority-support ranking (95% bootstrap interval 4.35 to 11.09). The difference was larger in the other sample periods and citation windows examined. At that budget, minority-support ranking was equivalent to a lottery within a margin of plus or minus 5 percentile points. Two further analyses, adjusted highest-score ranking and a matched comparison, found no clear citation advantage from minority support once the panel mean was accounted for. Ranking on score variance also outperformed minority-support ranking in a paired comparison. Estimates cover the 69.8% of eligible rejections whose later citations could be verified. These comparisons identify panel-mean ranking as a benchmark for evaluating whether golden tickets select work that would otherwise be overlooked.

0.41