CS Paper Radar

Testing Our Foundations: Citation Trends, Errors, and Emerging Hallucinations in the Computing Education Literature

Paul Denny, Gweneth Barbre, Musa Blake, Yan Cathy Hua, Juho Leinonen, Andrew Luxton-Reilly, James Prather, Brent N. Reeves

cs.DL cs.CY ·submitted 2026-09-15 03:18 UTC · v1
publishedsurvey

Abstract

Accurate references are foundational to scholarly work, enabling verification, attribution, and systematic review. However, the rapid adoption of large language models has introduced a serious integrity concern: plausible-looking but fabricated citations. Although hallucinated references are widely discussed, their visibility within specific research communities remains unclear. We address this gap by examining reference integrity at key computing education venues using ACM Digital Library data. We analyze referencing trends across 24,751 computing education papers and compare them with the broader ACM corpus of more than 723,000 papers and 15 million references. We then examine reference lists from these venues, classify common bibliographic errors, and manually identify LLM-generated hallucinations containing verifiably false information, including impossible page ranges, invented titles, and misattributed authors. In 2025, hallucinated references appeared across five SIGCSE-sponsored or in-cooperation venues. At the Technical Symposium alone, verified hallucinated references increased from 3 in 2025 to 17 in 2026, appearing in 2.3\% of 2026 proceedings papers. Although still relatively rare for now, this growth poses an integrity risk our community should not ignore.

Comment
Accepted to SIGCSE Technical Symposium 2027

Closest papers

Nearest neighbours by meaning across the whole index.

Dreaming Up Authors: Factual Prevalence and LLM Hallucinations in Academic Authorship Attribution

cs.CL ·2026-11-17 ·Sijia Zhong, Tamir Yirga, Yijia Ma, Thai Le, Yifan Hu
top venuecode/data similarity 0.73
Abstract · Published in EMNLP 2026 (Main Conference). 37 pages. Code and data: https://github.com/quinnzhong1/dreaming-up-authors. Anthology: https://a…

Large Language Models (LLMs) are widely used for question answering but are prone to hallucinations, yet the underlying causes, particularly the role of factual prevalence in training data, remain underexplored. In this work, we study this through academic authorship attribution, hypothesizing that hallucinations are more likely for less prevalent papers. Using citation count as a proxy for factual prevalence, we curate a dataset of 1.92 million papers across eight scientific disciplines with stratified samples by citation level. Experiments on three state-of-the-art LLMs confirm our hypothesis, revealing a robust inverse relationship between hallucination rate and citation count across all models and disciplines, with hallucination rates exceeding 98% for rarely cited papers and falling below 37% for highly cited ones. Our in-depth analysis further shows that these failures primarily reflect retrieval limitations rather than knowledge absence, and hidden-state probing of open-source models shows that factual prevalence is encoded in pre-generation representations, providing convergent mechanistic evidence for the prevalence-hallucination relationship. The code and data are available at https://github.com/quinnzhong1/dreaming-up-authors.

0.57

Zero-Cost Detection of Fabricated Citations: A Staged Heuristic Approach for Computing Dissertation Assessment

cs.DL cs.AI ·2026-10-05 ·Hazrat Ali, Mohamed Elawady
position similarity 0.68
Abstract · 2 pages abstract, submitted to ACM SIGCSE-virtual conference

Generative AI's growing role in academic writing has driven a rise in fabricated references across dissertations and student reports. We present the Fabrication Severity Stack (FSS), a three-tiered framework classifying citation fabrications by detection complexity - from non-existent references to subtle content misattribution. Building on the FSS, we propose a staged heuristic verification workflow, executable through mainstream LLM interfaces. We argue that this framework offers computer science educators/examiners a practical, zero-cost basis for embedding reference verification into the assessment of dissertation and students' reports. In a pilot study on 100 citations (20% seeded with Level 1 and Level 2 fabrications), the proposed heuristic workflow enabled mainstream LLM interfaces to correctly flag 9-10 of 10 non-existent references and partially detect metadata corruption. These preliminary results suggest that freely accessible, general-purpose LLMs can achieve high recall on citation verification without reliance on proprietary detection tools.

0.06

Biomedical Reference Generation Remains Unreliable across 26 Large Language Models

cs.CL ·2026-09-14 ·Maxim Topaz, Zhihong Zhang, Nir Roguin, Pallavi Gupta, Zichao Li +1 more
code/data similarity 0.62
Abstract

Background. Large language models are increasingly used to help write biomedical text but may fabricate references to nonexistent work. How often large language models do so is not well characterized. Methods. We prompted 26 language models from eight developers (2023 to 2026) to supply a missing reference for each of 69 biomedical passages across ten domains. References were classified as verifiable (real paper with a resolving identifier), partial matches (real paper without a resolving identifier), fabricated (no matching indexed paper), or declined (the model refused to supply a reference). A reference was considered correct in every evaluated bibliographic field only when it was verifiable and its journal, year, and listed authors matched those of the cited paper. Results. Fabrication ranged from 10.2% (Claude Opus 4.8, which declined 52.1% of prompts) to 98.4% (Ministral 3B, which produced no verifiable reference). Claude Opus 4.6 and Claude Sonnet 4.5 produced similar proportions of verifiable references (77.6% and 76.6%) but named authors correctly in 78.7% and 28.7% of author-evaluable verifiable references, respectively, and were correct in every evaluated field in 54.6% and 19.9% of responses. GPT-5.5 was correct in every field in 48.1%. Across all models, 55.4% of responses were fabricated and 14.9% were correct in every field. Among the five tested models first released in 2026, the corresponding proportions were 35.3% and 31.8%, respectively. Conclusions. Fabrication remained common, and no model was correct in every evaluated bibliographic field in more than 54.6% of responses. Models that identify real papers may still misstate their metadata, so references produced with model assistance require verification before use.

0.39

HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviews

cs.AI cs.CL ·2026-09-03 ·Tzu-Ling Lin, Dong-Ting Yao, Teng-Fang Hsiao, Wei-Chih Chen, Hong-Han Shuai
top venuebenchmarkcode/datahot direction similarity 0.61
Abstract · Accepted to EMNLP Findings 2026

The growing scale of academic peer review has motivated the use of Large Language Models (LLMs) as review assistants, yet LLMs can generate fluent but unsupported claims that undermine review reliability. Existing hallucination benchmarks are not designed for peer review, where verification requires grounding claims in long, technical papers. We introduce HalluPeer, a benchmark for detecting hallucinations in scientific peer reviews, providing aligned triples of paper content, human-written reviews, and hallucination-injected reviews, annotated for detection, classification, and localization. Our pipeline induces a peer-review-specific hallucination taxonomy, identifies review contexts, and injects hallucinations with automated filtering. Experiments on 12K papers and 38K reviews show that existing detectors struggle to separate hallucinations from legitimate critique, while evaluation on authentic reviews demonstrates that HalluPeer-defined hallucination patterns occur in real peer reviews, highlighting the critical need for source-aware verification. Our project page can be found in https://github.com/Lin-TzuLing/HalluPeer.git

0.56

Citing Less Critically: LLMs Reshape the Rhetoric and Reach of Scientific Citation

cs.DL cs.CLcs.CYcs.SI ·2026-09-01 ·Yixuan Liu, Lin Chen, Zhuoqi Liu, Jianglin Lu, Dakota Murray
top venue similarity 0.58
Abstract · Accepted at the EMNLP 2026 main conference

Scientific citations carry rhetorical intent. Scholars may cite prior work positively (supporting), negatively (contrasting), or neutrally (mentioning). As large language models (LLMs) increasingly assist scientific writing, whether they reproduce citations with the same rhetorical intent as humans remains unclear. We introduce a masked-citation task to compare human and LLM-generated citation behavior. For each citation context, an LLM generates a replacement citation sentence, producing a counterfactual corpus directly comparable to human citation. We analyze what, whom, and how models cite, using an LLM-as-a-judge to classify citation intent and a 20-million-edge coauthorship network to measure social distance between cited authors. Across six popular LLMs and 1,746 top NLP conference papers (63k+ contexts, 132k+ citations), three patterns emerge: (1) Compared with human citation, LLMs cite significantly less critically; (2) LLMs over-cite popular and older papers, a tendency amplified for contrasting citations where human writing more often draws on recent, niche work; (3) Whereas humans often cite within their close social network, especially for supporting citations, LLMs tend to draw on more socially distant authors. Together, these differences are double-edged: LLM citation reaches beyond a scholar's close collaborators while being less critical and amplifying visibility bias, reshaping the rhetoric and reach of scientific citation.

0.34

Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research

cs.CL ·2026-08-25 ·Eran Hirsch, David Wan, Han Wang, Elias Stengel-Eskin, Mohit Bansal +1 more
top venuecode/data similarity 0.58
Abstract · Accepted to EMNLP 2026 (Main Conference). Code: https://github.com/eranhirs/who-is-the-agent-to-blame

Deep research (DR) systems produce long-form cited reports by orchestrating multiple agents that search and synthesize information from the web. Citations are the primary mechanism for evaluating the faithfulness of these reports, yet current DR systems exhibit poor citation recall. Moreover, improving citation recall is challenging because DR systems are complex multi-agent architectures where information passes through agents like a telephone game, and both content and citations can get corrupted along the way. We propose an evaluation method that pinpoints which agent introduced each error by locally testing agent invocations for faithfulness and verifiability relative to their own inputs. Furthermore, we propose a four-type taxonomy to categorize the discovered errors: hallucination, uncited input reliance, uncited output, or insufficient citations. Applying our method to three top-ranked open-source DR systems, we obtain actionable diagnostics. Almost every agent makes a lot of mistakes with the exception being those that summarize a single document. We find that the dominant error type varies systematically across agents, where the orchestrator mistakes are mostly citation-related. We find that 84.7% of final-report errors in AI-Q originate at the orchestrator, roughly 31% of them hallucinations and the rest citation mistakes. Guided by these insights, we demonstrate that two simple interventions raise citation recall by 5% without degrading output quality.

0.64

ATIBA: Grounded Integrity and Quality Checking for Research Papers

cs.SE ·2026-09-03 ·Veli Karakaya, Semih Çağlar, Yusuf Yiğit Korkmaz, Eray Tüzün
published similarity 0.58
Abstract · Accepted at The First International Workshop on Automated Techniques for Integrity and Quality in Software-Engineering Research (ATIQSER 202…

Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in practice it is done inconsistently or skipped. We present ATIBA, a tool that runs five grounded integrity and quality checks on a manuscript: a reference-integrity check that verifies each citation against bibliographic sources and flags retracted or unfindable references; a venue/track compliance check that derives submission criteria directly from a venue's own call-for-papers page and evaluates the manuscript against them, each verdict anchored to a verbatim quote from that page; an empirical-standards compliance check against the ACM SIGSOFT Empirical Standards, with a hallucination defence that discards any evidence quote it cannot locate verbatim in the manuscript; a multi-mode AI review (venue-specific, formal, and page-anchored annotation) powered by GPT-5.4 through Azure OpenAI; and a citation-suggestion feature that proposes candidate references for a manuscript and verifies each against bibliographic sources before it is shown to the user. All five checks are designed around the same principle: an LLM is only trusted to judge, never to invent the evidence it judges against. We evaluated ATIBA through a moderated user study with 13 non-author participants. Agreement across the six survey items ranged from 69% to 92%, with a mean of 85%, providing initial evidence of positive perceived usefulness across the evaluated workflows. These findings establish perceived usefulness; objective accuracy remains to be measured.

0.40

Is this Citation on Point?

cs.DL cs.CL ·2026-08-12 ·Apurv Verma
published similarity 0.58
Abstract · Accepted to the 1st Workshop on AI for Law at ICML 2026

In 2023, a New York judge sanctioned two attorneys in \emph{Mata v. Avianca} for filing a brief containing hallucinated citations generated by ChatGPT. These failures can be largely caught by database lookups. The harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered, a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora. We either replace the cited case or change only the pinpoint page within the same case. Our evaluation of fourteen model configurations on the resulting examples shows that models catch $93$--$100\%$ of wrong-case corruptions, but they only catch $37$--$61\%$ of wrong-pinpoint corruptions on court opinions and $52$--$83\%$ on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Larger-scale models and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses $40\%$ of pinpoint mismatches on court opinions and $18\%$ on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. These results show that citation misrepresentations remain difficult to detect automatically. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.

0.20