CS Paper Radar

Copies Are Not Confirmations: How News Duplication Fools LLM Judges and Continual Learners

Abdelkader Metales, Miriam Grabenheinrich, Aicha Metales

cs.LG cs.CL ·submitted 2026-11-06 09:22 UTC · v2
theory

Abstract

A language model that keeps learning from the news sees the same claim many times, but many reports are copies of one source: wire stories, content shared within a media group, syndication and press releases. We ask whether LLM judges and continual learners treat such copies as independent confirmations. In 200,000 English articles from CC-News, a near-duplicate detector identifies copy links to another outlet for 18.9% of the articles, a detector-based lower bound. In a controlled experiment with claims about invented people, three identical reprints instead of one report raised the lies accepted by Llama 3.2 from 0.06 to 0.71 and by phi4 from 0.21 to 0.90; Gemini 2.5 Flash already believed a single report, and Gemini 2.5 Pro followed provenance information most consistently. In a simulated continual-learning stream, showing the judge one report per detected copy chain cut the copied lies accepted by 0.10-0.24 across four judges, at a mean cost of at most 4.2 points of accepted truths. Text similarity links none of the copies reworded by three LLMs. An entity-novelty test, the share of a report's named entities absent from an earlier look-alike, separates these rewrites from 100 author-labelled independent reports of the same event (AUC 0.98-0.99), statistically indistinguishable from TF-IDF and the best tested embedding model, but recognises only 17% of natural same-source derivatives. With both detectors run on 15,084 real articles carrying inserted claims, one report per chain cut the lies accepted by phi4 from 0.57-0.67 to 0.09 (copies), 0.14 (rewrites) and 0.34 (rewrites with invented details), and by Llama 3.2 from 0.56-0.62 to 0.07, 0.09 and 0.25, at a cost of 2.4 and 10 points of accepted truths. Counting reports or chains without a judge is unreliable, and colluding sources that do not copy each other remain an open problem.

Comment
v2: learning stream rerun on 15,084 real CC-News articles with the real detectors (new Section 7); threshold table added; abstract shortened and claims narrowed. 22 pages

Closest papers

Nearest neighbours by meaning across the whole index.

What Do LLM Judges Actually Verify? Matched Controls and Record Gaps for Document Currency

cs.AI cs.CL ·2026-10-31 ·Unnati Shah, Samruddhi Kakade, Aum Patel
code/data similarity 0.62
Abstract · 37 pages, 6 figures, including a 13-page supplementary case study. Code, data and all judge replies: https://github.com/unnatibshah/what-do-…

LLM judges often check, against an amendment record, whether the documents an answer cites are still in force. Such records are rarely complete. From a table of replacements, our test generates matched answers citing a replaced document, its successor, the replaced document before its replacement, and an older document that a newer one updated without replacing; it removes links from the record and scores each finding by its stated reason. On 48 replacements from the IETF RFC index, judges given a correct record caught nearly every replaced citation, as a lookup would. On the 24 faults whose link we removed, every judge caught more from the texts alone than with the record: Gemini 3.1 Pro 23/24 against 1/24, GPT-5.5 23/24 against 9/24, Claude Opus 4.5 17/24 against 9/24 and Gemma 4 26B 13/24 against 0/24. The drop recurred on a second draw of removed links. Most misses carried no finding; in an exploratory run with a conflict code, only Gemini used it often. Rewording the gap as "no replacement recorded" did not help; presenting the record as an unverified note, or warning that it was incomplete, restored most lost catches for Gemini, GPT-5.5 and Claude (significant after correction for the first two) but at most 5 of 13 for Gemma; where catches returned, so did false alarms on updates. With both texts, judges mostly told replacements from updates; every update flagged had a near-identical title. A descriptive case study on Indian rooftop-solar rules is appended; we release the harness.

0.36

Copies or Sources? Measuring How LLM Aggregators Count Restated Evidence in Multi-Agent Systems

cs.AI ·2026-10-05 ·Jianxin Gao, Runze Li, Tianyi Yu, Liangwei Ren, Bohan Chen +1 more
similarity 0.61
Abstract

Multi-agent systems built on large language models (LLMs) restate observations as a matter of course: relays forward them, shared boards repeat them and discussion rounds echo them. An aggregator that pools such messages should count sources, not statements. We convert a reported probability into units of independent readings, which assigns every restatement a copy weight, 0 for an aggregator that counts sources and 1 for one that counts every statement, and yields the implied decision under any cost structure. Three testbeds hold the evidence fixed and vary how it is restated: message logs with an exact Bayesian oracle, web documents with appended copies, and logs written by LLM agent teams under four communication protocols. Across four models from three providers, a forwarded copy counts for 0.06 to 0.42 of a new reading, mostly because some replies count every statement. On 5% to 40% of logs that state one reading three times, the reported belief implies an early commitment that the oracle never makes. The models that count copies least and most on controlled logs do so on web copies and agent-written logs as well. A one-paragraph declaration of what a copy contributes brings the copy weight on controlled logs to 0.08 or less. A rule that has agents refer to readings instead of restating them cuts belief-implied early commitment from 11.2% to 1.1% and preserves genuine corroboration.

0.37

Visible but Not Yet Curatable: Characterizing the Curatability of Compact and Derived Open LLM Artifacts

cs.HC cs.DL ·2026-08-28 ·Yiyi Lu, Yilai Qian, Yucheng Jin
publishedcode/data similarity 0.60
Abstract · 11 pages, 5 figures. Accepted at the ACM/IEEE Joint Conference on Digital Libraries (JCDL 2026). Yiyi Lu and Yilai Qian contributed equally;…

Open Large Language Model (LLM) research increasingly produces compact and derived artifacts, such as adapters, quantized checkpoints, merged models, and distilled variants, that are distributed across papers, model hubs, model cards, code repositories, and release statements. Although these artifacts are publicly visible, digital libraries often lack sufficient evidence to identify, preserve, and cite them as coherent scholarly objects. We introduce a framework that conceptualizes curatability as a record-level property of distributed scholarly records and operationalizes it through four evidence dimensions: artifact identity, scholarly linkage, upstream evidence, and release assets. Guided by this framework, we conduct the first collection-scale characterization of open LLM curatability using a May 2026 snapshot of 191,375 public Hugging Face repositories and a core corpus of 2,214 scholarly papers. Our results reveal a pronounced visibility-to-curatability funnel. While 90.7% of paper records contain at least one useful curation signal, only 18.1% combine usable upstream evidence with concrete release evidence, and only 6.1% provide sufficiently coordinated evidence to support high-curatability records. Based on these findings, we derive a minimal seven-field curatable record and complementary responsibilities for model hubs, scholarly indexes, and digital libraries, providing practical guidance for improving the preservation and bibliographic control of open LLM artifacts.

0.48

Surprise Alone Learns Lies: Evidence-Gated Continual Learning for LLMs from Streaming Claims

cs.LG cs.CLcs.CR ·2026-10-29 ·Abdelkader Metales, Miriam Grabenheinrich
code/data similarity 0.59
Abstract · 21 pages, 5 figures. Code: https://github.com/Metalesaek/sura-metales

Continual learning methods for large language models increasingly decide what to learn by surprise, keeping the inputs the model finds least predictable. This rule is unsafe when the stream contains lies. A planted falsehood about something the model knows has low probability under the model's belief, so it is highly surprising by construction, and a surprise-driven learner absorbs nearly every such lie, at a rate similar to random selection with the same budget. We study the problem with LoRA-adapted students (Qwen2.5-1.5B, Qwen2.5-3B and Llama-3.2-3B) that learn from a stream of true and false claims about invented people and real CounterFact facts. We report four findings. (1) A verifier key, a judge that must approve each claim, removes lies to the extent that its verifier is accurate; real LLM judges are cautious and reject unsourced truths and even source-backed updates. (2) Calibrated regression tests on unrelated knowledge slow learning down without filtering out lies. (3) The original model's own prior flags claims that contradict what it knows, without a judge. It separates lies about known facts from true new facts (AUC 0.78-0.83 across three models) but not from legitimate updates of the same kind of fact (AUC 0.53). It saves 43% of judge calls, but an attacker with access to the model can choose lies that pass it. (4) If trusted sources exist that an attacker cannot plant, routing contradicted claims to evidence recovers legitimate updates and the combined system has the lowest poison rate among learned claims; when an attacker plants matching sources for 30% of the lies, the system ends up worse than the judge alone. The acceptance signals answer different questions, and none of them decides on its own whether a claim is true.

0.53

Reconcile Once, Write Anytime: A Trust-Tiered Librarian and a Multi-Agent Writer for Drift-Free, Point-in-Time Research

cs.MA cs.CL ·2026-08-13 ·Xing Zhang, Yanwei Cui, Guanghui Wang, Peiyang He
similarity 0.59
Abstract

Long-form research reports generated by large language models drift, contradict themselves, and lose provenance: the same metric appears with different values, and rumor is quoted as confidently as an audited filing. We present a two-tier agentic system that separates a maintained, point-in-time knowledge library from report writing. A deterministic "librarian" ingests timestamped sources into a trust-tiered ontology, layering evidence cards, an authoritative metric ledger, and a claim graph into an always-current source of truth, not per-query RAG over raw chunks. A portable multi-agent "writer" runtime then composes a contradiction-free, evidence-grounded report at any knowledge cutoff T, reading only evidence with as_of <= T (no look-ahead); red-team verdicts flow back into the librarian. We evaluate on a self-collected, public corpus of 6,130 sources yielding 555,926 evidence cards (SEC EDGAR filings across 295 issuers and 11 sectors, U.S. Bureau of Labor Statistics releases, and Wikipedia). From the one library we compose four point-in-time reports on distinct theses and run eight reproducible experiments, whose headline metrics come from a deterministic quality-control gate, itself validated by defect-injection meta-evaluation at recall 1.0 and precision 1.0. A shared metric ledger removes 6,845 cross-section contradictions to zero. Tier-first selection is correct on 22/22 gold cases where a popularity-first baseline scores only 9/22; trust tiering leaks zero media-sourced numbers, and no government statistic displaces a company's own filing. A red-team refutation propagates back and self-corrects a later run with zero manual edits. Replay exhibits zero look-ahead violations across seven cutoffs while the library grows from 235,373 to 555,312 cards. Difficulty-tiered model routing exceeds the all-Opus quality ceiling while running 3.7x faster than serial.

0.04

LLM-Generated Investment Research Memos Need Validation

cs.CE ·2026-11-03 ·Xukang Gong, Chenri Luo, Binyan Yu, Xiao-Yang Liu
publishedhot direction similarity 0.58
Abstract · 24 pages, 4 figures. Accepted at the TAE (Trust-AI-Eval) Workshop, NeurIPS 2026

LLMs are increasingly adopted to draft long-form investment research memos, but their reliability is judged claim by claim, while readers read each memo as a whole. In this paper, we show that a low claim-level error rate does not necessarily mean that a memo is reliable: even with a refutation rate of just 1-2%, roughly four in ten memos contain at least one claim contradicted by official filings. First, we generate 1,440 memos with four recent LLM-powered agents for 30 companies over 12 quarters, decompose them into 77,272 financial claims, and verify each against the same official source bundle by retrieving evidence, matching it to the claim, and verifying the outcome. Second, we find that only 1.17-1.77% of claims are refuted, yet 36.94-41.11% of memos contain at least one refuted claim, and agents differ in severity: Codex has the lowest memo-level rate; a qualitative review further groups errors into four patterns (time, unit, context, concept).

0.70

SAFE-MR: Evidence Sufficiency Learning for Selective Multimodal Rumor Detection

cs.CL ·2026-10-05 ·Shiwen Ni
similarity 0.57
Abstract

Multimodal rumor detectors increasingly rely on retrieved evidence, yet relevant evidence is not necessarily sufficient for verification. Missing provenance, duplicated reports, and unresolved contradictions can produce confident predictions without adequate support. We introduce SAFE-MR, a framework that separates claim veracity from evidence sufficiency. The method decomposes image-text posts into verifiable claims, constructs a relation-aware claim-evidence graph, and aggregates evidence using provenance and contextual compatibility. Separate veracity and sufficiency heads support selective prediction, while evidence interventions encourage stability under irrelevant additions and sensitivity to evidence removal. On NewsCLIPpings, VERITE, and XFacta, SAFE-MR achieves macro-F1 scores of 91.2%, 75.8%, and 85.2%, respectively. Against the matched backbone with evidence, its macro-F1 gains are 2.2, 4.9, and 4.8 percentage points. On the diagnostic selection set, SAFE-MR reduces AURC from 0.105 for maximum-probability rejection to 0.075 and lowers error at 80% coverage from 13.8% to 8.5%. Evidence-perturbation and ablation results support the role of sufficiency learning and intervention training in improving selective verification.

0.21

Fact-check? True!: Tracking Verification Requests and the Absence of Truth Verdicts across Correction Systems

cs.SI cs.CY ·2026-10-12 ·Morgan Wack, Patrick Warren
similarity 0.57
Abstract

Misperceptions can run in two directions. Individuals can either believe false information or reject true information, with experimental work finding that debunking efforts are effective at correcting both forms of misperception. Yet despite the efficacy of true corrections, current debunking systems focus almost entirely on the correction of falsehoods, with professional fact-checkers publishing few true verdicts and crowdsourced systems like X's Community Notes program unable, by rule, to display notes verifying true claims. As a result, we do not have a clear understanding of the extent to which overall misperceptions are driven by doubting true information. To answer this question we first draw on a novel dataset of 47,899 queries to X's AI assistant which track users asking for verification on post content. Surprisingly, among questioned posts that received a true or false verdict, Grok judged nearly identical shares as either accurate or inaccurate. Of the questioned posts, more than 83\% were not flagged by Community Notes and only 3.3\% carried a displayed note. Among questioned posts judged accurate by Grok, nearly 90\% received no note at all while fewer than 2\% displayed a note flagging them as misleading. This discrepancy does not seem to have stemmed from a lack of effort, with contributors writing 2,981 notes verifying questioned posts despite there being no way for Community Notes to display them publicly. Alongside further exploration of these findings we discuss both the system- and code-level changes that would be required to effectively address this asymmetric correction paradigm.

0.21