CS Paper Radar

What Breaks When Facts Become Stories? The Distortions Source-Agreement Metrics Miss When Models Retell Real Events

Kwangsuk Park, JiWoong Yang, Vaniushkina Kseniia

cs.AI ·submitted 2026-10-29 07:02 UTC · v2
publishedcode/data

Abstract

Language models are now widely used to turn recorded facts into media for public consumption, yet evaluation of such output still assumes agreement with the source as its measure of success. In reconstruction, where departing from the source can itself be the goal, this criterion measures the wrong construct: applied as is, it counts intended departures as errors; withheld, it lets real distortion through. We had three model families rewrite 2,000 news articles as summaries and as stories, measured what changed on every axis of the five Ws and how, and asked of each detected distortion whether the assigned perspective warranted it, which set a third of them aside. Time and manner were not lost but replaced, while place and causation lost the expressions that carry them. Accountability distortion behaved otherwise. Barely present in summaries, it rose up to sevenfold in stories, and of those whose direction could be determined, two in three lightened the perpetrator's responsibility. Such distortions survive claim-level fact checking and leave the retelling grammatically cleaner than its source. Where our judges disagreed, three authors relabeling the same cases converged on whether a distortion had occurred but not on whether it was warranted, a boundary contested rather than merely hard to detect. To support work on detecting and reducing them, we release ReTold-2k, a reconstructable corpus with the scoring protocol and per-judge baselines kept unaggregated.(https://github.com/Retold2K/retold2k).

Comment
14 pages, 7 figures, 11 tables. Accepted at the Trust-AI-Eval (TAE) Workshop at NeurIPS 2026. Data and code: https://github.com/Retold2K/retold2k

Closest papers

Nearest neighbours by meaning across the whole index.

Recognizing Is Not Reversing: A Controlled Inversion Test of Fact-Preserving News Framing

cs.CL cs.AI ·2026-09-10 ·Yi Liu
similarity 0.67
Abstract

Large language models (LLMs) are increasingly used to analyze and rewrite news, yet current framing studies mainly evaluate generation, detection, or whether rewritten text appears more neutral. They do not directly show whether a model can undo a known framing transformation while keeping the facts fixed. We introduce a controlled inversion test over three established textual realizations of framing: evaluative lexis, agency realization, and information salience. Across 60 news articles and three intervention strengths, this yields 540 paired variants with preserved atomic facts and recorded edits. Across Qwen, DeepSeek, and Kimi, factual preservation remains near 0.84, whereas intervention reversal is 0.044--0.068. Even when both framing type and direction are recognized correctly, pooled reversal reaches 0.071. These results reveal a clear separation between factual fidelity, framing recognition, and framing inversion: recognizing how an article is framed does not imply that the framing can be undone.

0.51

When Generated Information Returns: Error Persistence in LLM-Mediated Information Ecosystems

cs.AI ·2026-10-24 ·Yi Liu
similarity 0.63
Abstract

Large language models (LLMs) increasingly produce online articles from material that later becomes input to other models. This circulation connects individual responses to the information available for subsequent generation. We study a consequential distinction: an article can avoid endorsing a false claim while preserving the concrete details that enable its reproduction. Our framework connects claim content, expressed stance, and downstream reuse to explain selective error persistence under a continuing supply of accurate and inaccurate information. We implement a controlled community writing process that varies whether published articles return as inputs, matching external supply, input capacity, and generation volume. Across twelve factual targets and four production models, recirculation increases the presence of complete false claims by 7.29 percentage points at the final round while reducing explicit endorsement by 8.33 points. Effects differ across facts and models. Paired interventions then abstract eligible error details within fixed histories while preserving correct content and the surrounding controversy. Error reproduction remains largely unchanged across the evaluation targets, while a development quotation target exhibits local reductions. These findings connect persistent errors to the information available throughout a writing process and motivate evaluation of both what models affirm and what their publications supply to future writers.

0.27

Framing-Induced Vulnerability in Large Language Models: Evidence from Chinese News Headlines

cs.CL ·2026-11-10 ·Mo Zhou, Yiwei Zhang
similarity 0.63
Abstract · 7 pages, 5 figures

Large language models increasingly evaluate, route, and moderate news, so a model's judgment of a headline has become a gatekeeping decision. We ask whether that judgment follows the fact being reported or the way the fact is framed. We combine an ecological audit of 63,926 Chinese news headlines with a controlled experiment. The experiment covers 30 real events, 30 synthetic events, and 29 natural headline pairs, tested on four Chinese models. Changing only the headline frame shifted both perceived misleadingness and event severity, and the direction and ordering of the shifts were the same for DeepSeek, MiMo, Dots, and LongCat. Emotionally intensified headlines raised severity ratings by 0.65 to 0.88 points, whereas positively reframed headlines lowered them by 0.34 to 0.52 points. Supplying the article body reduced this severity bias by 54 to 80 percent, and all four reductions were significant at p < .01. A small human reference study showed the same direction of effect. Headline-only LLM evaluation therefore carries a measurable robustness gap, and framing sensitivity belongs among the reliability properties that deployment and governance check.

0.14

Reporting Under Pressure: Separating Factual and Tonal Sycophancy in LLM Statistical Analysis

cs.AI cs.CL ·2026-08-15 ·Paras Balani, Subhrakanta Panda
similarity 0.62
Abstract · 9 pages, 2 figures

Large language models are increasingly asked to analyze data and report what the results mean, a task distinct from the belief- or preference-alignment settings studied in most sycophancy research. We test whether editorial framing in the prompt, ranging from a neutral request to an explicit instruction to search exhaustively for reasons to discredit or to support a finding, changes not just the tone but the substance of a model's report. Across a 4 x 4 factorial design crossing four framing conditions with four ground-truth data patterns (a genuine effect, a confound that mimics an effect but fails a robustness check, a well-powered null, and an underpowered null), we collect 480 responses and score each along two independent dimensions: whether its factual claim about the data diverged from the correct interpretation, and whether only its tone diverged while the claim stayed correct. Factual misrepresentation is concentrated in two cells: brutally critical framing applied to a genuine effect, where the model talks itself into unwarranted skepticism (97% of responses), and significance-seeking framing applied to an underpowered null, where the model overstates confidence in a null conclusion the data cannot support (100% of responses). Tone shifts far more broadly than factual content does, with critical framing producing a defensive, hedge-heavy register across every data pattern regardless of what the data show, while significance-seeking framing shifts tone only where the data leave genuine ambiguity. A confound present in the data itself blocks both kinds of shift almost entirely under every framing condition tested. These results indicate that the risk of framing-induced distortion in LLM-assisted data analysis is neither uniform across framings nor uniform across data patterns, and that a model can hold a correct conclusion in place while its tone shifts substantially around it.

0.44

Beyond the Evidence Coherence Trap: Agent-Driven Evidence Selection for Temporally Disordered Fake News Detection

cs.AI ·2026-10-27 ·Chi Wang, Min Gao, Yifei Zhang, Bincheng Gu, Zongwei Wang +1 more
benchmark similarity 0.61
Abstract

Fake news undermines public trust and the integrity of online information ecosystems. Temporally disordered fake news is especially elusive because it rearranges factual events into a false chronology. Individual events may remain factually valid even when their temporal relations are misrepresented. When similar events involving the same entity recur, retrieval may return semantically plausible evidence from the wrong period. Consequently, independent event verification may rely on locally coherent but temporally mismatched evidence and incorrectly validate the article. We call this the \textit{evidence coherence trap}. Because existing benchmarks rarely isolate this setting, we construct ChronoFake, a source-disjoint benchmark of 30,706 articles derived from temporally anchored real events. It distinguishes temporal manipulation from valid narration, including flashbacks. To address this, we propose DelSA, a \underline{D}ocument S\underline{el}ection \underline{S}elf-Updating \underline{A}gent that iteratively refines evidence selection during training and applies a frozen rulebook for one-pass evidence selection at inference. DelSA uses an LLM to characterize candidate evidence, while a verifier-trained reranker balances local relevance against timeline consistency. Finally, the classifier compares the article-implied chronology extracted by the LLM with the evidence-derived chronology to reveal global temporal conflicts. DelSA outperforms existing methods on ChronoFake, while remaining competitive on PolitiFact and GossipCop.

0.21

Trapped in Your Story: Evaluating Narrative-Induced Sycophancy in Large Language Models

cs.AI ·2026-10-16 ·Runtai Yang, Dandan Zhu, Kaiwei Zhang, Guangtao Zhai
benchmark similarity 0.60
Abstract

Large language models (LLMs) are known to exhibit sycophancy by adjusting their responses toward users' expressed views. However, real users often justify their opinions through reasons, personal experiences, causal explanations, and broader narratives, raising the question of whether such narratives can further steer model judgments. We construct a controlled benchmark of 500 debatable questions across ten domains, where the user stance is held fixed while its justification is systematically varied without introducing new task-relevant evidence. Each question is instantiated under 11 matched conditions: one neutral baseline and five mirrored narrative levels for each of two opposing positions, yielding 5,500 evaluation instances. We evaluate seven LLMs and observe substantial differences in narrative-induced agreement. The effect is not monotonic with narrative elaboration: personal experience can reduce agreement, whereas causal framing and worldview-based synthesis increase it. Controlled GPT-5.1 ablations further show that causal framing produces the strongest positive effect, while claim-preserving bullet presentation also increases agreement over paragraph form. These findings suggest that sycophancy is shaped not only by user stance, but also by how supporting explanations are structured and presented.

0.17

What Would Falsify It? A Variable Specific Evidence Standard for Mechanistic Claims About Self Explanation

cs.AI cs.LG ·2026-09-26 ·Arshia Eftekhari zadeh
similarity 0.60
Abstract

When a language model explains an answer it has already given, does it reuse the computation that produced the answer or reconstruct a story from the answer alone? Attribution, transportability and recoverability are each compatible with causal use without establishing it. We propose an evidence standard: pair each positive statistic with a variable specific null that removes the tested variable's identity while matching relevant nuisance dimensions as far as possible, and audit unmatched dimensions. We apply this standard to a known cause. A cue naming a wrong option raises the rate of choosing that option by 64 to 68 percentage points across three models. Explanations mention the cue in 1.8 percent of items or fewer in three of four models tested. Three estimator classes yield favorable statistics, but none establishes causal sensitivity to the cue contrast under its own control in the three-model analysis. In the strongest case, a recovered cue direction reaches $R^2$ of 0.95 and exceeds a geometry matched random direction in all three seeds, while a direction fitted by the same pipeline with cue labels scrambled reproduces 61 to 76 percent of its effect at comparable realized edit magnitude. A fourth model passes one interchange endpoint, but unequal edit magnitudes and a contrast that changes both cue identity and cue-answer agreement limit its interpretation. These experiments leave causal access unresolved. They establish an evidentiary requirement: favorable mechanistic statistics must survive controls for variable identity and nuisance structure. Reusable controls separate generic from identity specific transport effects, fit null directions with scrambled labels, and audit realized intervention magnitudes.

0.25

What Counts as Evidence? Epistemic Cue Utilization in Large Language Models

cs.CY ·2026-10-08 ·Jinchang Zhang, Kaijun Jiang, Guoyu Lu
benchmark similarity 0.60
Abstract

Which information actually shapes a language model's judgment when sources disagree? We study epistemic cue utilization through a controlled benchmark of 500 scenarios and five complementary tracks separating cue identification, cue influence, and competition between source, temporal, and presentation cues. Across 13 LLMs, nearly identical recognition of relative source reliability can accompany effects ranging from negligible to complete choice reversal when source histories are exchanged. The resulting profiles distinguish models that rely on source history from those more responsive to evidence lineage, event time, or presentation order. We then intervene on three Qwen checkpoints using matched sham material, fact restatement, structured evidence records, and an additional weighting principle. Restatement increases reliability sensitivity in the two 27B models, and adding the principle increases it by a further 39.2 and 47.2 percentage points relative to the ledger. The 9B model instead changes its sensitivity to evidence lineage without a comparable reliability response. Verified-state accuracy remains 100\% throughout these interventions. Together, the benchmark and interventions show that identifying an epistemic cue and allowing it to influence judgment are distinct, experimentally measurable aspects of model behavior.

0.08