CS Paper Radar

Framing-Induced Vulnerability in Large Language Models: Evidence from Chinese News Headlines

Mo Zhou, Yiwei Zhang

cs.CL ·submitted 2026-11-10 03:54 UTC · v1

Abstract

Large language models increasingly evaluate, route, and moderate news, so a model's judgment of a headline has become a gatekeeping decision. We ask whether that judgment follows the fact being reported or the way the fact is framed. We combine an ecological audit of 63,926 Chinese news headlines with a controlled experiment. The experiment covers 30 real events, 30 synthetic events, and 29 natural headline pairs, tested on four Chinese models. Changing only the headline frame shifted both perceived misleadingness and event severity, and the direction and ordering of the shifts were the same for DeepSeek, MiMo, Dots, and LongCat. Emotionally intensified headlines raised severity ratings by 0.65 to 0.88 points, whereas positively reframed headlines lowered them by 0.34 to 0.52 points. Supplying the article body reduced this severity bias by 54 to 80 percent, and all four reductions were significant at p < .01. A small human reference study showed the same direction of effect. Headline-only LLM evaluation therefore carries a measurable robustness gap, and framing sensitivity belongs among the reliability properties that deployment and governance check.

Comment
7 pages, 5 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Framing by Wording, Framing by Selection: A Large-Scale Two-Dimensional Audit of French News Headlines, 2022-2025

cs.CL ·2026-08-12 ·Amr Sobhy
published similarity 0.68
Abstract · 20 pages, 1 figure, includes appendices. Accepted for oral presentation at ICNLSP 2026

News headlines frame public issues both by what they select and by how they word it, yet computational framing work typically collapses these operations into a single score. We introduce a two-dimensional framework that separates salience framing, measured through four wording devices (loaded vocabulary, blame attribution, threat framing, rhetorical question), from selection framing, measured through outlet-level story-form and high-charge distributions. We build a 10,000-headline French supervision set using three LLM annotators with majority-vote resolution and human arbitration, validate the labels against two annotator-independent blind human studies, and apply the strongest classifier to 902,111 deduplicated headlines from 25 French outlets (2022-2025). Three main findings emerge. First, salience and selection divergence are positively correlated yet leave nearly half of outlet-level variance unexplained, populating interpretively distinct off-diagonal cells in a four-cell outlet typology. Second, default classification thresholds systematically inflate corpus-level salience estimates; a precision-floor recalibration protocol corrects this distortion. Third, group-mention analysis reveals sharply unequal salience contexts: headlines mentioning Jews, the Far-right, and Muslims carry the highest detected salience rates, which broad event-context composition does not fully explain (residuals are descriptive, not same-event causal estimates; per-group lexicon precision is reported alongside). To our knowledge, this is the largest framing-focused French headline audit to date; we release the supervision set, lexicons, and analysis code.

0.25

Recognizing Is Not Reversing: A Controlled Inversion Test of Fact-Preserving News Framing

cs.CL cs.AI ·2026-09-10 ·Yi Liu
similarity 0.68
Abstract

Large language models (LLMs) are increasingly used to analyze and rewrite news, yet current framing studies mainly evaluate generation, detection, or whether rewritten text appears more neutral. They do not directly show whether a model can undo a known framing transformation while keeping the facts fixed. We introduce a controlled inversion test over three established textual realizations of framing: evaluative lexis, agency realization, and information salience. Across 60 news articles and three intervention strengths, this yields 540 paired variants with preserved atomic facts and recorded edits. Across Qwen, DeepSeek, and Kimi, factual preservation remains near 0.84, whereas intervention reversal is 0.044--0.068. Even when both framing type and direction are recognized correctly, pooled reversal reaches 0.071. These results reveal a clear separation between factual fidelity, framing recognition, and framing inversion: recognizing how an article is framed does not imply that the framing can be undone.

0.50

Veracity labels flip the emotional tone of language models but leave racial disparities intact

cs.CY cs.CL ·2026-10-08 ·Nouar AlDahoul, Myles Joshua Toledo Tan, Yasir Zaki
code/data similarity 0.68
Abstract · 14 pages, 7 figures; Supplementary Information (42 pages, 6 figures, 27 tables) included. Data and code: https://github.com/Nouar-NYUAD/llm-…

Large language models (LLMs) increasingly deliver information through personas, but it is unclear whether correcting the facts also makes their framing consistent across identity labels. We audited four LLMs (GPT-5.4 mini, NVIDIA Nemotron 3 Nano, Llama 4 Scout, and Claude Haiku 4.5), which rated simulated affect on six dimensions (mood, anger, fear, confidence, perceived control, and mental state) in response to fact-checked false and true headlines presented without and with a veracity label. Experiment 1 evaluated 1,000 false and 1,000 true headlines without personas; Experiment 2 evaluated 300 of each under 12 personas crossing race, gender, and age. The label strongly changed the emotional tone: ordinal logistic regressions with headline-cluster bootstrap inference showed that refuting false headlines shifted ratings from mostly negative to mostly positive for GPT, Llama, and Claude (e.g., Claude's "confident" ratings rose from 22% to 99.6%), whereas Nemotron showed smaller, mixed shifts, and confirming true headlines produced only small shifts. The label did not, however, remove persona-associated disparities. The models assigned more negative simulated affect to Black than to White personas in 82 of 84 statistically significant comparisons across models and conditions, both without and with the label, and the disparity persisted on headlines unrelated to race; Asian and female personas were also predominantly assigned more negative affect, whereas age effects varied by model and condition. Factual correction and demographic consistency are therefore distinct properties of LLM behavior, and persona-conditioned systems should be audited for both before deployment in sensitive contexts.

0.39

How China-Origin Vision-Language Models Move from Refusal to Reframing in State Alignment

cs.CR cs.AIcs.CL ·2026-08-12 ·Guang Yang, Fengchen Liu, Alex Wang, Homa Hosseinmardi, Amir Ghasemian
benchmarkposition similarity 0.65
Abstract · 41 pages, 31 figures, 9 tables. Preprint

State-aligned distortion has been documented in China-origin text-based large language models (LLMs), but whether, and in what form, it arises in multimodal systems has not been systematically examined. We construct a balanced benchmark of 200 core entries spanning ten politically sensitive topics, plus a seven-variant visual-abstraction probe, and run nine vision-language models (VLMs), seven China-origin and two non-China, across four elicitation paradigms and two prompt languages, yielding 21,708 trials. Each response is audited on six dimensions -- explicit refusal, information integrity, visual grounding, state-aligned framing, language consistency, and response length -- by two independent frontier LLM judges, validated against three human experts on a 200-trial sample. Measuring each dimension separately lets us decompose multimodal censorship into individual signals rather than a single refusal-based score; in particular, refusal and framing are measured independently, so a model can stop refusing while still reframing. We find that (i) Chinese-language prompting roughly triples the odds of state-aligned framing, within every model; (ii) China-origin models reframe more than non-China models (direction robust across judges and human raters; magnitude 1.6--3.2x); (iii) the effect is strongest in text-only political commentary (36.5%) and is gated by recognition of the depicted subject rather than pixel detail, persisting even at silhouette for iconic images; and (iv) across four Qwen generations, state-aligned framing rises while explicit refusal falls: censorship migrates from a visible act (refusal) to an invisible one (fluent reframing). We argue this shift to invisible reframing is fundamentally a problem of human-AI interaction: it removes the very signal users rely on to recognize that information has been withheld.

0.44

Threat Amplified, Blame Restrained: LLM-Assisted Media Framing Analysis of the 2026 Bangladesh Measles Outbreak

cs.SI cs.CY ·2026-09-23 ·Shahan Ahmed
similarity 0.65
Abstract · 10 pages, 2 figures

How news media frame and emotionally code a public health emergency shapes public risk perception and trust, yet outbreak-coverage dynamics remain understudied for low- and middle-income countries (LMICs). We examine sentiment and stance in English-language Bangladeshi coverage of the 2026 measles outbreak -- the country's most severe in two decades, with over 97,000 suspected cases and 600 deaths across 61 of 64 districts, unfolding after the 2024 change of government and a 2024-2025 vaccine stockout. Using the Internet Archive, we build a reproducible corpus of 403 headlines from seven national outlets (396 in-window in 2026), label them for binary sentiment and four-way stance via a large language model under a locked codebook, and validate against a two-coder human-adjudicated gold standard (n=153; Cohen's kappa=0.89 stance, 0.75 sentiment). Aligned to the DGHS epidemic curve, coverage grew significantly more negative (56% to 88% negative; Cochran-Armitage z=4.12, p<.001) and risk-amplification framing intensified (44% to 84%; z=4.15, p<.001). Media negativity lagged incidence, tracking cumulative mortality. Contrary to the political backdrop, blame remained a minority frame (~9% overall) and was overwhelmingly systemic (32 of 37, 86%) rather than directed at named actors. The pipeline offers a scalable, transparent method for LMIC outbreak-media analysis; Bangladeshi coverage amplified threat far more than it assigned political blame.

0.32

Evaluating Bias in Chinese Large Language Models across Tasks and Registers: Task Operationalization Effects and Irony Classification Risks

cs.CL cs.CY ·2026-10-10 ·Mo Zhou
similarity 0.65
Abstract · 11 pages, 3 figures

In Chinese content governance, large language models are increasingly used to identify toxic and discriminatory expressions, yet both task operationalization and ironic expression can alter bias measurement readings. We conducted a preregistered multi-task behavioral audit of multiple Chinese large language models. In the content classification experiment, the core panel's classification error rate rose from 10.9% in the formal register to 23.5% in the yinyang (an echoic-irony register) version, a risk difference of +12.7 percentage points. (95% CI [+9.7, +15.7]). Inter-rater agreement was lower in the yinyang register (majority-vote kappa = 0.299), so the increase contains a component of label uncertainty propagation, but the direction of increase was consistent under both adjudicated and majority-vote reference standards. Explicit irony-aware prompts reduced the intervention panel's error rate by 9.6 percentage points. We operationalized the 29 shared proposition anchors into three task forms: social consensus estimation (Form A), high-stakes decision (Form B), and controlled association rating (Form C); independent manipulation checks found that all 58 A-C comparison units received unanimous agreement, whereas the low A-B convergence was confirmed by calibrated majority adjudication (87.8% supporting the minority judgment); on the irony-probe materials, the interactions of B and C relative to A were +0.577 and +1.121, respectively. Thus, when evaluating large language models for Chinese content governance, the effect of task operationalization on bias readings is an empirical question that requires data to answer, and safety evaluation should incorporate the ironic register as a test condition.

0.17

Reporting Under Pressure: Separating Factual and Tonal Sycophancy in LLM Statistical Analysis

cs.AI cs.CL ·2026-08-15 ·Paras Balani, Subhrakanta Panda
similarity 0.65
Abstract · 9 pages, 2 figures

Large language models are increasingly asked to analyze data and report what the results mean, a task distinct from the belief- or preference-alignment settings studied in most sycophancy research. We test whether editorial framing in the prompt, ranging from a neutral request to an explicit instruction to search exhaustively for reasons to discredit or to support a finding, changes not just the tone but the substance of a model's report. Across a 4 x 4 factorial design crossing four framing conditions with four ground-truth data patterns (a genuine effect, a confound that mimics an effect but fails a robustness check, a well-powered null, and an underpowered null), we collect 480 responses and score each along two independent dimensions: whether its factual claim about the data diverged from the correct interpretation, and whether only its tone diverged while the claim stayed correct. Factual misrepresentation is concentrated in two cells: brutally critical framing applied to a genuine effect, where the model talks itself into unwarranted skepticism (97% of responses), and significance-seeking framing applied to an underpowered null, where the model overstates confidence in a null conclusion the data cannot support (100% of responses). Tone shifts far more broadly than factual content does, with critical framing producing a defensive, hedge-heavy register across every data pattern regardless of what the data show, while significance-seeking framing shifts tone only where the data leave genuine ambiguity. A confound present in the data itself blocks both kinds of shift almost entirely under every framing condition tested. These results indicate that the risk of framing-induced distortion in LLM-assisted data analysis is neither uniform across framings nor uniform across data patterns, and that a model can hold a correct conclusion in place while its tone shifts substantially around it.

0.44

Who Judges the Frame? Auditing Multimodal LLM Judges for News Framing Across Event-Level Perspectives

cs.CY ·2026-09-04 ·Antonela Tommasel, Markus Schedl
published similarity 0.64
Abstract

News coverage of major world events is shaped not only by what is reported, but also by how events are framed through text, images and their combination. At the same time, Large Language Models (LLMs), including multimodal LLMs, are increasingly used as scalable instruments for analysing framing, sentiment, ideological slant and perspective differences in multimodal media datasets. This creates a methodological challenge. When used as measurement instruments, LLM outputs may reflect not only content properties, but also model-specific tendencies, prompt design choices, and social, political, cultural, linguistic or modality-specific assumptions. This work audits LLMs as instruments for large-scale framing and perspective analysis in multimodal news coverage. Using an event-centered dataset of 2025--2026 news coverage, where each event includes left-, center- and right-oriented articles about the same headline, we combine embedding-based measures of within-event viewpoint similarity with model-based assessments of framing constructs across modality-specific and metadata-visible input conditions. Rather than treating either dataset labels or model outputs as ground truth, our goal is to examine the usefulness and limitations of LLM-based media analysis. The study contributes an audit protocol that highlights the need to report modality effects, metadata sensitivity, and prompt-induced artifacts alongside substantive claims about news framing and ideological viewpoint differences.

0.30