CS Paper Radar

SWARM: A Multilingual Human-Annotated Dataset for Russian Propaganda Detection in Search Engine Results

Manuel Tonneau, Abhinav Dubey, Farhan Shaikh, Ilaria Vitulano, Martha Stolze, Hale Dedeoglu, Clara Riechert, Ella Kuka, Maryna Sydorova, Mykola Makhortykh, Elizaveta Kuznetsova

cs.CL ·submitted 2026-09-11 09:58 UTC · v1
benchmark

Abstract

Russian state propaganda spreads across many languages and online spaces. Yet, most computational work examines only one such space, usually social media, in one or two languages, and analyses sources rather than content. We introduce SWARM (Search-Web documents Annotated for Russian propaganda, Multilingual), a dataset of 2,183 search engine results across nine languages and diverse web domains (e.g., news, blogs, government sites), each annotated by trained coders for whether it supports a recurring Russian propaganda narrative. We benchmark a source-based blocklist, supervised classifiers, and zero-shot LLMs against these labels. The blocklist misses most propaganda-supporting documents, because such content is not confined to flagged "propaganda" outlets but also appears on mainstream ones. Content-level analysis helps, though how much depends on the model: the strongest LLM reaches a positive-class F1 of 0.73, whereas the supervised classifiers reach only about 0.5, with the smaller LLMs over-predicting support, mistaking topical relevance for endorsement. Detecting search-borne propaganda thus requires per-language, content-level evaluation, which we hope SWARM and our evaluation code enable.

Comment
Accepted at the 10th Workshop on Online Abuse and Harms (WOAH 2026)

Closest papers

Nearest neighbours by meaning across the whole index.

The World as Viewed by 136 Countries: An Open, Curated, Annotated, and Multilingual Dataset of 118 Million State Media Articles

cs.CL cs.CY ·2026-10-18 ·Bryan Li, Samar Haider, Grace Kim, Fiona Luo, Jinqi Luo +4 more
datasetcode/data similarity 0.73
Abstract

State media outlets often project their governments' narratives to target both domestic and foreign audiences. Because this content is plentiful and free to access, it appears widely in the web corpora that large language models (LLMs) are trained on and that retrieval systems draw from. However, the prevalence of state media in these corpora and its potential influence on LLM outputs remain largely unexplored. In this paper, we introduce WorldStateMedia, an open dataset of 118 million state media articles from 136 countries in 114 languages. We extract these articles from existing pretraining corpora and annotate each outlet with level of state control from State Media Monitor. Using WorldStateMedia to audit these corpora, we find that, in English web crawls, the share of state media pages from editorially controlled (vs. independent) outlets rose from 39% in 2013 to 93% in 2025. In several low-resource languages, state media make up a substantial share of all web data, such as 17.8% for Uyghur and 7.8% for Belarusian. To characterize the differing narratives of news outlets, we compare Russian and Ukrainian state media in English and in their native languages, revealing asymmetric topical focus and opposing framings of contested issues such as Crimea. Furthermore, in a retrieval-augmented generation (RAG) experiment on territorial disputes, retrieving a country's state media shifts model answers toward that country's claims, most strongly for English-language (foreign-targeted) media. We release WorldStateMedia to support multilingual, cross-national insights on state media at https://huggingface.co/collections/manestay/worldstatemedia

0.24

Propasafe-Hybrid: A Text-Based Hybrid Propaganda Detection Tool

cs.CL cs.CY ·2026-10-24 ·Thomas Kimmeth, Avijit Roy, Vivek Sharma
published similarity 0.68
Abstract · Published in the Proceedings of FLAIRS-39 (2026), Special Track: Applied Natural Language Processing

Propagandistic content increasingly circulates through online news and social media, where readers often encounter it with limited scrutiny, highlighting the need for reliable and fine-grained detection. This paper introduces Propasafe-Hybrid, a sentence-level system that integrates a fine-tuned transformer classifier with LLM-based technique classification to identify, label, and explain specific propaganda strategies. The pipeline generates actionable outputs, including highlighted sentences, technique assignments, and concise rationales, so users can immediately understand why a sentence was flagged and how each label was determined. To control inference cost, Propasafe-Hybrid employs a cost-aware pre-filtering stage that forwards only high-likelihood sentences to LLMs, reducing token usage while preserving the underlying decision logic. Together, these design choices enhance the explainability, efficiency, and practical usability of sentence-level propaganda detection in real-world news environments.

0.24

Unmasking Propaganda: A Comparative Analysis of Masked and Causal Language Models

cs.CL ·2026-10-02 ·Claudiu Creanga, Ioachim Lihor, Liviu P. Dinu
similarity 0.68
Abstract

Propaganda detection is an essential task in natural language processing (NLP), particularly in the context of manipulative political communications. However, identifying specific propaganda techniques presents a significant challenge due to their often subtle nature and reliance on context, making them difficult to distinguish from legitimate persuasive language. Propaganda often involves highlighting certain facts while downplaying or ignoring others to create a desired perception. This biased communication aims to influence attitudes, beliefs, or behaviors towards a particular cause or position. This paper explores advances in detecting propaganda techniques through a comparative analysis of modern language models, using the SemEval-2020 Task 11 dataset. We evaluated both masked language models (based on XLM-RoBERTa or DeBERTa V3) and causal models (from OpenAI, Google, Mistral, Anthropic and Meta), employing two prompting strategies: base and chain-of-thought prompting. Our results demonstrate improvements over state-of-the-art models, with the best-performing MLM achieving an F1 score of 63.18 in technique classification and the best causal model achieving 63.62. We also observed that certain models excel in specific techniques, such as loaded language and name-calling, while struggling with others like bandwagon and black-and-white fallacy. These findings suggest that fine-tuning, ensemble modeling, and the use of larger datasets can further enhance propaganda detection capabilities.

0.14

ProBel: Propaganda Detection with Techniques, Spans, and Explanations

cs.CL cs.AIcs.LG ·2026-08-23 ·Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Elisa Sartori, Giovanni Da San Martino, Firoj Alam
similarity 0.67
Abstract · Under review at

Propaganda detection includes several related prediction levels, ranging from sentence-level decisions to technique classification and span identification. However, it remains unclear how supervision at these levels interacts when learned jointly across Arabic and English. We present ProBel, an Arabic and English resource that aligns binary labels, multi-label annotations over 23 propaganda techniques grouped into six coarse categories, technique-labeled spans, and reference explanations for the same news sentences. It includes a substantially larger English collection and supports matched binary, coarse-grained, multi-label, and span-level tasks in both languages. We evaluate zero-shot prompting, task-specific fine-tuning, and joint training under a shared setup. A single bilingual multi-task model achieves the best overall performance and remains competitive across tasks and languages. Cross-task analysis shows that transfer depends on the supervision level. Joint classification training preserves binary performance, whereas span-only training can weaken sentence-level prediction. Joint bilingual training yields the most stable results, while monolingual fine-tuning can reduce transfer to the other language. We will release the data, code, and evaluation scripts.

0.17

Propaganda Forensics: Recovering the Generation Pipeline of an AI-Driven Influence Campaign

cs.AI cs.CL ·2026-08-16 ·Benjamin Icard, Elouan Vuichard, Louis Lefebvre, Lila Sainero, Thomas Girault +10 more
similarity 0.67
Abstract · To appear in the Proceedings of the 10th Workshop on Online Abuse and Harms (WOAH), EMNLP 2026

We present a forensic analysis of the generation pipeline behind a recent AI-driven influence campaign. We introduce PROPAGIA, a corpus of 2,646 propagandist French articles from the Storm-1516/CopyCop campaign disclosed by VIGINUM and INSIKT GROUP in 2025. For comparison, we rely on SIPA, a corpus of human-written French mainstream press from the same period. Using topic modeling, vagueness and sentiment analysis, we first isolate persuasion techniques characteristic of propaganda, with PROPAGIA far exceeding SIPA in vagueness, subjectivity and negativity, and citing fewer sources. We then find prompt instruction leaks on 50 of the 84 PROPAGIA websites, including a verbatim ten-point editorial specification accounting for several of these differences, together with high cross-article redundancy. Finally, we show that rewriting-based detection supports INSIKT GROUP's attribution to the Llama 3 family, but also suggests the involvement of Mistral-family models.

0.15

MIL-BERT: Classification of Arbitrarily Large Text with Performance and Explanatory Guarantees

cs.CL cs.LG ·2026-08-21 ·John Cadigan, Dayne Freitag, Eric Yeh
similarity 0.65
Abstract

Many text classification decisions are viable based on constituent excerpts alone. Taking inspiration from the field of multiple instance learning, we present an algorithm for training a neural network to classify text by selecting such excerpts. We show that our approach is also scalable with demonstrated learning against samples with nearly 1M tokens. We evaluate our methods on 7 datasets with emphasis on long-textual collections that far exceed the encoding limit of our base model. We present state-of-the-art results with this algorithm on 3 datasets: identification of political bias in news outlets, trigger warnings in long stories, and demographic characteristics of authors in tweet collections. Furthermore, the model trained on weakly-labeled collections of text (bags) generalizes to accurately classify constituent, smaller instances. Besides a new state-of-the-art for these problems, this approach is one of the few neural methods to excel in these datasets.

0.12

A Decade of Climate Polarization on Brazilian YouTube using Language Models

cs.SI cs.CLcs.CY ·2026-08-18 ·Daniel Morais, Diego H. M. Magalhaes, Gabriel H. Silva, Andrea Failla, Valeria de C. Santos +2 more
published similarity 0.65
Abstract · Accepted at ASONAM 2026

Online platforms have become arenas for the public contestation of climate change, shaping how scientific knowledge, denial, and uncertainty are expressed and disputed. Yet longitudinal evidence remains limited for YouTube, especially for Portuguese-language discourse. Addressing this gap, we characterize how climate stances are expressed and contested over time in a large corpus of Portuguese-language YouTube comments retrieved through Brazil-oriented climate-related searches. To support this analysis in a noisy, imbalanced, and low-resource setting, we collect more than 240,000 comments posted between 2014 and 2024 and formulate stance detection as a three-way classification task (Believer, Denier, and Inconclusive). We operationalize stance attribution through a scalable self-training pipeline based on Llama 3.1, using Low-Rank Adaptation (LoRA) and hybrid instance selection to expand the training set with high-confidence pseudo-labeled examples while preserving class diversity. This approach improves coverage and class balance for minority and rhetorically complex classes, enabling large-scale stance attribution without extensive manual annotation. Our results show that polarisation is marked by interactional asymmetries: denialist comments are less prevalent, but they are associated with a comparatively higher share of cross-stance contestation, while pro-consensus discourse is more strongly reinforced within stance-homogeneous threads.

0.18

Where Does Rhetorical Distortion Live? A Multi-Platform, Multi-Language Measurement of Influencer Content

cs.SI cs.CLcs.CY ·2026-10-19 ·Xucheng Yu, Jiangping Chen, Haohan Wang
similarity 0.65
Abstract · 12 pages

Much of the manipulation in influencer content is not false but disproportionate, framing accurate facts as historic, urgent, or catastrophic. Existing approaches either verify factual claims or detect clickbait and propaganda in single-platform, single-language text; however, they overlook where such framing originates, because they treat a post and the replies beneath it as one unit. To address this gap, we measure rhetorical distortion with one instrument applied identically across six platforms and two languages, classifying creator posts and their replies separately. Our three-tier pipeline detects five literature-grounded dimensions (Significance Inflation, Anxiety Manufacturing, Novelty Claims, Loaded Language, and Temporal Distortion): bilingual lexical rules propose candidates, negative filters cancel known false positives, and an LLM verifies or relabels the candidates. This design keeps labels transparent and comparable across platforms, calls the LLM on only 14% of posts, and supports validation against human annotators and a second model. On 13,060 posts from 330 accounts on RSS/newsletters, YouTube, Bluesky, Reddit, Weibo, and X/Twitter, platform is far more strongly associated with account-level distortion than content category (partial $η^2=0.49$ vs. $0.09$), and X/Twitter scores highest (DI 7.74 vs. 0.78 for RSS). On X/Twitter, the top-5 replies under a post carry 3.5x the distortion of the tweet they answer (1.5x at equal length), while creator tweets alone are indistinguishable from YouTube and Bluesky; two human annotators and GPT-4o confirm this gap. Measured distortion on threaded platforms thus lives largely in the audience layer, which calls for measuring creator and reply text separately.

0.24