CS Paper Radar

Beyond Truth Discovery: A Two-Stage Framework to Assess the Severity of False Claim during Disasters

Ruichen Yao, Tejna Dasari, Gulshat Baispay, Aizhan Zaurbek, Yifan Liu, Yaokun Liu, Zelin Li, Dong Wang

cs.SI cs.HC ·submitted 2026-08-21 11:09 UTC · v2
publishedbenchmark

Abstract

False information spreads rapidly on social media during disasters and can undermine emergency response efforts, public trust, and crisis communication. Existing research primarily focuses on determining whether social media posts contain false information, but provides limited insight into the specific false claims embedded within posts and the severity of individual false claims. To address the limitations, we propose a two-stage framework to assess the severity of false claims during disasters. In the first stage, we develop a false claim extraction agent that identifies false claims from multimodal social media posts containing text, images, videos, and links. A subsequent verification step validates extracted claims with supporting evidence. In the second stage, we define false claim severity as the combination of two complementary dimensions: believability, which determines the likelihood that a claim will be believed, and harmfulness, which captures the potential consequences if it is believed. Human annotators assess both dimensions to construct a claim-level severity benchmark using false claims extracted from Reddit posts related to hurricanes and wildfires. Building upon this benchmark, we investigate false claim severity assessment as a human-AI alignment problem, evaluating whether models can reproduce human judgments under a shared evaluation rubric rather than merely predicting severity labels. Experiments on the benchmark show that traditional supervised models exhibit limited alignment with human judgments, whereas Large Language Models (LLMs) achieve substantially stronger performance. Among the evaluated strategies, in-context learning consistently achieves the strongest alignment with human judgments, highlighting the importance of human examples and shared decision criteria for severity assessment.

Comment
The 2026 ACM Conference on Human-AI Complementarity and Alignment (HCOMP 26)
DOI
10.1145/3834580.3838746

Closest papers

Nearest neighbours by meaning across the whole index.

CrisisFake: Benchmark Validity of AI-Generated Text Detection for Disaster Social Sensing

cs.CL cs.CY ·2026-09-21 ·Xiaoshan Zhou, Zaifu Zhan
benchmark similarity 0.78
Abstract

Disaster social sensing converts public social-media posts into evidence for situational awareness and humanitarian response, but plausible LLM-generated posts can contaminate this information stream and distort assessments of needs, damage, and resource priorities. This study empirically investigates whether text-based detectors can distinguish human-authored from LLM-generated disaster posts and what textual cues underlie their judgments. We construct CrisisFake, a Qwen2.5-7B-based dataset of 12,000 texts organized into 3,000 matched semantic units from nine disasters. Each unit contains an original human post, a minimally LLM-proofread human post, a fact-preserving LLM-generated post produced using LoRA, and an affectively reframed version of the artificial post. A separate 6,000-text corpus spanning 42 disaster events is independently constructed to support model selection and threshold calibration. We evaluate OSM-Det, Fast-DetectGPT, Binoculars, and direct LLM judges across five model families, and further examine how disaster-domain calibration and superficial linguistic cues, such as retweet markers, user mentions, hashtags, URLs, punctuation, and text length, affect detector performance. Across fourteen frozen configurations, AUROC ranges from 0.402 to 0.517, while the best prospective recall at a calibration-derived low-false-positive operating point is only 3.6%, indicating near-chance discrimination. For OSM-Det, a disaster-calibrated linear head improves AUROC to 0.817; however, a seven-feature textual classifier alone reaches AUROC 0.784 on the original-versus-factual-LLM contrast, and neutralizing identified surface asymmetries reduces the corresponding linear-head AUROC from 0.733 to 0.594. These findings provide empirical evidence that LLM-generated text detection is largely driven by linguistic cues and remains insufficiently robust for short-form disaster social media.

0.05

Can We Defend Against AI-Generated Video Attacks on Real-World Crisis Events? A Systematic Evaluation of Detectors, Generators and Social Dissemination

cs.CV cs.AI ·2026-08-14 ·Shuo Liang, Yixing Ma, Pengfei Zhou, Zhenglin Wan, Xingyan Chen +31 more
benchmarkcode/data similarity 0.67
Abstract · 63 pages, 20 figures, 32 tables

Recent video generators can fabricate realistic depictions of wars, disasters, public emergencies, and other real-world crises, creating substantial risks of misinformation. Existing benchmarks, however, provide limited evidence on detector and generator behavior in such settings, including how detectability varies with generation conditions, how people perceive generated videos, and whether detectors remain reliable during social dissemination. To address this gap, we introduce RA-Bench, a benchmark for AI-generated video detection that uses Real videos as Anchors. RA-Bench contains 17,886 videos, comprising 1,830 real-video anchors across 10 social-risk categories and 16,056 generated clips from four open-source and five closed-source generators. Based on RA-Bench, we organize our evaluation along three dimensions. We first assess detector generalization across seven traditional detectors, ten zero-shot multimodal models under three review settings, and two MLLMs specifically fine-tuned on AI-generated video detection. Across these methods, none of the three detector families generalizes consistently across RA-Bench instances. We then examine how detectability varies with generation quality, conditioning information, and sampling seeds. These analyses show that generation properties affect detector families differently, while source-level detection patterns remain stable across seeds. Finally, we study human authenticity judgments and detector reliability during social dissemination. We find that videos that mislead people are also difficult for current detectors, and that social dissemination makes detection harder. Together, these findings show that current methods struggle to detect realistic AI-generated videos, highlighting the need for detectors robust to evolving video generators.

0.56

Edge-Deployable LLMs for Classification and Value-of-Information Ranking of Disaster-Related Tweets

cs.IR ·2026-11-06 ·Christopher D. Molek, K. Brent Venable, Niranjan Suri, Stephen Russell
similarity 0.65
Abstract · 6 pages, 6 figures, 1 table, IEEE MILCOM 2026

Humanitarian-aid and disaster relief decisions often rely on noisy, incomplete, and uncertain information from social media. Although substantial research has focused on automating the classification of such data, limited work has examined whether individual posts contain sufficient information to support decisions to act. In this study, we use large language models (LLMs) to (1) classify social media content and (2) assess its Value-of-Information (VoI) in terms of operational actionability. We evaluate seven LLMs for the classification of 77,484 tweets across 10 humanitarian-aid categories and identify the Qwen3 (8B parameters) model as the top-performing classifier, achieving the highest F1 scores in all but one category. In a second task, the classified categories serve as contextual inputs for assessing VoI with respect to responder actionability across six well established metrics for each tweet. We observed that the Gemma 3 (4B parameters) model outperformed the other models on this task, achieving the highest recall across the majority of categories. This work demonstrates that relatively small, edge-deployable LLMs can prioritize actionable information in resource-constrained environments. By integrating VoI assessment, our framework advances the automated elicitation of high-utility knowledge from unstructured social media streams for disaster-response decision-making and highlights a pathway toward tactical edge applications using LLMs that fit on a single 8 GB GPU.

0.22

Effective and Efficient Rumor Assertion Over Large-Scale Social Networks (Technical Report)

cs.SI cs.CY ·2026-11-16 ·Qi Wen, Vipin Chaudhary, Xiang Lian, Yutong Ye, Mingsong Chen
similarity 0.64
Abstract

Nowadays, social networks, like Facebook, X, Reddit, YouTube, and so on, have become essential online platforms for users to connect with friends daily, post/share news, and express their opinions/comments. Meanwhile, however, the rapid and widespread dissemination of rumors and misinformation across social networks has become a serious concern. Such misleading information can distort public opinion and potentially cause significant social and economic consequences in real-world scenarios, including product promotion, elections and voting, and even cyberbullying. Therefore, accurately and efficiently verifying whether news spread on social networks is rumor or truth has become an increasingly important yet challenging problem. While previous works usually identified rumors by their contents, in this paper, we propose a novel \textit{user behavior model} to capture the users' judgment/discriminative abilities, which is learned via our well-designed evidence-aware probability reconstructor. We develop a general framework for tackling \textit{rumor assertion over large-scale social networks} (RA-SN), which can efficiently and effectively determine whether a given news item is a rumor with high confidence, based on users' behaviors, in a scalable manner. Extensive experiments demonstrate the efficiency and effectiveness of our proposed RA-SN approach on both real-world and synthetic graphs.

0.07

Can Multimodal Large Language Models Generate and Detect Multimodal Social Media Fake News?

cs.CL cs.AI ·2026-09-20 ·Jiyao Yang, Yang Liu, Zhenyue Qin, Qingyu Chen, Xiuzhen Zhang
top venuebenchmarkcode/data similarity 0.64
Abstract · 15 pages, 6 figures. Accepted for publication in the Findings of EMNLP 2026

The rapid advancement of generative AI raises concerns about the misuse of Multimodal LLMs (MLLMs) for large-scale disinformation campaigns on social media. Despite existing research on textual disinformation, a fundamental question remains unanswered: can MLLMs be exploited to fabricate realistic multimodal fake news, and can they reliably detect it? We introduce a multi-agent framework in which a story agent, an image agent, and a critic agent collaborate to produce fake social media posts that plausibly counter true news. We apply the framework to generate over 9,000 paired multimodal news posts across science, health, and entertainment domains, and benchmark 16 open- and closed-source MLLMs for automated detection. We find that most models fall substantially short of human-level accuracy and fail critically on identifying image authenticity. Our research provides a foundation for developing robust defenses against social media fake news. Code and data are available at https: //github.com/xiuzhenzhang/Multimodal.

0.42

RAEGNet: Relation-Aware Evidence Graph Network for Harm-Aware Multimodal Fake News Detection

cs.CL cs.CVcs.MM ·2026-09-29 ·Wenbin Shen, Guoxuan Qin, Guangxu Yao, Baodong Wang, Yuanbo Rui +2 more
similarity 0.63
Abstract

Existing multimodal fake news detection methods often introduce external information to assist detection. However, most of them rely on entity-level retrieval and are therefore prone to introducing event-irrelevant noise. Meanwhile, existing methods mainly focus on improving overall performance and do not account for differences in the degree of harm posed by different instances of fake news. To address these limitations, we design an Event-Level Evidence Retrieval Framework (ELERF) and propose a Relation-Aware Evidence Graph Network (RAEGNet). ELERF retrieves external evidence based on the complete event semantics of a news item. RAEGNet constructs a directed graph that incorporates news-evidence stance relations and evidence-evidence interaction relations, and introduces a conditional-harm branch to jointly model authenticity and potential harm. Experimental results demonstrate that RAEGNet outperforms multiple baseline methods across all evaluated metrics on Weibo-21, Fakeddit, and our self-constructed SSS dataset.

0.17

DisasterAgent: An Evidence-Driven Agent for Verifiable Disaster Report Generation From Remote Sensing Imagery

cs.CV cs.AI ·2026-11-10 ·Zhijian Li, Jiawei Song, Chao Ren, Yidan Zhang, Peijin Wang +2 more
code/data similarity 0.62
Abstract

Remote sensing disaster assessment is moving from simple damage classification toward comprehensive report-level analysis. In this setting, responders need not only damage descriptions but also verifiable evidence to guide response and recovery. Recent vision-language models (VLMs) and tool-augmented agents offer strong multimodal reasoning and perception, but they struggle to provide the evidence required for high-stakes verification. To address these limitations, we propose DisasterAgent, an evidence-driven agent that uses a Perception-Augmentation-Generation workflow to generate verifiable disaster reports from bitemporal optical imagery. DisasterAgent first invokes a specialized vision expert toolkit in the perception phase to build a Spatiotemporal Perception Graph (STPG), serving as a structured memory to store and organize disaster-induced changes across space and time. Evidence-based Augmentation (EA) then extracts evidence from the STPG, attaching inline citations that form a closed evidence loop from each statement to the supporting visual evidence. Extensive experiments on the DisasterM3 and RSCC benchmarks demonstrate that DisasterAgent consistently outperforms general-domain VLMs, remote sensing-specialized models, and existing remote sensing agents. Notably, compared with the original model, it improves the average score for disaster report generation on DisasterM3 by 13.5\% and ST5-SCS on RSCC by 15.34\%, providing a more accurate and verifiable solution for disaster emergency response. The code will be available at https://github.com/ZhijianLi2003/DisasterAgent.

0.38

Verifiable Disaster Storylines and Causal Knowledge Graphs: A Citation-Grounded Pipeline from Heterogeneous Humanitarian Sources

cs.AI cs.CL ·2026-09-01 ·Ivan Decostanzi, Michele Ronco, Sergio Consoli, Christina Corbane, Lorenzo Bertolini +6 more
publishedhot direction similarity 0.62
Abstract

Effective humanitarian response depends on the rapid synthesis of heterogeneous, high-volume information sources - a task that routinely exceeds human analytical capacity in the critical early hours of a crisis. We present a pipeline that combines structured disaster records from EM-DAT with unstructured documents from ReliefWeb and the European Media Monitor (EMM) to produce source-grounded disaster storylines and causal knowledge graphs supporting situational awareness for responders and analysts. Using Retrieval-Augmented Generation, the pipeline extracts structured storylines - tabular event profiles covering 17 fields, from severity and key drivers to child-sensitive impact indicators - and constructs causal knowledge graphs where each node and edge is enriched with citation-grounded explanatory narratives, enabling full traceability back to primary sources. We evaluate the system on three diverse crisis use cases through a human evaluation involving 9 domain expert and 9 non-expert evaluators. Results confirm high retrieval precision, strong faithfulness of extracted causal relations, and a clear expert preference for citation-grounded components over ungrounded alternatives. The pipeline is designed to scale to the full EM-DAT catalogue, with the goal of publicly releasing a new collection of disaster stories complementing the EM-DAT database.

0.49