CS Paper Radar

Using LLM Agents to Evaluate Prosocial Interventions on Social Media Platforms

Lucen Li, Erin Wertz, Muhammad Haroon, Anshuman Chhabra, Magdalena Wojcieszak

cs.SI cs.CL ·submitted 2026-11-01 01:24 UTC · v1

Abstract

Social media platforms are a primary source of information for many users. Although research explores various interventions to promote prosocial content or demote harmful content on platforms, there are few systematic methods for evaluating competing intervention designs before conducting costly human-subject experiments. We develop a simulation-based evaluation pipeline built on the OASIS framework. We generate 13,835 LLM-based agents, each with a unique profile. Over 20 time steps, we expose them to real-world Twitter/X posts and let them comment, like, repost, or follow. We use those agents to systematically evaluate three interventions that increase exposure to verified news: adding content, upranking content, and celebrity source cues. All interventions significantly increase news engagement, with upranking news to prominent feed positions producing the largest effects. These effects hold across agents' gender, age, and education levels. We validate our injection treatment results against a comparable human-subject study and find strong directional alignment and similar dose-normalized efficiency. Our results suggest that LLM-based agent simulations can support early-stage evaluation of platform interventions before human-subject deployment, and upranking may be more effective than news volume or source.

Closest papers

Nearest neighbours by meaning across the whole index.

"If It Looks Like a User": Measuring Real-Time Moderation Effects via Social Media Simulation

cs.SI cs.CYcs.MA ·2026-08-17 ·Enrico Verdolotti, Gianluca Nogara, Luca Luceri, Silvia Giordano
publishedtheory similarity 0.68
Abstract · 15 pages, 4 figures, 2 tables. Accepted for presentation at the Social Simulation Conference (SSC) 2026

Agent-based social media simulators offer a controlled environment to study content moderation, yet their value hinges on how faithfully they reproduce real platform dynamics. We develop a calibrated extension of SimSoM, an agent-based model of information diffusion on social networks, grounded in a real-world dataset of online vaccine discourse during the COVID-19 pandemic. Our approach replaces ad-hoc parametrisations with empirically fitted distributions, optimised via CMA-ES (Covariance Matrix Adaptation Evolution Strategy) and validated against real data across temporal, distributional, and structural dimensions. Using this validated simulator, we provide three key contributions. First, we show that the calibrated model reproduces key statistical signatures of the empirical data, including activity distributions, post/reshare ratios, and temporal patterns. Second, we apply established misinformation-spreader detection and prevention methods to both empirical and simulated data, progressively removing top-ranked users and showing that the resulting decline in low-quality content is consistent across the two. Third, comparing static (retroactive) and dynamic (in-simulation) moderation across 30 network realisations, we show that static evaluation significantly overestimates the effectiveness of user bans for the most effective detectors: when moderation is applied in real time, compensatory resharing by the remaining users dampens the expected reduction in low-quality content, so static estimates should be read as an upper bound. These findings highlight the necessity of simulation-based evaluation for content moderation policies and contribute a reusable, empirically grounded simulation framework.

0.26

Counterfactual assessment of social media interventions by separating exposure from engagement

cs.CY physics.soc-phcs.SI ·2026-10-27 ·Tommaso Pisani, Pietro Gravino, Giulio Prevedello
similarity 0.68
Abstract

Social media interventions are usually evaluated in individual-level experiments, which are costly to scale and often miss collective effects. We propose the Exposure--Engagement model, whose parameters separately control what social media users are exposed to (recommendation) and how they engage (behaviour). After fitting it to the Italian Twitter debate (2018--2022), we validate it at individual and collective levels. We simulate behavioural interventions (prebunking and accuracy prompts), calibrated on effect sizes, and a recommender-system intervention (chronological feed), comparing effects on engagement, attention diversity, and segregation. All three reduce engagement. Behavioural interventions slightly reduce diversity, leaving segregation unchanged, whereas the chronological feed increases diversity and reduces segregation. For these interventions, changing recommendation does more for the debate's structure than changing behaviour. We design a recommender-system intervention that increases engagement and diversity while reducing segregation, although with about a tenth of the chronological feed's gains, showing the framework's potential for pro-social intervention design.

0.34

Illusory Truth or Mere Exposure? Model-Dependent Repetition Effects in LLM-Based Social Media Simulations

cs.AI cs.SI ·2026-09-28 ·Azza Bouleimen, Nicolò Pagan, Anikó Hannák
published similarity 0.67
Abstract · Accepted at COLM 2026

Generative agent-based models (GABMs) are increasingly used to simulate social media dynamics, including misinformation spread. For such social simulations to be valid proxies of human behavior, LLM agents should replicate established human cognitive biases, among them the Illusory Truth Effect (ITE), where repeated exposure to a claim increases its perceived truth value. We investigate whether and how the ITE manifests across four LLMs (Gemma-3-4b-it, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and GPT-5-nano) in a social media simulation context. We propose a two-phase within-context experimental design that embeds the repetition manipulation inside a realistic news feed interaction. Using this design, we collect 336,000 truth, importance, sentiment, and interest ratings across 100 statements, 10 feed variants, and 3 replications. The key comparison is between ratings assigned to repeated statements, seen throughout a simulation phase, and completely unseen ones, rated within the same experimental context window. We distinguish genuine ITE (truth-specific repetition boost) from mere exposure effects. We run an OLS regression followed by a Linear Mixed-Effect Model to account for differences across models and ratings. Our results reveal four qualitatively distinct patterns: Gemma-3 exhibits a genuine ITE; Qwen2.5 shows a mere exposure effect; GPT-5-nano displays no repetition effect on truth and mild skepticism toward repeated content; Llama-3.1 shows a small truth boost alongside decreases in evaluative dimensions. Crucially, temperature has no effect on these findings, and a variance decomposition highlights the high context-sensitivity of LLM rating behavior. Our findings caution against assuming uniform ITE replication across LLMs in social simulations, while suggesting that Gemma-3-4b-it may offer the most behaviorally realistic approximation for misinformation-related simulations.

0.48

No Note Is an Island: Probing LLMs' Understanding of Crowdsourced Fact-Checking Environments

cs.SI ·2026-11-01 ·Weijian Fan, Chichen Lin, Xiao Han, Jiaying Wu
similarity 0.64
Abstract

Large language models (LLMs) increasingly assist crowdsourced fact-checking, yet evaluations focus on individual annotations. A correction's reach and reception depend on visibility, audience responses, and platform policies. We introduce correction context understanding: assessing corrective content, interpreting audience responses, and reasoning about platform dynamics within a shared environment. We build NoteHabitat from 36 months of Weibo Annotations data, linking 102,994 posts (42,853 annotations), 431,407 comments, and five policy events. Evaluating diverse model families across seven tasks spanning content, interaction, and management reveals three gaps. Annotation quality correlates only weakly with engagement ($ρ=0.068$), while four zero-shot LLMs systematically underrate quality relative to human judgments, especially evidence support and objectivity. Models also miss toxic comments supporting corrections (36.4% average recall across ten models). At the platform level, LLMs partially predict policy-associated outcome directions, but LLM-guided simulations poorly reproduce daily activity. Cross-group analyses expose blind spots tied to contribution types and reader stances that aggregate scores conceal. Reliable fact-checking AI must understand the note, the conversation around it, and the platform in which both unfold.

0.25

From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencers

cs.CL cs.AI ·2026-09-18 ·Ji-Lun Peng, Yi-Zhen Zhang, Chun-Nan Chou, Yun-Nung Chen
top venue similarity 0.64
Abstract · Accepted by EMNLP 2026 Findings

Large language models have shown strong potential as role-playing agents for real individuals, yet faithful impersonating remains challenging. Existing in-context learning-based methods fail to capture how individuals react under different situations. In addition, LLM-based evaluation is difficult for obscure individuals. To address these challenges, we propose Situation--Internal state--Behavior Persona method to incorporate situation-dependent behavioral strategies. We further design an evaluation protocol that provides LLM evaluators with references about the impersonated individual. We evaluate our approach on a newly constructed dataset for the task of generating replies on social media. Experimental results show that our proposed method outperforms state-of-the-art ICL-based baselines, while our evaluation protocol achieves moderate correlation with human judgment. Besides, experiments on fictional-character benchmarks demonstrate that our proposed method is applicable beyond the social media setting. These findings suggest that incorporating behavioral information broadly improves the fidelity of role-playing for real individuals on social media or fictional characters.

0.35

LLM-Augmented Agent-Based Simulation of Cyber Social Agents

cs.MA cs.AIcs.SI ·2026-08-22 ·Lynnette Hui Xian Ng, Kathleen M. Carley
similarity 0.63
Abstract

Social media bots are often portrayed as malicious automations to be detected and removed, but automation on social media platforms encompasses a wide spectrum of roles, behaviors, and rhetorical strategies that can shape online discourse in both constructive and destructive ways. This work conceptualizes these automated actors as \textbf{Cyber Social Agents} through a taxonomy of archetypes characterized by their operational tactics and rhetorical style. We integrate this taxonomy into an LLM-driven social simulation, where agents communicate and coordinate in a network environment. Using empirically grounded CSA personas, we simulate heterogeneous online societies to study how agent diversity shapes information diffusion and social influence. Our approach bridges the gap between rule-based agent models and LLM-powered simulations with generated agents that are parameterized from empirical data.

0.08

LLM Recommenders Produce Demographic Inequality Through Content Preferences

cs.CY cs.SI ·2026-10-30 ·Nicolò Pagan, Petter Törnberg, Christopher A. Bail, Christopher Barrie
similarity 0.63
Abstract

Large language models (LLMs) are increasingly being integrated into systems that rank and recommend online content. We examine how this shift can produce demographic inequality in whose voices are surfaced. Using 10,000 Twitter/X posts written by 1,000 survey respondents and linked to verified self-reported characteristics across nine demographic attributes, we conduct a controlled recommendation experiment across three LLM providers, six recommendation objectives, and five levels of contextual information. By varying whether models receive post text alone, platform metadata, or explicit author demographics, we isolate three pathways to unequal recommendation: preferences for demographically patterned content, reliance on unequal engagement signals, and direct use of disclosed identity. We find that demographic inequality emerges from text alone. Models consistently favour longer, more negative, and more opinionated posts, particularly those concerning news and social issues, producing systematic differences in the demographic composition of recommended authors. Platform metadata generally amplifies these inequalities, while explicitly disclosing authors' demographic characteristics produces little additional change. Recommendation objectives mainly affect the magnitude of these patterns: controversial framing intensifies both content and demographic skew, whereas popularity-oriented framing dampens it, although for some attributes - most notably age - the objective can also reverse which group is favoured. LLM-based recommenders can therefore reproduce demographic inequality by proxy, not through explicit demographic targeting, but through preferences for content and engagement signals already structured by social inequality.

0.24

Predicting Engagement with Sponsored Content Across Account Types on Instagram

cs.SI ·2026-10-05 ·Pedro Victor de Sousa Lima, Olga Goussevskaia
similarity 0.63
Abstract

In this work we explore how users interact with sponsored content on social media platforms by collecting and analyzing a large-scale dataset of sponsored Instagram posts. To maintain transparency, we favor robust statistical analysis and explainable models over deep learning techniques. Our pipeline categorizes Instagram accounts along multiple dimensions, including audience size and entity type. We complement this with semantic features extracted from post captions and hashtags, and use these elements to train regression models that forecast engagement. We validate our approach on a dataset comprising over 15M Instagram posts authored by over 700K accounts featuring sponsored content. Our analysis shows that per-post engagement is partly predictable and highly optimized when accounts are segmented by entity type or audience tier. Our models achieve competitive predictive power, while remaining fully transparent about which features drive engagement.

0.22