CS Paper Radar

Discussing politics with a chatbot: How political beliefs shape prompt formulation for generative AI

Victoria Vziatysheva, Reinhold Kliegl

cs.HC cs.AI ·submitted 2026-10-31 17:42 UTC · v1

Abstract

By generating comprehensive and highly personalized answers to user prompts, AI chatbots such as ChatGPT and Gemini can serve as alternatives to search engines and substantially transform online information seeking. However, little research has examined how users interact with these tools to find information, particularly about politics. In this experiment, we examine how Swiss citizens (N = 563) formulate prompts to search for migration-related issues using AI chatbots and whether this process is affected by selective exposure. We find evidence of attitude-consistent prompting, whereby participants who were more critical of migration were more likely to formulate prompts with an anti-migration stance. This effect was stronger for a broad topic (migration in general) than for a specific one (anti-migration popular initiative put to a vote in Switzerland). However, exposure to an attitude-consistent AI response did not further amplify selectivity in prompting. We also did not find evidence that either AI attitudes or thinking styles moderated selectivity in prompting. These findings contribute to the understanding of selective exposure via AI chatbots and the use of generative AI for political information-seeking.

Closest papers

Nearest neighbours by meaning across the whole index.

Templated or fully synthetic? Prompt construction as a confound in measuring LLM political stance beyond writing assistance

cs.CL cs.CY ·2026-08-11 ·Ilias Chalkidis
position similarity 0.68
Abstract

Political stance detection in LLMs has long been dominated by closed-ended, multiple-choice political survey questions---originally designed for humans, and thus lacks the realism and nuance of human-AI interactions in the wild, while also being susceptible to sandbagging. The recent IssueBench framework substantially mitigates these limitations with templated prompts anchored in real-world chat logs. Given the rise in non-work-related use of GenAI assistants, we extend IssueBench beyond writing assistance to include two additional tasks, information seeking and opinion sharing. We argue that templated prompts still lack the nuance of real ones, especially for open-ended tasks, and remain recognisable as evaluation artefacts. We propose the use of fully synthetic (LLM-generated) prompts, produced under detailed instructions with real prompts as seeds. We assess the ecological validity of real, templated, and LLM-generated prompts in a small-scale study covering 3 highly contested policy issues and 3 recent geopolitical conflicts. Human and LLM annotators rank LLM-generated prompts as no less realistic than real ones and clearly more realistic than templated ones, and find that they carry their intended intent and stance more clearly; the LLMs separate templated prompts from the other two far more sharply than the humans do. In a case study, templated and LLM-generated prompts yield systematically different stance estimates for the same model, most visibly under neutral framings, where templated prompts overstate the model's leaning in the direction encoded by the topic-and-stance text (filler) slotted into their templates.

0.04

A light-touch AI literacy intervention helps protect against AI political persuasion

cs.HC ·2026-09-14 ·Reed Orchinik, David Rand
similarity 0.67
Abstract

Conversations with large language models (LLMs) can substantially shift beliefs and attitudes, raising concerns about manipulation using AI persuasion. Here we test whether a light-touch AI literacy intervention - a brief warning that LLMs can be prompted to persuade and may present information selectively - helps protect users. Across two experiments (total N = 3,208 Americans) in which participants conversed with an LLM instructed to shift their views about different political topics, the presence of a warning reduced belief change by roughly one-half (-48.1%, 95% CI [-59.5%, -36.8%]) relative to the control. Importantly, the warning did not significantly reduce trust in generative AI more broadly. Light-touch literacy interventions can help protect users against AI political persuasion.

0.24

Issue Bias in Generative AI Writing Assistance: Political Issues and LLMs in the Swedish 2026 Election

cs.AI cs.CYstat.AP ·2026-09-14 ·Bastiaan Bruinsma, Annika Fredén, Paul Röttger, Moa Johansson, Asad Sayeed
similarity 0.66
Abstract

Generative AI writing assistants and the Large Language Models (LLMs) that power them are increasingly part of how voters gather information before elections. With growing evidence that they influence users' opinions, it is increasingly important to understand the views and positions of these tools. To better understand these views, we examine the stances supplied by six LLMs on a variety of Swedish-language writing tasks ahead of the 2026 Swedish parliamentary election. We cross 107 policy propositions with 77 writing templates and neutral, positive, and negative prompt framings, producing 24,717 prompts per model and 148,302 responses. To study these, we look at the models' default stance tendencies, compare how they respond to similar issues, and compare their responses with those of each of Sweden's eight parliamentary parties on the same issue. We find that Claude, DeepSeek, Gemini, and Mistral have similar profiles; ChatGPT more often supplies neutral or ambivalent text; and Grok differs most on topics such as migration, crime, and gender. When comparing the political parties, we find that the Social Democrats are closest to all six models. Still, after correcting for multiple comparisons, none of the within-model differences in party distances remains significant. Overall, we find that no model has a clear preference, nor a clear preference for a party, but that this depends on the specific issue or task the user asks about.

0.21

Investigating the Impacts of Generative AI on Information Seeking

cs.HC cs.AIcs.CY ·2026-09-13 ·Alexi Orchard, Shannon Lodoen
published similarity 0.65
Abstract · 4 pages, accepted to CSCW Workshops 2026

This paper is an encore submission of our 2026 journal article "Expertise and Information Seeking in the Age of Generative AI: New Procedures, New Problematics" with an extended discussion for the CSCW 2026 "Broader Impacts of GenAI in Communication" Workshop on October 10, 2026. In the original article, we employ procedural rhetoric to analyze how generative AI chatbots leverage natural language signifiers of expertise and intelligence to influence users' perception of their trustworthiness. In this submission, we extend our conversation in the CSCW community with the goal of cultivating a cross-disciplinary vocabulary for describing, analyzing, and mitigating the risks posed by the integration of generative AI into human communication practices. It is important to develop an understanding of how the procedures surrounding information-seeking practices are informed by users' values, experiences, and expectations - and how these procedures might in future be altered by the emerging turn toward AI "experts" and authority.

0.28

User Attributes and Political Ideology Shape LLM Receptiveness in Online Political Debates

cs.AI cs.CL ·2026-11-02 ·Akram Elbouanani, Yousra Taleb, Gianluca Demartini, Adrian Popescu
similarity 0.65
Abstract · Preprint. Submitted for conference review

Online political discussions increasingly involve AI-mediated participation, as users invoke LLMs to assess contested claims, arbitrate disputes, and respond to opposing viewpoints. As these systems become participants in online exchanges, they may alter which speakers and argumentative strategies receive validation within the discussion. We study this interactional dimension by reconstructing 5,376 Debate.org debates as LLM conversations and running nearly 150,000 controlled multi-turn conversations that vary user political orientation, expertise, rhetorical strategy, and defended position, including conversations with competing users. Across naturalistic and controlled debates, we find that LLMs are systematically more receptive to left-leaning than right-leaning users. The asymmetry persists when users with different political orientations defend the same position and when opposing users appear in the same conversation, indicating that it is not fully explained by stance agreement or simple accommodation to the current speaker. Repeated interaction increases receptiveness without eliminating this ordering; expertise also matters, while rhetorical differences are smaller. The pattern appears across multiple models, but is not fixed: fine-tuning on different political corpora can largely erase or reverse it. These results show that LLMs can respond systematically differently to participants in the same political debate, making such interactional asymmetries important for understanding their emerging role in online public discourse.

0.20

When Should I Trust This? Designing and Evaluating Factuality Indicators for LLM-based Chatbots

cs.HC ·2026-10-23 ·Greta Warren, Christine M. Lehane, Jessica Heal, Sila Eser, Denise M. Leto +1 more
survey similarity 0.62
Abstract

Generative AI systems increasingly mediate how people seek information, yet communicating AI factuality to laypeople remains challenging. Through a mixed-methods investigation involving interviews (N=8), a survey (N=1,000), and an experiment (N=5,151), we examine how people evaluate AI-generated information for gardening, mathematical, and medical queries, and how factuality indicators (confidence and sources) shape perceptions. Our interviews and survey show people selectively verify AI responses using external and AI-provided evidence, moderated by contextual factors. Although confidence-indicators affected trust and accuracy perceptions, people hold diverse, ambiguous interpretations of AI confidence. Sources were viewed as crucial, yet users raised concerns about source reliability and verification effort. Our experiment showed that confidence and source-based indicators influenced initial perceptions, but effects disappeared once participants learned the response's correctness. Medical queries elicited skepticism regardless of indicator. Findings motivate helping users interpret model confidence and integrating source information into the interface to support calibrated trust in AI-supported information-seeking.

0.07

From Caricature to Collective Artifact: How Persona Construction Drives Polarization in Generative Social Simulations

cs.SI cs.AI ·2026-10-12 ·Dimitris Michailidis, Petter Törnberg, Fernando P. Santos
position similarity 0.61
Abstract · 17 pages

Generative Social Simulation (GSS) leverages Large Language Models (LLMs) to create sophisticated personas that define agents with demographics, emotions, and political opinions. While prior work shows that personas compress outputs into stereotypical caricatures, there is no comprehensive study of how these caricatures evolve into collective artifacts when they interact. Using a Twitter-like GSS platform with personas derived from the American National Elections Study (ANES), we ablate three layers of partisan identity: affective polarization markers, party identification, and vote choice, while retaining demographics, ideological positioning and important issues. We find that collective outcomes depend heavily on how survey responses are translated into personas. Removing a few explicitly partisan statements moves the network from near-complete segregation to cross-party ties approaching those of real Twitter networks. We argue that matching one outcome is not evidence for behavioral validity and point to salience as a possible explanation. Surveys ask for a position on every question regardless of its importance to the respondent, and persona construction turns each answer into a direct behavioral instruction. We conclude with a checklist for more comprehensive evaluation of personas in future works.

0.20

Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments

cs.CY cs.CL ·2026-08-30 ·Lin Chen, Yitong Chen, Yong Li
top venuecode/data similarity 0.60
Abstract

Large language models (LLMs) are increasingly deployed as proxies for human participants in social simulations, yet whether they update their beliefs in response to persuasive arguments, as humans do, remains poorly understood. We conduct a systematic comparison using a naturally occurring online persuasion corpus in which original posters explicitly verify whether a reply changed their view. Our results show that LLMs achieve only slight agreement with humans (Cohen's kappa ranging from 0.079 to 0.178). Content-level analyses show that humans and LLMs agree on the strongest persuasion cues but diverge on finer ones: humans are more swayed by novel content and assertive language, whereas LLMs favor topical similarity and surface-level formatting. At the level of persuasion strategy, LLMs underweight emotional appeals and overweight credibility signals relative to humans, while the type of proposition under debate exerts no measurable effect on the degree of divergence. Furthermore, switching from first-person role-playing to third-person observation shifts all models toward greater resistance to persuasion, with the effect varying across persuasion strategies and textual features. These findings highlight the risk of treating LLM judgments as faithful proxies for human belief updating and point to structural differences in how LLMs and humans process persuasive discourse. Our code is available at https://github.com/tsinghua-fib-lab/LLM-belief-update-cmv.

0.51