CS Paper Radar

Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models

Indira Sen, Georg Ahnert, Leah von der Heyde, Jana Lasser, Bernd Weiß, Markus Strohmaier

cs.CY ·submitted 2026-09-09 14:59 UTC · v1

Abstract

Large Language models (LLMs), having been trained on vast amounts of human-generated data, may encode the attitudes and behaviors of these humans. As such, LLMs show promise in mimicking human-like patterns that facilitate their use in simulating people in a wide variety of contexts. One such context is using LLMs as 'silicon samples', i.e., proxies of people in answering survey questions to establish public opinion, design policies, or use as (social) scientific data. However, several critical questions of social biases, generalization, and technical limitations remain, further complicated by a vast design space open to simulation designers. Multiverse analyses might help us make sense of the impact of different design choices, however, we lack a systematic understanding of the design space of LLM-generated surveys as well as how these decisions interplay with inherent LLM limitations. Therefore, how do we systematically identify, trace, and document limitations in LLM-generated survey responses? Building on traditions in the quantitative social sciences, specifically survey methodology and measurement theory, we investigate threats to the validity of LLM-generated survey responses. To do so, we design a framework that enumerates conceptual errors and systematic biases that can occur at different stages of the survey simulation lifecycle. Our framework, called the Total Simulated Survey Error (TS2E) Framework, provides a unified and end-to-end perspective on LLM-generated survey data. The framework, illustrated through a theoretical and empirical case study, enables survey simulation designers to systematically identify and reflect on errors in LLM-generated surveys.

Comment
Preprint

Closest papers

Nearest neighbours by meaning across the whole index.

Data-Driven Personas for Survey Simulation: Insights into Simulation Alignment Across Data-Access Regimes

cs.AI ·2026-10-05 ·Dongryeol Lee, Weronika Łajewska, Leonardo Perelli, Saab Mansour
top venue similarity 0.73
Abstract · Work accepted at REALM EMNLP 2026

Large language models (LLMs) offer new opportunities for public opinion research by enabling early prediction of survey responses, potentially reducing the cost and time of traditional surveys. However, many existing steering approaches rely on target-domain human data for fine-tuning or prompting that is costly to collect and raises privacy concerns. In this paper, we study demographic group-level survey simulation, where personas induced from heterogeneous, anonymized public behavioral data condition agents that simulate responses of individuals from specific demographic groups. We examine whether representative personas can be induced from diverse sources and analyze how the domain, scale, and granularity of the source data affect survey simulation alignment. We find that personas induced from out-of-domain sources rarely outperform simulations conditioned only on basic demographic information, largely due to population mismatch. However, when personas are accurately assigned to the target demographic groups, alignment improves substantially. Finally, personas induced from target-domain survey data generalize better as more survey question history becomes available, suggesting that richer behavioral evidence enables more stable persona trait inference that transfers to better unseen questions simulation alignment.

0.42

Efficiency Gains in Social Science Research: A Prospective Silicon Sampling Protocol for Pre-Treatment and Code Optimization

cs.CY ·2026-11-13 ·Noaïm Astitou, Etienne Aucouturier
survey similarity 0.70
Abstract · 14 pages

This study evaluates the methodological potential of using large language models (LLMs) to generate synthetic datasets prior to real-world data collection, thereby streamlining subsequent processing and statistical analysis. The approach is applied to a survey conducted within the France Life Imaging (FLI) network, leveraging three distinct LLMs (Sonnet 4.6, GPT 5.3, and Kimi 2.6) to simulate responses and prepare data processing workflows. The findings demonstrate significant time savings in code development (85% to 90% reuse rate) and highlight the conditional validity of this preprocessing method, which depends on the thematic predictability of the models. While synthetic data effectively captures collective perceptions, it struggles with specialized or contextualized practices, reflecting limitations documented in the silicon sampling literature, such as homogenization and reduced variance. This work extends the Phase 0 concept (Trojacka et al. 2025) from instrument validation to a full-fledged methodological preprocessing step, paving the way for scalable applications in survey research.

0.29

Silicon Samples Match the Answers but Not the Respondents

cs.CL cs.CY ·2026-10-22 ·Yusuke Kumagae, Masahiro Kaneko
survey similarity 0.69
Abstract

Silicon sampling generates survey respondents by having a large language model (LLM) answer a survey as a person with given demographic attributes, and the resulting synthetic populations are increasingly used in place of human ones. Existing evaluations focus on individual-answer accuracy, per-question answer distributions, and correlations across questions. None of these compares the respondents themselves, so a generator whose respondents have combinations of answers found in no human respondent can still score well if it reproduces every answer distribution. We propose the respondent coverage gap (RCG), which represents each respondent as a point defined by their complete answer profile and measures how far each human respondent is from the nearest generated one, relative to the distance between human samples. On two US social surveys, all six LLMs we evaluate leave human respondents uncovered, and under our main settings, they fall behind a baseline that samples each answer independently from the human marginals. Across the prompting and inference settings we test, every LLM has a higher RCG than a Gaussian copula fitted to the human data.

0.27

From Simulated Citizens to Simulated Deliberation: Challenges in Representation and Interaction

cs.AI ·2026-09-07 ·Chaemin Jang, Junsik Min, Jaewoo Choi, Donggyu Lee, Haiin Lee +7 more
similarity 0.69
Abstract

Can LLM agents simulate how a real public would deliberate? For such simulations to be meaningful, agents must reflect the population's opinions, and their views must be shaped by what others say during discussion. We evaluate these requirements across six LLMs using national surveys, controlled deliberation experiments, and a real deliberative poll. Comparing census-grounded Korean personas against national surveys, we find that agents often produce overly concentrated opinions and fail to reliably reproduce demographic differences. During simulated deliberation, agents exchange arguments, respond to one another, and frequently change positions. Yet similar changes occur even when agents cannot see one another's contributions, while simply restating their starting positions sharply reduces movement. We further test whether these limitations persist in a real-world setting, using agents constructed from actual respondents in the 2017 Shin-Kori deliberative poll. Even with real respondent profiles, agents fail to reproduce the panel's initial opinion distribution or how its opinions change across survey waves. Our findings show that generating deliberative discussions, representing public opinion, and reproducing interaction-driven opinion change are distinct capabilities. These results suggest a different potential role for LLM deliberation: surfacing arguments and perspectives, rather than simulating a real public.

0.19

Silicon sampling answers with country-level assumptions, not individual attitudes: Cross-national evidence from the European Social Survey

cs.CY ·2026-09-14 ·Chuyao Wang
code/data similarity 0.67
Abstract · 49 pages, 11 figures, 6 tables, including appendix. Replication materials: https://github.com/chuyao-wang/silicon-replication

Silicon sampling uses large language models (LLMs) to simulate survey respondents. Whether it recovers cross-national variation, and why, remains unresolved. This study evaluates it against European Social Survey Round 11 (30 countries, 42 items) with two open-weight LLMs under first- and third-person prompts, plus backstory and response-format experiments. Aggregate recovery is moderate and uneven across items. Adding the country name to a three-variable demographic backstory raises the median per-item correlation between simulated and observed country means from -0.03 to 0.52, and the richer profiles tested add no consistent gain. The respondent's country label acts as a country-level assumption that respondent detail does not revise. Naming the response-scale endpoints in words stops the model from ranking countries backwards, so the answer format sets the direction of the ranking. Individual-level recovery remains negligible in every condition and does not track aggregate recovery across countries. An average of neighboring countries, using no LLM, recovers country levels more accurately than every model condition and ranks them about as well. Silicon sampling can thus support exploratory country-ranking comparison after item-level validation and with the response format reported. It does not support individual or distributional inference.

0.47

Measurement Validity in LLM Cultural Alignment

cs.AI physics.soc-ph ·2026-08-29 ·An Duy Nguyen, Muhammad Aurangzeb Ahmad
similarity 0.65
Abstract

Researchers increasingly treat LLM survey responses as a proxy for human cultural values. This includes projecting model outputs onto instruments like the Inglehart-Welzel Cultural Map and drawing conclusions about which cultures a model resembles. While a model's answer to a value-laden questions may be interpreted as a cultural signal, it also carries sampling noise and, can be quite sensitive to question framing. In this paper, we separate survey responses, sampling noise and question framing for multiple LLMs. We decompose response variance from these models into variation across random seeds, prompt rewordings. We employ noise-to-signal ratio (NSR) to test whether a model's apparent cultural position is distinguishable from noise. When applied across a dozen models from four geographic origins, calibrated against 88 Integrated Values Survey countries, the answer is often no. NSR exceeds 1.0 on 49 of 117 valid model-question pairs (42%), reaching 5.56 in the worst case. Two models even refuse to answer sufficient number of survey questions outright. Our results corroborate previous findings that LLMs cluster toward Western, English-speaking cultural positions. However, what does not hold up in this study is the precision with which anyone can currently interpret a specific model's coordinates: prompt tone alone can shift a model by 2.4 map units, comparable to the distance between actual countries in the Inglehart-Welzel Cultural Map. These findings suggest that cultural attribution from LLM survey responses requires establishing the reliability of the underlying measurements before interpreting model coordinates as evidence of cultural representation.

0.35

Hybrid Panels: Toward Human-AI Collaboration in Survey Research

cs.CL cs.AIcs.CYcs.HC ·2026-08-23 ·Julia Romberg, Tobias Gummer, Gabriella Lapesa, Tanja Kunz, Claudia Wagner
similarity 0.64
Abstract · 15 pages, under review

Large-scale population surveys are essential for generating robust social and scientific insights, yet they face significant challenges, including declining response rates, increasing data collection costs, long delays between data collection and data provision, and the risk of nonresponse bias. Advances in artificial intelligence (AI) have opened up new opportunities for AI-supported survey infrastructures where the goal is to overcome these challenges without limiting the data quality. A promising AI-enabled survey infrastructure for which we build a first pilot is a hybrid panel. A hybrid panel is a longitudinal AI-enabled survey which allows to iteratively improve the alignment between large language models (LLMs) and the population they aim to simulate and use the errors to inform the design and implementation of the next survey wave (e.g., inform the participant recruitment, assignment of questions to participants). It incorporates both human participants and LLMs as fundamental elements of its design. In this research note, we introduce the concept of a hybrid panel by providing a definition and outlining an overarching framework, spanning data collection to data validation. We detail results from a first pilot study to illustrate (open) challenges that we identify for hybrid panels.

0.31

Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases

cs.CL ·2026-08-11 ·Davood Wadi, Mohsen Ghodrat, Matthew Philp
similarity 0.64
Abstract

As Large Language Models (LLMs) are increasingly deployed as autonomous agents, accurately evaluating their latent values and biases is critical. The NLP community typically evaluates models using large, unstructured benchmarks. While effective for general capabilities, these datasets fundamentally conflate causal mechanisms: even when an aggregate bias is detected, unstructured evaluations cannot disentangle whether it stems from baseline traits, contextual confounders, or complex interactions. To address this, we introduce an analytically exact framework for the controlled behavioral evaluation of LLMs. We bridge human psychometrics with LLM mechanics by resolving gaps in design, measurement, and analysis. First, we replace unstructured prompting with fully crossed factorial experiments to systematically isolate causal main and interaction effects. Second, we eliminate Monte Carlo text sampling noise by operating directly on exact, token-level Probability Mass Functions (PMFs). Third, we derive a multivariate ordinal consensus metric and a distributional ANOVA to process these PMFs analytically. We validate our framework with a case study on consumer ethnocentrism across five LLMs, demonstrating how our approach isolates systemic country-of-origin biases that aggregate benchmarks otherwise obscure.

0.39