CS Paper Radar

AI-Moderated Interviews for Market Research and Digital Twins Calibration

Yuting Deng, Jingxuan Liu, Olivier Toubia, Naman Jain

cs.CY cs.AIcs.HCcs.MA ·submitted 2026-09-24 07:21 UTC · v2

Abstract

AI-moderated interviews are emerging as a scalable market-research method for generating consumer insights and building consumer "digital twins." Yet it remains unclear whether they match human-moderated interviews and whether they improve on simpler, static data collection methods. In a pre-registered study with three industry partners, the authors randomly assign participants to AI-moderated interviews (N = 139), static interviews (N = 154), or interviews conducted by professional human moderators (N = 24). AI moderation matches human moderation in depth, covers more themes, and, at equal budget, recovers more customer needs than either alternative, although participants sound more emotionally engaged with a human interviewer. The authors then build a digital twin of each participant and evaluate it against the participant's own held-out responses to six real-world marketing stimuli. Twins built from AI-moderated interviews predict consumer responses better than demographics-only personas, but no better than twins built from static interviews: the richer conversation does not necessarily yield more accurate predictions. Comparing the open-ended explanations provided by participants and their twins shows that prediction errors arise when twins reason differently than the people they represent and when questions fall far outside what the interview covered. AI moderation thus pays off most for discovering customer needs, and less for predicting customer responses.

Closest papers

Nearest neighbours by meaning across the whole index.

Synthetic Data in Marketing Research: How to Evaluate and When to Trust

cs.AI ·2026-09-12 ·Oded Netzer, Rajan Sambandam
position similarity 0.66
Abstract

Debate over synthetic data in marketing research has polarized between claims that large language models (LLMs) make human respondents obsolete and calls to avoid them entirely. We argue that both positions obscure the more useful question: not whether synthetic respondents work, but when. Building on Brand, Israeli, and Ngwe (2026), we make three contributions. First, we distinguish three types of synthetic data (ungrounded LLM responses, segment-level personas, and individual-level digital twins) and map each to the decisions it can support. Second, we develop a taxonomy of four families of accuracy measures and suggest that the wide range of reported twin accuracy, from near-perfect to near-chance, largely reflects differences in what is being measured rather than in method quality. Aggregate measures often perform well even when little information is supplied to the LLM, and can mask a complete absence of respondent-level differentiation. Third, we introduce the forgotten question problem, in which a question is omitted from a fielded study, as a setting for twin-based augmentation of existing data. We propose an ex-ante answerability diagnostic that requires no ground truth: the R^2 of a random forest predicting twin outputs from the data used to construct the twins. Across 108 attitude questions from a nationally representative survey (N = 3,063), screening at R^2 above 0.7 raises the mean twin-human individual-level correlation by 15% and reduces the share of poorly answered questions from 25.9% to 4.3%. Embedding similarity and experienced-researcher judgment provide correlated but weaker screens.

0.45

The Interviewer's Perspective: Unpacking the Impact of Real-Time AI Interviewing Assistance on Social Dynamics

cs.HC ·2026-09-24 ·Zhe Liu, Jiamin Dai, Joanna McGrenere
similarity 0.60
Abstract · 23 pages, 4 tables, 5 figures, under review for CHI 2027

Eliciting rich data in semi-structured interviews is cognitively demanding, prompting recent work to explore real-time AI assistance for interviewers. However, introducing AI into the interviewer-interviewee interaction creates a triadic context whose social dynamics remain underexplored. We investigated how interviewers experience AI assistance for probing during semi-structured interviews. To elicit rich participant reflections, we implemented two variants of AI assistance differing in initiation and granularity in a high-fidelity prototype, ProbeAssist. We conducted a qualitative-first comparative structured observation study where 18 participants each completed three simulated interviews: one without AI and two with different AI variants. Findings showed that participants leveraged AI as a supportive tool but resisted it as an assessor or competitor. As they navigated AI's benefits and interaction costs, tensions emerged around agency, ownership, creativity, and interpersonal communication. We propose three implications for AI-assisted human-to-human interaction: managing social pressure, balancing idea alignment with inspiration, and preserving interpersonal presence.

0.21

InterviewPlayground: A Validated Simulation Environment for Evaluating AI Interviewers

cs.AI cs.CL ·2026-10-08 ·Jonathan Ivey, Aimee Liang, Arthur Y. S. Wang, Madeline Mandell, Ziang Xiao +1 more
similarity 0.60
Abstract · Preprint. 23 pages

Increasingly, AI interviewers are being developed to elicit open-ended responses in applications like market research, public polling, preference elicitation, and social science research. However, evaluating AI interviewers is challenging because they function in extended, multi-turn interactions where they must adapt to participant behaviors. To address this need, we develop InterviewPlayground, a simulation environment for evaluating AI interviewers using simulated study participants whose behaviors are grounded in social theory. Simulated studies in InterviewPlayground produce an InterviewReportCard, which assesses the performance of AI interviewers using a suite of validated measures. To test whether our simulation-based evaluations predict performance with human participants, we conduct 15 real qualitative studies with five AI interviewers, three interview topics, and 450 human participants and compare them to simulated studies in InterviewPlayground. We find that AI interviewer performance in InterviewPlayground predicts performance in human studies with an average Pearson correlation of 0.86 across 12 measures, and the simulated interactions from InterviewPlayground reproduce key findings from behavioral analysis of AI interviewers in the human studies. Together, these findings support the validity of InterviewPlayground in assessing AI interviewer performance and examining potential failure modes. Our work contributes a simulation environment for AI interviewers supported with empirical validation, and more broadly, a roadmap for future work to develop validated, simulation-based evaluations of conversational AI systems.

0.43

When Can LLM Digital Twins Reduce Human Measurement? From Behavioral Fidelity to Statistical Substitutability

cs.AI stat.AP ·2026-09-07 ·Steven Wang, Kyle Hunt, Shaojie Tang, Kenneth Joseph
similarity 0.59
Abstract

LLM-based digital twins promise to reduce repeated human data collection by generating person- specific responses, yet existing evaluations provide little evidence about whether they can reduce human measurement while preserving valid inference. To address this, we introduce statistical substitutability, an inferential criterion that evaluates the extent to which twin predictions can reduce human measurement for a particular estimand while preserving valid inference. We develop a framework, grounded in mixed-subject and prediction-powered inference, that evaluates statistical substitutability along four dimensions: aggregate fidelity, paired respondent-level signal, finite-sample human-label recovery, and stability across populations. Across two empirical evaluations spanning behavioral experiments, multiple models, and alternative respondent representations, we find that digital twins can reproduce average human effects while providing little information about which individuals differ from those averages. Newer models and richer respondent information improve some dimensions of performance but do not reliably translate into human-data savings. Human calibration can reduce aggregate prediction error, yet limited labeled samples often fail to produce stable precision gains. Importantly, these findings demonstrate that behavioral fidelity is neither necessary nor sufficient for statistical substitutability. More broadly, they suggest that AI-generated evidence should be evaluated based on its ability to support valid scientific inference rather than its ability to reproduce human outcomes alone. Digital twins should therefore be judged for confirmatory use by whether they reduce uncertainty about human quantities, not merely by whether they reproduce human means, distributions, or effects.

0.61

Structuring Survey Histories for LLM Digital Twins: A Controlled Study of Background, Decision Procedure, and Evaluation Profiles

cs.CL ·2026-11-05 ·Iris Ye, Tianze Deng, Ozan Candogan
published similarity 0.57
Abstract · 10 pages. Camera-ready version for SocialAgent: Second Workshop on Large Language Models for Social Reasoning and Simulation, NeurIPS 2026

LLM-based digital twins aim to predict how individuals would respond in new settings from representations of their prior responses. Existing evidence that compression of survey histories need not reduce predictive accuracy suggests that representation structure, rather than information volume alone, may be an important design choice. We study BDE, a fixed representation that organizes each respondent's survey history into Background, Decision procedure, and Evaluation. We evaluate it on Twin-2K-500 (Toubia et al., 2025) by predicting each respondent's held-out answers on 17 tasks. In a paired evaluation with 100 respondents, a fully inferred BDE profile improves overall accuracy over the raw survey transcript by 1.27 percentage points (pp) on gpt-5.4-nano (p < 0.01) and 1.19 pp on gpt-5.4-mini (p < 0.001). Three analyses then examine the schema, extraction depth, and component composition. Against a length-matched free-form profile written from the same transcript by the same extractor, BDE improves pricing accuracy on both simulators (2.15 pp on nano, p < 0.01; 1.38 pp on mini, p < 0.05), while its overall advantage is significant only on nano (1.05 pp, p < 0.05). Allowing deeper inference during extraction yields no statistically significant gain on either simulator. By contrast, adding Evaluation to a Background--Decision procedure profile improves overall accuracy by 1.44 pp on nano (p < 0.001) and 1.22 pp on mini (p < 0.001). These results indicate that the advantage over raw transcripts is not explained by compression or length alone, and attribute it primarily to profile composition rather than deeper extraction.

0.33

ExploraTwin, a Non-Profit Research Platform for Digital Twin Simulations

cs.CY cs.AIcs.HC ·2026-08-20 ·Naveen Venkat, Yuchen Qiu, Tianyi Peng, George Gui, Olivier Toubia
survey similarity 0.55
Abstract · 44 pages (32-page manuscript plus web appendix), 8 figures. Platform: https://exploratwin.org

Digital twin simulations show promise, but current empirical evidence suggests that the approach should be tested before being deployed in any particular context. To lower the friction for researchers and practitioners to test and deploy digital twin simulations, this brief commentary introduces ExploraTwin (https://exploratwin.org), an open-access, non-profit research platform for digital twin survey simulations. ExploraTwin supports two modes. In survey mode, researchers can upload a Qualtrics survey file or create a survey within the platform; select an available sample of digital twins; configure and run the simulation, and export analysis-ready data. In panel mode, researchers can assemble a small group of twins for open-ended conversations, document annotation, and moderated, focus-group-style voice discussions. We also developed CroissantTwin, a standardized data format for adding samples of digital twins to the platform. We demonstrate the survey mode workflow by using the platform to replicate 19 experiments on digital twins from the Twin-2K-500 dataset. ExploraTwin's survey execution fidelity is high: 99.6% of 197,000 answer units returned a structurally valid response on the first run.

0.35

A Battery for AI Propensity Measurement: Rubrics, Synthetic Item Banks, and Annotations

cs.AI cs.CYcs.LG ·2026-10-25 ·Peter Romero, Haotian Li, Hugo Save, Luning Sun, Zachary R. Tidler +6 more
similarity 0.55
Abstract · 10 pages, 2 figures, instrument release

We release a battery of 28 propensities for AI research, with expert-written rubrics, synthetic datasets, and annotated demand intervals, built with the method of Romero-Alvarado et al. (2026). We selected a wide collection of generic propensities that may influence the behavior of AI systems and humans. They are inspired by psychometric, psychological, and behavioral research, so that experimental results can be understood in the context of familiar constructs previously used also for human behavior research. The rubrics are useful for annotating existing benchmarks whose items may be relevant or not for one or more propensities. For easier profiling, however, the synthetic datasets are defined in such a way that they should be sensitive to one propensity only, allowing for a quick prototyping of propensity profiles using the battery of datasets. We also outline directions for future propensity research.

0.23

The Impact of Personalized AI Response Suggestions on Content Creators' Direct Messaging

cs.HC cs.AI ·2026-10-26 ·Eun Jeong Kang, Jingruo Chen, Jadon Geathers, Susan Fussell
published similarity 0.54
Abstract · Group 2027

Direct messaging is often employed by social media content creators to directly interact with followers. Those creators attempt to engage with followers in private communication, but they often encounter a large volume of messages. Through formative interviews with 15 creators, we found that creators frequently experience fatigue from writing suitable content, as they moderate audience expectations while concurrently integrating their media personas into responses. To explore how AI can address these challenges, we designed a system providing personalized, AI-generated response suggestions tailored to creators' writing styles to maintain their media personas. In a comparison study with 13 creators that examines how the system affects response behaviors, creators valued the system's ability to soften potential impoliteness and anticipated higher response rates, helping them maintain a professional image. However, creators reported additional labor in verifying the authenticity of AI outputs to mitigate risks caused by AI misalignment. Our results indicate that while AI suggestions enhance efficiency, they can undermine a creator's engagement in communication by shifting their attention toward monitoring AI.

0.21