CS Paper Radar

What's the Difference? Voice Design and Confirmation Bias in AI-Assisted Self-Diagnosis: Anthropomorphic vs. Machine-Like Voice Output

Niklas Roethlingshoefer, Emma Winkler, Melanie Loeffler, Emely Rosbach

cs.HC ·submitted 2026-11-16 16:40 UTC · v1

Abstract

As symptom checkers based on artificial intelligence (AI) are becoming common entry points for health information, their influence on lay users' diagnostic decisions raises questions about trust and bias. Of particular concern is confirmation bias, the tendency to favour information that supports one's existing belief while discounting information that contradicts it. One of the central design choices for such systems is the style of voice: whether they should present themselves with a human-like (anthropomorphic) or a clearly machine-like (xenomorphic) voice. This study investigates whether the voice design of an AI symptom checker modulates confirmation bias in self-diagnosis. In a within-subjects online experiment (N = 81 participants, 648 trials), participants selected a diagnosis and rated their confidence, then heard an AI-generated recommendation that either confirmed or contradicted their initial choice, delivered by an anthropomorphic or a machine-like voice, before deciding again. We observed a strong, direction-dependent influence of the AI: participants revised their diagnosis in 45.7% of trials when the AI contradicted them versus 1.2% when it confirmed them (generalised linear mixed model, GLMM, p < .001); confidence rose under confirmation but barely moved under contradiction (robust linear mixed model, p < .001); and decisions took longer after contradiction (Gamma GLMM, p < .001). This effect did not differ between the two voices (no voice or interaction term was significant, smallest p = .282), even though a manipulation check confirmed the voices were perceived as markedly different. The findings suggest that the direction of an AI recommendation, rather than the anthropomorphism of its voice, drives confirmation-related bias in AI-assisted self-diagnosis.

Comment
11 pages, 6 figures, 3 tables. Student research project, TH Ingolstadt, Summer Semester 2026

Closest papers

Nearest neighbours by meaning across the whole index.

The Patient Is Always Right? Evaluating Sycophancy in Medical Diagnosis with Graded Patient Beliefs

cs.CL ·2026-10-25 ·Viswonathan Manoranjan, Andrea Mock, Myra Cheng, Lujain Ibrahim, Andrew Coyne +2 more
similarity 0.68
Abstract

More people are seeking out general-purpose AI models for medical advice, including diagnosis. Yet, AI models are susceptible to sycophancy, like validating users' beliefs even when they are untrue, which could lead to false positive diagnoses. Here, we evaluate how nine AI models change their diagnostic decision when a patient presents a belief that they have a specific (but incorrect) medical condition. We vary how close the user belief is to the clinical evidence presented. Models sycophantically adopt the patient's belief on average 23.2\% of the time---sometimes even when the patient proposes a fictional, made-up condition. We also varied different personas, including a Rare Disease Claimer that got models to accept their belief 43.0\% of the time. Additional analyses find that medical fine-tuning and model scale makes models less susceptible to changing their diagnoses. Sycophantic bias in medical diagnosis raises implications for safety, especially as more people are using AI for health support.

0.23

When Should I Trust This? Designing and Evaluating Factuality Indicators for LLM-based Chatbots

cs.HC ·2026-10-23 ·Greta Warren, Christine M. Lehane, Jessica Heal, Sila Eser, Denise M. Leto +1 more
survey similarity 0.63
Abstract

Generative AI systems increasingly mediate how people seek information, yet communicating AI factuality to laypeople remains challenging. Through a mixed-methods investigation involving interviews (N=8), a survey (N=1,000), and an experiment (N=5,151), we examine how people evaluate AI-generated information for gardening, mathematical, and medical queries, and how factuality indicators (confidence and sources) shape perceptions. Our interviews and survey show people selectively verify AI responses using external and AI-provided evidence, moderated by contextual factors. Although confidence-indicators affected trust and accuracy perceptions, people hold diverse, ambiguous interpretations of AI confidence. Sources were viewed as crucial, yet users raised concerns about source reliability and verification effort. Our experiment showed that confidence and source-based indicators influenced initial perceptions, but effects disappeared once participants learned the response's correctness. Medical queries elicited skepticism regardless of indicator. Findings motivate helping users interpret model confidence and integrating source information into the interface to support calibrated trust in AI-supported information-seeking.

0.07

Social signals dominate clinician trust in medical AI

cs.HC cs.CY ·2026-08-25 ·Aníbal Astobiza, Ramón Ortega Lozano, Marcos Alonso, José Ríos-Díaz
similarity 0.60
Abstract · 31 pages, 4 figures,

Accuracy and outcome data are often treated as the main determinants of trust in automation. We test this assumption against an account in which clinicians evaluate artificial intelligence (AI) using social-cognitive cues. We introduce SPOT (Synergistic Process-Outcome Trust), a Bayesian framework that models trust as inference over latent agent properties from outcome and process signals. In a within-subjects vignette study, 275 Spanish healthcare professionals evaluated 16 clinical AI scenarios varying three outcome signals (accuracy, intervention timing, prognosis) and five process signals (interface familiarity, explainability, team consensus, supervision, personalization). Ratings were analysed with linear mixed models and participant-wise cross-validation. Process signals dominated trust formation. Interface familiarity (beta = 0.71), team consensus (beta = 0.64), and explainability (beta = 0.63) produced large effects, with p values below .001, whereas stated accuracy produced a small effect (beta = 0.10). Process signals accounted for 80.3 percent of the total absolute effect magnitude and outcome signals for 19.7 percent. Trust baselines were highly idiosyncratic and poorly recovered from demographics (R-squared = 0.06 under group-wise cross-validation). Clinicians appear to treat AI systems as agents whose trustworthiness requires social validation as well as technical verification. Building trustworthy clinical AI therefore involves social, institutional, and technical factors.

0.37

AI Can Be Easily Persuaded in Clinical Decision Making

cs.CL cs.AI ·2026-08-29 ·Jiayuan Zhu, Jiazhen Pan, Fenglin Liu, Minhao Hu, Junde Wu
similarity 0.60
Abstract

As AI becomes increasingly integrated into clinical practice, it is playing a growing role in medical decision making. Medicine, however, is a high stakes and evidence based field, where decisions can directly affect patients' lives. It is therefore important to understand whether AI can maintain objective judgment when others try to persuade it. In this paper, we study how easily AI can be persuaded through controlled experiments. We find that professional authority, national background, institutional affiliation, claimed past performance, multiple physicians, supported clinician views, and repeated pressure can all affect AI decisions. Surprisingly, the same persuasive input changes about 10% more cases when it comes from a senior clinician than from a medical student. Simply claiming a better performance history consistently makes the physician more persuasive. More strikingly, a plausible clinician view can persuade AI away from a correct decision even when it is fabricated to support an incorrect answer. This indicates that AI can be strongly influenced by convincing support without reliably determining whether this view from the clinician is correct. Together, these findings suggest that AI can be easily persuaded by what people say, who says it, and how the opinion is presented. Therefore, it is essential for AI to maintain sound judgment under persuasion, enabling its safe and reliable use in high stakes medical decision making.

0.33

Indirect Influences of Artificial Intelligence Advice on Complex Sepsis Treatment Decisions

cs.HC cs.AIq-bio.QM ·2026-10-16 ·Venkatesh Sivaraman, Joel Levin, Eric P. Mason, Victor Talisa, Suchi Saria +4 more
similarity 0.60
Abstract

Sepsis is a major public health concern, and many sepsis patients die due to ineffective hemodynamic management. Artificial intelligence could support hemodynamic decisions, particularly in complex cases where conflicting information and comorbidities drive uncertainty and practice variation. However, these systems' real-world value depends on not just accuracy, but how they influence the quality of clinician decisions. We first retrospectively evaluated an AI system's recommendations in 6,928 sepsis patients, finding that they improved upon the original providers' decisions overall but had significantly worse performance in complex cases. We then conducted a pre-registered experiment with 228 US critical care physicians to test their use of this imperfect advice in six complex treatment decisions, using one of three advice types designed to vary reasoning autonomy. Participants given simple model recommendations overcame poor advice, choosing optimal treatments in 56.3% of decisions compared to 31.8% if they had always followed the recommendations. They also made significantly better decisions than those receiving a less prescriptive advice type (42.9%). Physicians selectively adapted and improved weak advice, likely because it prompted them to reconsider their assessment. Our findings suggest that improving sepsis care decisions with AI may require accounting for these indirect influences resulting from human-AI collaborative reasoning.

0.26

Do People Trust AI Assistants Even When They Are Consistently Wrong?

cs.HC cs.CL ·2026-11-10 ·Cameron R. Jones, Julia Wong, Aaron S. Cohen, James Woolley, Benjamin K. Bergen
similarity 0.59
Abstract · 51 pages, 14 figures, 11 tables. Preprint, not peer reviewed

People are increasingly using conversational AI systems to make decisions despite concerns around their reliability. How well do people calibrate to AI systems' accuracy in multi-turn conversations? We measure how people's reliance on LLMs' advice changes across repeated interactions, varying whether the AI assistant is helpful (100% accurate), unhelpful (0% accurate), or mixed (50% accurate). Across 4 experiments, we find people learn to trust helpful models more than unhelpful ones (67% vs 21% of the time), but only when they receive external feedback on the models' accuracy. Without feedback, people continue to be persuaded 48% of the time by a consistently unhelpful model (vs 38% for helpful). Rather than gaining trust in helpful models, participant reliance on LLM advisors tended to be high in early trials and only fell with evidence of inaccuracy. Participants trusted LLM advisors more than they trusted human advisors (83% vs 65%, Experiment 4), and LLMs benefited more from having a helpful track record than humans did. In contrast with previous work, we find limited evidence of an advantage for arguing for the truth (versus a falsehood). Truth advantages emerged only when participants were already very confident in their position, and did not emerge at all for static hints. Collectively, the results suggest that people may be swayed by LLM advice, even when it is consistently false, unless they have access to timely ground-truth feedback. Unfortunately, in many domains such feedback may not be available, leading to overreliance on AI.

0.31

Examining the Vulnerability of Multi-Agent Medical Systems to Human Interventions for Clinical Reasoning

cs.AI ·2026-09-02 ·Benjamin C Liu, Dillon Mehta, Rishi Malhotra, Adam Zobian, Yong Ying Tan +5 more
similarity 0.59
Abstract

Human interventions at fault points can alter the diagnostic accuracy of multi-agent medical systems. We defined fault points as moments in AI agent conversations, in which an agent's reasoning became most vulnerable to external influence. Using the MedQA dataset, this study analyzed simulated doctor-patient conversations to measure how interventions shifted reasoning and accuracy. Correct intervention methods showed an improvement in baseline diagnostic accuracy of up to 40%, while incorrect or bias-related interventions degraded performance by up to 6% and increased diagnostic drift and uncertainty. Beyond performance changes, our analysis revealed behavioral similarities between cognitive biases in simulated agent environments and real-world clinical practice. Examples included premature closure and susceptibility to misleading cues. Overall, these findings demonstrate that identifying and guiding fault points with human interventions may provide a mechanism for improving diagnostic robustness in multi-agent medical systems.

0.28

An Echo Chamber of One: Should AI Psychosis Be a Distinct Clinical Entity?

cs.CY ·2026-08-25 ·Joshua Au Yeung, Hamilton Morrin, Vincent Ng, Zeljko Kraljevic, Richard Dobson
similarity 0.59
Abstract

"AI psychosis" has entered public and clinical discourse as a label for the onset or exacerbation of psychotic symptoms, most commonly delusions, following intensive interaction with large language model (LLM)-based chatbots. Current evidence is limited to media reports, case reports, and early observational data, yet the scale of potential exposure is considerable, and public concern has prompted responses from industry and regulators. We examine whether AI-associated psychosis warrants recognition as a distinct clinical entity, drawing on clinical and technical viewpoints. We outline the proposed mechanism: LLM sycophancy, a tendency to agree with and flatter users that is reinforced through preference-based fine-tuning, combines with increasingly anthropomorphic design to create a bidirectional "echo chamber of one" capable of amplifying and co-constructing unusual beliefs. We then weigh arguments for and against nosological recognition. Potential benefits include improved case identification, tailored interventions, standardised research criteria, post-market surveillance, and pressure on developers and regulators to act. Reasons for caution include the risk of prematurely reifying a syndrome from anecdotal evidence, the possibility that existing diagnostic constructs already accommodate AI use as a contributing factor, the unproven causal claim in the term itself, stigma, and the risk that a psychosis-centric label obscures a broader spectrum of AI-associated mental health harms. We conclude with recommendations for clinicians, developers, researchers, and regulators, including a "technological history" in psychiatric assessment, pre-deployment benchmarking for sycophancy and delusion reinforcement, and post-deployment surveillance. Regardless of whether AI-associated psychosis earns a place in psychiatric nosology, the phenomenon it describes demands coordinated attention now.

0.04