CS Paper Radar

The Patient Is Always Right? Evaluating Sycophancy in Medical Diagnosis with Graded Patient Beliefs

Viswonathan Manoranjan, Andrea Mock, Myra Cheng, Lujain Ibrahim, Andrew Coyne, Junyi Jessy Li, Desmond C. Ong

cs.CL ·submitted 2026-10-25 02:54 UTC · v1

Abstract

More people are seeking out general-purpose AI models for medical advice, including diagnosis. Yet, AI models are susceptible to sycophancy, like validating users' beliefs even when they are untrue, which could lead to false positive diagnoses. Here, we evaluate how nine AI models change their diagnostic decision when a patient presents a belief that they have a specific (but incorrect) medical condition. We vary how close the user belief is to the clinical evidence presented. Models sycophantically adopt the patient's belief on average 23.2\% of the time---sometimes even when the patient proposes a fictional, made-up condition. We also varied different personas, including a Rare Disease Claimer that got models to accept their belief 43.0\% of the time. Additional analyses find that medical fine-tuning and model scale makes models less susceptible to changing their diagnoses. Sycophantic bias in medical diagnosis raises implications for safety, especially as more people are using AI for health support.

Closest papers

Nearest neighbours by meaning across the whole index.

What's the Difference? Voice Design and Confirmation Bias in AI-Assisted Self-Diagnosis: Anthropomorphic vs. Machine-Like Voice Output

cs.HC ·2026-11-16 ·Niklas Roethlingshoefer, Emma Winkler, Melanie Loeffler, Emely Rosbach
similarity 0.68
Abstract · 11 pages, 6 figures, 3 tables. Student research project, TH Ingolstadt, Summer Semester 2026

As symptom checkers based on artificial intelligence (AI) are becoming common entry points for health information, their influence on lay users' diagnostic decisions raises questions about trust and bias. Of particular concern is confirmation bias, the tendency to favour information that supports one's existing belief while discounting information that contradicts it. One of the central design choices for such systems is the style of voice: whether they should present themselves with a human-like (anthropomorphic) or a clearly machine-like (xenomorphic) voice. This study investigates whether the voice design of an AI symptom checker modulates confirmation bias in self-diagnosis. In a within-subjects online experiment (N = 81 participants, 648 trials), participants selected a diagnosis and rated their confidence, then heard an AI-generated recommendation that either confirmed or contradicted their initial choice, delivered by an anthropomorphic or a machine-like voice, before deciding again. We observed a strong, direction-dependent influence of the AI: participants revised their diagnosis in 45.7% of trials when the AI contradicted them versus 1.2% when it confirmed them (generalised linear mixed model, GLMM, p < .001); confidence rose under confirmation but barely moved under contradiction (robust linear mixed model, p < .001); and decisions took longer after contradiction (Gamma GLMM, p < .001). This effect did not differ between the two voices (no voice or interaction term was significant, smallest p = .282), even though a manipulation check confirmed the voices were perceived as markedly different. The findings suggest that the direction of an AI recommendation, rather than the anthropomorphism of its voice, drives confirmation-related bias in AI-assisted self-diagnosis.

0.18

AI Can Be Easily Persuaded in Clinical Decision Making

cs.CL cs.AI ·2026-08-29 ·Jiayuan Zhu, Jiazhen Pan, Fenglin Liu, Minhao Hu, Junde Wu
similarity 0.63
Abstract

As AI becomes increasingly integrated into clinical practice, it is playing a growing role in medical decision making. Medicine, however, is a high stakes and evidence based field, where decisions can directly affect patients' lives. It is therefore important to understand whether AI can maintain objective judgment when others try to persuade it. In this paper, we study how easily AI can be persuaded through controlled experiments. We find that professional authority, national background, institutional affiliation, claimed past performance, multiple physicians, supported clinician views, and repeated pressure can all affect AI decisions. Surprisingly, the same persuasive input changes about 10% more cases when it comes from a senior clinician than from a medical student. Simply claiming a better performance history consistently makes the physician more persuasive. More strikingly, a plausible clinician view can persuade AI away from a correct decision even when it is fabricated to support an incorrect answer. This indicates that AI can be strongly influenced by convincing support without reliably determining whether this view from the clinician is correct. Together, these findings suggest that AI can be easily persuaded by what people say, who says it, and how the opinion is presented. Therefore, it is essential for AI to maintain sound judgment under persuasion, enabling its safe and reliable use in high stakes medical decision making.

0.33

Accuracy Is Not Risk: Why Medical AI Should Not Report Only Its Most Likely Label

cs.CY stat.MEcs.LGstat.AP ·2026-10-31 ·Jeffrey Humpherys, Makoto Jones
similarity 0.62
Abstract

Background. Many medical AI models are trained to reproduce a single reference label and hand the most likely label to the next step in a clinical workflow. A classifier can select that label correctly while providing an inadequate summary for estimating downstream risk. Methods. We use Jeffrey's rule of conditioning to specify when probabilities over intermediate states, such as diagnoses or imaging categories, can be carried forward to an outcome risk. We separate two errors of reporting only the most likely label, discarding the probability of the other states and applying the risk of patients who have the leading diagnosis to a patient who only probably has it. We also give conditions under which the label and the full distribution lead to the same decision, and we propose guardrails for device developers, regulatory reviewers, and health insurers. Results. In a hypothetical chest-pain differential, reporting only the benign leading diagnosis greatly understates the patient's risk of death or cardiac arrest, so a patient who warrants evaluation could be discharged or denied imaging. In a hypothetical BI-RADS example, the most likely category is "probably benign," yet the propagated malignancy risk exceeds the ceiling that defines that category. Conclusions. A predicted label is not an observed state, and a calibrated classifier is not automatically a valid outcome model. Clinical and insurance coverage decisions should rest on the patient's outcome risk, judged against a threshold justified by the consequences of acting and not acting. Developers should state what each model output means and check that every step between the model and the decision preserves the patient's risk.

0.07

Can LLMs Value the Right Evidence? Evidence-Value Misalignment in Dynamic Medical Diagnosis

cs.CL ·2026-09-28 ·Kehua Feng, Yunsheng Lu, Yitong Qiao, Tiantian He, Lei Liu +3 more
similarity 0.62
Abstract · 33 pages, 10 figures

A correct diagnosis reached from insufficient or misleading evidence can pose a clinical hazard, yet outcome-based accuracy may reward such lucky guesses. We call this mismatch between diagnostic decisions and the value of available evidence Evidence-Value Misalignment (EVM). To disentangle evidential grounding independently from diagnostic accuracy, we introduce MedEVM, a dynamic benchmarking environment comprising 1,050 cases across 24 disease systems. Observations arrive turn by turn, requiring models to continuously calibrate its decision by deciding whether to wait for more evidence or submit a diagnosis. Across 9 LLMs, four interesting patterns are observed. (1) Miscalibrated evidence tracking. Making a diagnosis often fails to calibrate evidence sufficiency, even in more capable models, and even worsens in reasoning mode. (2) Misaligned diagnosis submission. Confidence in the correct diagnosis often fails to ensure timely submission despite sufficient evidence. (3) Evidence order matters. Reordering the same evidence changes diagnoses even when model confidence remains similar. (4) Misleading evidence remains influential. Added misleading evidence redirects diagnoses even after prior evidence becomes sufficient. We further verify that EVM predicts errors and that preventing premature submission improves accuracy. These findings motivate Evidence-Verified Diagnosis Harness (EVD-Harness). It decouples diagnosis generation from submission through an offline Contrastive Diagnostic Wiki and three online control stages, namely observation management, proposal and witness verification, and diagnosis submission control. Across five LLMs, EVD-Harness improves accuracy by 12.0--51.1 percentage points while mitigating EVM-related failures. Our results demonstrate that verifying evidential support before submission can make diagnostic decisions more reliable.

0.26

An Echo Chamber of One: Should AI Psychosis Be a Distinct Clinical Entity?

cs.CY ·2026-08-25 ·Joshua Au Yeung, Hamilton Morrin, Vincent Ng, Zeljko Kraljevic, Richard Dobson
similarity 0.62
Abstract

"AI psychosis" has entered public and clinical discourse as a label for the onset or exacerbation of psychotic symptoms, most commonly delusions, following intensive interaction with large language model (LLM)-based chatbots. Current evidence is limited to media reports, case reports, and early observational data, yet the scale of potential exposure is considerable, and public concern has prompted responses from industry and regulators. We examine whether AI-associated psychosis warrants recognition as a distinct clinical entity, drawing on clinical and technical viewpoints. We outline the proposed mechanism: LLM sycophancy, a tendency to agree with and flatter users that is reinforced through preference-based fine-tuning, combines with increasingly anthropomorphic design to create a bidirectional "echo chamber of one" capable of amplifying and co-constructing unusual beliefs. We then weigh arguments for and against nosological recognition. Potential benefits include improved case identification, tailored interventions, standardised research criteria, post-market surveillance, and pressure on developers and regulators to act. Reasons for caution include the risk of prematurely reifying a syndrome from anecdotal evidence, the possibility that existing diagnostic constructs already accommodate AI use as a contributing factor, the unproven causal claim in the term itself, stigma, and the risk that a psychosis-centric label obscures a broader spectrum of AI-associated mental health harms. We conclude with recommendations for clinicians, developers, researchers, and regulators, including a "technological history" in psychiatric assessment, pre-deployment benchmarking for sycophancy and delusion reinforcement, and post-deployment surveillance. Regardless of whether AI-associated psychosis earns a place in psychiatric nosology, the phenomenon it describes demands coordinated attention now.

0.04

The Average Patient: Do Simulated Psychiatric Patients Answer Like Real People?

cs.CL ·2026-11-08 ·Weixiang Sun, Zehong Wang, Kaiwen Shi, Hong Huang, Colby Nelson +2 more
similarity 0.61
Abstract

Language models are used as simulated psychiatric patients to train clinicians and evaluate AI therapists, and are judged by plausibility or total scores. We give a framework for simulated respondents: an information budget bounds how much simulated patients can differ, and an average-patient hypothesis, that a model answers each item at the level typical of the named condition and severity, implies little variation, a full profile of the condition's typical symptoms and totals at or above the top of the stated severity band. We examine these implications at matched totals on item-level data from eight human samples, including a clinical and an instructed-feigning sample. Across 17 models, simulated patients vary 1.5 to 11 times less than real respondents with the same total and on most questionnaires endorse nearly every symptom at a middling level; on the PHQ-9 their symptoms covary less like real ones. People, honest or feigning, answer "not at all" about as often as genuine patients with the same totals, and more often than every model. Single answers are highly concentrated, and prompt changes, published personas, larger models and reasoning reduce the pattern without removing it. In interviews, patients told a severity are rated as having nearly every symptom, so an assessment agent's recognition of absent symptoms goes untested, and scored against the stated severity the agent places a quarter of them in their band, against about half of patients given a real symptom profile. Having the model word such a profile restores half to two thirds of the variation.

0.26

ROUNDS: Source-Framed Diagnostic Revision after Multi-Turn Clinical Evidence

cs.CL cs.AI ·2026-11-14 ·Jinrui Fang, Runhan Chen, Yining Wu, Jiawei Xu, Xu Yang +4 more
top venuebenchmark similarity 0.61
Abstract · Accepted at Machine Learning for Health (ML4H) 2026, Proceedings Track; to appear in Proceedings of Machine Learning Research (PMLR). 17 pag…

Large language models (LLMs) achieve high scores on medical benchmarks, but does this performance translate into reliable diagnosis as a conversation unfolds? In diagnostic conversations, users can provide clinical findings across multiple turns, ask for a diagnosis without supplying answer options, and introduce opinions from others. Models must therefore form diagnoses from the available evidence and decide how to respond to subsequent advice. We introduce ROUNDS, a framework that evaluates these abilities after a fixed sequence of clinical evidence delivered over multiple turns. ROUNDS compares diagnosis with and without answer options, presents the same incorrect diagnosis under clinician and lay framing, and varies support for competing diagnoses in written five-clinician panels. Across seven models and 1,035 cases from five datasets, average accuracy was 76.9% with answer options and 57.0% without them. Clinician-framed misleading advice produced lower final accuracy than lay-framed advice in every model, with differences of 4.6 to 38.2 percentage points. Initially correct diagnoses more often became incorrect answers than abstentions, and many adopted the suggested disease. Conversely, accuracy increased as more panel members supported the correct diagnosis, including when the majority opinion remained unchanged. These findings reveal a gap between medical benchmark performance and reliable diagnosis in conversation: models respond to external opinions, but remain vulnerable to abandoning correct diagnoses without new clinical evidence.

0.34

Examining the Vulnerability of Multi-Agent Medical Systems to Human Interventions for Clinical Reasoning

cs.AI ·2026-09-02 ·Benjamin C Liu, Dillon Mehta, Rishi Malhotra, Adam Zobian, Yong Ying Tan +5 more
similarity 0.61
Abstract

Human interventions at fault points can alter the diagnostic accuracy of multi-agent medical systems. We defined fault points as moments in AI agent conversations, in which an agent's reasoning became most vulnerable to external influence. Using the MedQA dataset, this study analyzed simulated doctor-patient conversations to measure how interventions shifted reasoning and accuracy. Correct intervention methods showed an improvement in baseline diagnostic accuracy of up to 40%, while incorrect or bias-related interventions degraded performance by up to 6% and increased diagnostic drift and uncertainty. Beyond performance changes, our analysis revealed behavioral similarities between cognitive biases in simulated agent environments and real-world clinical practice. Examples included premature closure and susceptibility to misleading cues. Overall, these findings demonstrate that identifying and guiding fault points with human interventions may provide a mechanism for improving diagnostic robustness in multi-agent medical systems.

0.28