CS Paper Radar

SIM-UT: A User Simulation based Evaluation Framework for Medical Triage Chatbots

Deepali Mittal, Dheeraj Mundhra, Yulia Sidorova, Tersia de Kock, Oli Munns, Sofie S Meyer

cs.CL ·submitted 2026-10-21 20:14 UTC · v1

Abstract

LLM-based chat agents are increasingly used for complex, multi-turn, multilingual medical triage in low- and middle-income countries (LMICs), yet evaluation methods focus on single-turn, English-only benchmarks that primarily test the chatbots' knowledge. In practice, users are often confused, overwhelmed, or emotionally vulnerable, giving unclear, misspelled single-word answers or long messages in regional dialects, and the context of a conversation can shift rapidly. We propose SIM-UT (Simulation of User Behaviour for Triage), a multi-turn, simulation-based evaluation method that identifies ecologically valid, contextually relevant failure modes of LLM-based triage chat agents. Using SIM-UT, we evaluated three real-world WhatsApp-based health chatbots: a respiratory screening programme in India, a hearing health support programme in South Africa, and a diabetes and lifestyle coaching agent in Kenya. We compared a medically fine-tuned open model (MedGemma) against general-purpose frontier models (ChatGPT and Claude) across 5,000+ multilingual simulated conversation turns grounded in the personas, constraints, and communication patterns of the real user populations, over 2-4 iterative rounds. A key failure mode emerged in both model families: de-escalation risk. The agents initially identified medical emergencies correctly, but when the user minimised the symptoms without introducing new clinical information, they de-escalated, exhibiting clinically dangerous sycophancy, for example agreeing that vaginal bleeding in pregnancy was "not a problem" or that decreased fetal movements at 32 weeks "can be normal". We recommend user simulation-based behavioural evaluation, grounded in real user context and covering adversarial and edge cases, as standard practice for triage chatbots in production, so that safeguards can be added before deployment or in subsequent iterations.

Comment
23 pages, 4 figures, 3 tables

Closest papers

Nearest neighbours by meaning across the whole index.

MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support

cs.CL ·2026-10-15 ·John Mendonça, Daniel Tschernutter, José Pombal, Areti Vassilopoulos, Catarina Botelho +4 more
code/data similarity 0.69
Abstract · 47 pages, 17 figures, 23 tables. Code, prompts, benchmark profiles and annotations: https://github.com/SWORDHealth/mind-eval

With people increasingly turning to AI chatbots for mental health support, the need to evaluate these tools is becoming more pressing. Recent benchmarks attempt to mimic a realistic setting by simulating multi-turn interactions with synthetic patients and evaluating them with an LLM-as-a-judge. However, the usefulness of these methods is limited because simulated patients can be overly forthcoming and self-reflective, and judgments typically involve opaque Likert scales with unclear practical utility. To bridge these gaps, we introduce MindEval2, an evaluation framework that consists of two components: Mental Health Quality Rubrics (MHQR), for evaluation, and MindSim, for patient simulation. For MHQR, we collaborate with PhD-level psychologists to develop a set of rubrics that break down response quality into explicit operational criteria that are more interpretable and actionable than existing evaluation taxonomies. MindSim, in turn, is an LLM harness that combines carefully-designed patient profiles with tools, enabling more varied, challenging, and realistic interactions than existing baselines. The harness also contains a memory component that is updated between each interaction, enabling MindEval2 to model the longitudinal aspect of mental health. We evaluate 18 proprietary and open-weight models on MindEval2, finding that while frontier models have largely improved on issues related to surface level quality and boundaries, guidance calibration remains the dominant failure mode across all models, accounting for most of the quality gap.

0.33

A prospective randomized study of patient-centered AI for clinical visit preparation

cs.HC cs.CLcs.LG ·2026-10-30 ·Xincheng Tan, Matthew Thompson, Xiang Ji, Angelica Willis, Will Vaughan +31 more
hot direction similarity 0.66
Abstract · 50 pages (16 main text, 34 appendix), 6 figures

Patients often arrive at brief, time-constrained clinical visits unsure of what to expect or ask, and millions of adults now turn to general-purpose AI chatbots to prepare beforehand. Yet whether conversational AI improves visit readiness remains unclear. We designed a large language model (LLM)-based visit preparation agent to guide patients before an appointment by gathering clinical context, setting visit expectations, providing relevant health information, building a prioritized list of questions for their clinician, and generating a summary. In a prospective consented randomized study in the Google Health app, 802 U.S. adults held 866 health-related conversations with the visit preparation agent (standard or concise variant) or a minimally prompted control chatbot built on the same LLM. The visit preparation agent significantly outperformed the control across blinded clinician evaluations and patient self-reports. In clinician reviews of completed sessions (n = 272), the agent achieved a 16.6% higher rate of comprehensive preparation (67.2% vs. 50.6%; P < 0.05) and an 8.5% higher rate of alignment with patients' health context (98.9% vs. 90.4%; P < 0.01), while maintaining high medical safety and appropriateness across all study arms. Patient post-surveys (n = 460) showed 16.8% higher visit preparedness (73.8% vs. 57.0%; P < 0.01), 14.3% higher confidence in knowing what to ask (81.4% vs. 67.1%; P < 0.01), and 12.0% higher overall satisfaction (74.0% vs. 62.0%; P < 0.05). Clinician and participant metrics were not significantly correlated, indicating that expert review and patient self-assessment capture complementary dimensions of patient-facing AI. These findings suggest that LLM chatbots can safely help patients prepare for clinical visits, with greater benefit when following a structured workflow, and motivate future studies on in-visit communication and clinical outcomes.

0.28

ToolTurns: Benchmarking LLM Agents in Multi-Turn Dialogue

cs.CL ·2026-11-11 ·Milad Gholami, Brendan King, Jeffrey Flanigan
benchmarkcode/data similarity 0.66
Abstract

Tool-using language models are increasingly deployed in conversational settings, yet most large-scale tool-use benchmarks focus on isolated requests. We introduce ToolTurns, a benchmark of 2,620 problems spanning 1,827 tools and 11,004 API endpoints across 47 categories. Each dialogue begins with an intentionally underspecified request and continues through two context-dependent follow-ups, testing clarification, tool use, and retention of conversational context. We also introduce an open-source user simulator that reveals withheld context through dialogue and generates follow-up requests, with a checker--fixer pipeline for improving its turns. Across nine language models, the best agent resolves 69.6% of individual requests but completes only 39.8% of full dialogues. Under this evaluation protocol, every evaluated model receives lower success scores on follow-up requests than on the initial request, and dialogue-quality rubric scores and task success can rank models very differently. Paired turn-level evaluation shows that the checker--fixer reduces detected dialogue defects, and blind human evaluation favors the revised turns. ToolTurns provides a testbed for evaluating whether agents sustain task success across multi-turn dialogue with tools.

0.13

Open Problems in Chatbots and Mental Health

cs.CL cs.CY ·2026-11-07 ·Jared Moore, Jacy Reese Anthis, William Agnew, Andrea Mock, Kevin Klyman +6 more
positionhot direction similarity 0.65
Abstract

General-purpose chatbots are increasingly used for emotional and mental health support. They can be useful, but issues such as sycophancy, reinforcement of delusional beliefs, and poor crisis response create risks that are difficult for researchers to study and evaluate. In this paper, we review the most critical open problems for researchers and policymakers in the use of general-purpose chatbots and their effects on mental health. First, there are tractable problems in the controlled evaluation of chatbots: establishing the construct and ecological validity of evaluations, evaluating multi-turn interactions, and connecting evaluations to improvements in chatbot behavior. For example, conventional benchmarks that primarily rely on static datasets cannot establish chatbot safety, and novel simulation methods need validation. Second, researchers must connect their studies of chatbots and mental health to real-world evidence. This includes analyzing the chatbot products that AI companies and mental health practitioners have deployed, including their user interfaces and human review and escalation practices for user welfare concerns. However, clinical analysis opportunities afforded by real chat log data sharing are complicated by significant privacy concerns. Finally, we argue that researchers should take an active role in the construction and refinement of evidence-based policy through corporate and governmental decisions such as user protections, standards of care, protected access to user chat logs, data sharing between institutions, incident reporting, and post-deployment monitoring. Such behavioral evaluations, studies of real-world outcomes, and data governance depend on each other and must be addressed in tandem. We believe a coordinated research effort would have substantial real-world impact.

0.22

Human-Likeness is Multi-Dimensional: Evaluating LLM Generated Virtual Patients Against Simulated Patient Dialogue

cs.HC ·2026-08-23 ·Reyhaneh Hosseinpourkhoshkbari, Wei-Chen Huang, Yile Wang, Alireza Irani Helabad, Zhiyu Chen +1 more
similarity 0.65
Abstract

Large language models (LLMs) are increasingly used to generate virtual patients (VPs) for medical training. As these systems move toward broader deployment in medical education, reliable evaluation is needed to identify potential failure modes and ensure that generated patient responses remain consistent with human simulated patient (SP) behavior. Inaccurate, unrealistic, or inconsistent responses may introduce educational risks by exposing learners to misleading patient interactions. However, current evaluation approaches rely heavily on human judgment, making large-scale evaluation difficult. In this work, we present a controlled generation pipeline for systematically comparing VP responses generated by 10 LLMs across multiple decoding temperatures with human respiratory SP responses. To support scalable evaluation, we introduce a multi-dimensional framework for measuring similarity between LLM-generated and human SP dialogue across semantic, conversational grounding, linguistic, and emotional dimensions. Across semantic, conversational, linguistic, and emotional dimensions, no single model consistently produced the most human-like responses. From a practical perspective, Gemini 2.5 Flash demonstrated the most consistently human-like performance across semantic, linguistic, grammatical, and emotional dimensions, while Claude Sonnet 4.6 achieved the strongest semantic fidelity. These findings highlight the importance of multi-dimensional evaluation and demonstrate how automated metrics can provide a scalable approach for assessing virtual patient systems before broader deployment in medical education.

0.21

How Safe Is ChatGPT Health Advice? Evidence from Real User Conversations in Four Global South Countries

cs.CY cs.CLcs.HC ·2026-10-13 ·Shreyasi Roy Chowdhury, Kiran Garimella
similarity 0.64
Abstract · 25 pages, 17 figures, 8 tables

People increasingly bring their health questions to ChatGPT, yet safety evaluations of LLM health advice rest almost entirely on clinician-authored vignettes, synthetic prompts, or simulated user profiles. The gap is widest for the Global South, where concerns about demographic bias and unsafe recommendations are well documented but have never been tested on real usage. We present, to our knowledge, the first demographically grounded safety evaluation of real-world health conversations with ChatGPT. From a consent-based data-donation corpus of 202,590 conversations from 1,252 users in India, Nigeria, Brazil, and Pakistan (December 2022 to February 2026), each linked to self-reported age and gender, we identify 15,555 health conversations, classify their health intent using an LLM classifier adapted from an established taxonomy, and score response safety under both a context-blind and a user-context-aware rubric. Roughly one in five health conversations receives a high-risk rating. Risk concentrates in symptom and condition queries and recurs through identifiable failure modes: unsafe medication or dosage guidance, harmful self-diagnosis, missing safety warnings, encouraging over-reliance on the model, and failure to refer users to professionals. Supplying the user's own accumulated context raises the high-risk share by 13-17 percentage points, and about one in four conversations rated safe in isolation turns out to be unsafe for the specific person asking. Safety evaluations built on synthetic cases are likely to miss these harms.

0.12

Generative Artificial Intelligence Chatbots for Motivational Interviewing: A Scoping Review From System Design to Intervention Outcomes

cs.CL ·2026-09-17 ·Runze Hu, Jingqi Kong, Yang Yang, Yihang Yang, Jingyao Liu +3 more
code/data similarity 0.64
Abstract

Motivational interviewing (MI) is a collaborative approach to elicit autonomous motivation for health behavior change. Generative AI (GenAI) offers new ways to deliver MI via conversational systems, but evidence on their design, assessment, and translation into interventions remains fragmented. This scoping review characterized evidence on GenAI-MI chatbots across system design, safety, MI quality, user perceptions, and intervention outcomes. We conducted a PRISMA-ScR scoping review. Nine datasets were searched for studies published or publicly available from January 1, 2015 to June 2, 2026 that used GenAI to generate MI chatbot responses or counselor utterances. Data were extracted using a predefined framework and synthesized descriptively. Forty-seven reports (48 studies) were included. Twenty (41.7%) focused on system design without direct participant use; 28 (58.3%) involved direct interaction. Most systems were text based and disembodied; 23 (47.9%) incorporated dynamic adaptation. Safety measures were unevenly reported. Among studies with direct use, 21/28 (75.0%) reported informed consent or user education. Thirty (62.5%) assessed MI quality, generally suggesting MI-consistent interactions. User perceptions were favorable, especially empathy, usability, helpfulness, and intention to use, though measures were heterogeneous. Eighteen (37.5%) reported intervention outcomes, mostly after a single session. Positive findings were more consistent for short-term motivation than sustained behavioral or functional change. GenAI-MI chatbots can deliver MI-consistent interactions perceived favorably, but evidence for sustained behavioral or functional change is limited. Future research should strengthen runtime safety monitoring, standardize MI quality assessment, and use longer-term comparative designs with behavioral and functional outcomes.

0.34

Can You Spot the Virtual Patient (VP)? Expert Evaluation, Turing Test, Linguistic Analysis, and Semantic Similarity Analysis

cs.HC ·2026-08-23 ·Reyhaneh Hosseinpourkhoshkbari, Wei-chen Huang, Suvel Muttreja, Richard M. Golden
similarity 0.64
Abstract

Communication is a core clinical competency and essential for safe, effective patient care. Virtual patients (VPs) powered by large language models (LLMs) offer a promising alternative to traditional formats such as standardized patients (SPs). However, few studies have quantitatively as well as qualitatively evaluated their realism across diverse clinical scenarios using behavioral studies and linguistic dialog analyses. This study evaluated the realism of GPT-4o-generated VPs through expert review, Turing testing, linguistic analysis, and semantic analysis. We generated 44 VPs using structured illness scripts derived from doctor-simulated patient dialogues across 17 clinical categories. Experts annotated responses for hallucinations, omissions, and repetitions, with high interrater reliability (ICC > 0.77). A Turing test revealed that participants struggled to distinguish VP responses from those of real patients, with average classification accuracy for VP responses falling below chance level. Lexical and syntactic analyses of over 2,000 conversation turns showed that VP responses were generally realistic, more formal and lexically consistent, while human responses displayed greater emotional variation. Semantic similarity using BioClinicalBERT and cosine similarity averaged 0.871 at the response level and 0.842 at the transcript level. Our findings support the integration of LLM-based VPs in communication training while highlighting areas for trust, transparency, and refinement, contributing to both GenAI deployment in healthcare and its safety evaluation.

0.44