CS Paper Radar

A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform

Yongjin Yi, Jinyong Park, Sejoong Kim

cs.HC ·submitted 2026-10-21 07:08 UTC · v2
top venuecode/data

Abstract

The Objective Structured Clinical Examination (OSCE) is widely used to assess clinical communication skills in medical education, requiring trainees to take histories and reason clinically with standardized patients (SPs). Large language model (LLM)-based virtual standardized patients (VSPs), typically built by persona-injected prompting, offer a practical alternative. However, existing evaluations rely on a single backbone LLM, small human rater panels, or LLM-as-a-judge scoring without an utterance-aligned human-expert reference, and so cannot identify the specific shortcomings of individual models. We evaluated four LLMs (GPT-4o, Claude 3.5 Sonnet, Llama-3.1 70B, and HyperCLOVA X), spanning commercial, open-source and Korean-developed models, against a human-expert reference across three Korean OSCE cases (hematuria, fever in pregnancy, and chronic cough). In a double-blind, fully crossed design, 17 senior medical-student raters scored every model utterance on three sentence-level and six encounter-level Likert metrics, yielding 7,480 data points. We analyzed the scores using a stratified linear mixed-effects model based on an 11-category history-taking and 3-level linguistic-form taxonomy. At the encounter level, three of the four LLMs fell 0.08 to 0.47 Likert points below the human-expert reference on Comprehensibility, Consistency and Hallucination-freeness. The difference was not statistically significant for Hallucination-freeness in Claude 3.5 Sonnet and GPT-4o, Consistency in GPT-4o, and Engagement in HyperCLOVA X. Llama-3.1 70B fell significantly below the reference on all six metrics. We also introduce a web-based OSCE platform that supports virtual practice across diverse clinical cases and continuously collects student-VSP dialogues under research and privacy consent.

Comment
Accepted at NeurIPS 2026, Track on Evaluations and Datasets. 39 pages (10 pages of main text), 3 figures and 3 tables in the main text, 4 figures and 4 tables in the appendices. Dataset: https://huggingface.co/datasets/nwirandx/Korean-OSCE-history-taking-dialogues Code: https://github.com/JINAILAB/OSCE-SimBench

Closest papers

Nearest neighbours by meaning across the whole index.

From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models

cs.AI cs.CL ·2026-10-30 ·Navapat Nananukul, Mayank Kejriwal
benchmark similarity 0.69
Abstract

Large language models (LLMs) now answer medical exam questions at or above expert level, but it is unclear whether such scores reflect clinical competence, or are instead affected by the format of the questions being asked (e.g., multiple-choice versus open-ended), prior exposure to the benchmarks, and the difficulty of the questions. These benchmarks are also public and widely used online, which may be resulting in LLM memorization. Models may perform better on questions that they have seen before. It is also not evident whether smaller models, which are more locally deployable in HIPAA-compliant hospital settings, have achieved human performance. In this study, we evaluate fourteen current models from six leading providers, one smaller and one larger model from each provider with two additional open-weight models, on 3,076 questions from four medical benchmarks (MedQA-USMLE, MedMCQA, PubMedQA, and MedBullets) under standard multiple choice, the same questions with the answer choices withheld, and reconstruction of a hidden question from its choices alone. In the standard multiple-choice setting, the strongest models exceed the human-expert baseline on MedQA-USMLE and PubMedQA, but the smaller models trail their larger models by 8.7 percentage points and only two of the six pass the human-expert bar on the licensing examination. Withholding the answer choices, however, makes every model struggle: mean accuracy falls from 82.3% to 50.9%, and the two largest drops belong to the medically specialised models. On memorization, the strongest models reconstruct one question in four from the answer choices alone, yet accuracy holds on the questions they cannot reconstruct. We further weigh accuracy against inference cost and test whether ensembling several models improves medical question answering.

0.15

Can You Spot the Virtual Patient (VP)? Expert Evaluation, Turing Test, Linguistic Analysis, and Semantic Similarity Analysis

cs.HC ·2026-08-23 ·Reyhaneh Hosseinpourkhoshkbari, Wei-chen Huang, Suvel Muttreja, Richard M. Golden
similarity 0.69
Abstract

Communication is a core clinical competency and essential for safe, effective patient care. Virtual patients (VPs) powered by large language models (LLMs) offer a promising alternative to traditional formats such as standardized patients (SPs). However, few studies have quantitatively as well as qualitatively evaluated their realism across diverse clinical scenarios using behavioral studies and linguistic dialog analyses. This study evaluated the realism of GPT-4o-generated VPs through expert review, Turing testing, linguistic analysis, and semantic analysis. We generated 44 VPs using structured illness scripts derived from doctor-simulated patient dialogues across 17 clinical categories. Experts annotated responses for hallucinations, omissions, and repetitions, with high interrater reliability (ICC > 0.77). A Turing test revealed that participants struggled to distinguish VP responses from those of real patients, with average classification accuracy for VP responses falling below chance level. Lexical and syntactic analyses of over 2,000 conversation turns showed that VP responses were generally realistic, more formal and lexically consistent, while human responses displayed greater emotional variation. Semantic similarity using BioClinicalBERT and cosine similarity averaged 0.871 at the response level and 0.842 at the transcript level. Our findings support the integration of LLM-based VPs in communication training while highlighting areas for trust, transparency, and refinement, contributing to both GenAI deployment in healthcare and its safety evaluation.

0.44

KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy

cs.CL cs.AI ·2026-09-29 ·Xueting Fang, Zehui Li, Yang Yang, Camilla Giovino, Shubh K. Patel +3 more
benchmarkcode/data similarity 0.68
Abstract

Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients present information in different ways, and clinicians must obtain relevant history and determine which examinations are needed before reaching a diagnosis. Diagnostic accuracy alone therefore cannot establish whether an agent gathered essential information or conducted an appropriate clinical assessment. Furthermore, existing benchmarks lack professional clinicians' verification. To address this gap, we introduce KlinikeBench, a benchmark of 333 clinician-authored tasks, each providing an isolated sandbox environment with a virtual patient, clinical tools, and task-specific success criteria. More than 35 clinicians contributed to case authoring and benchmark evaluation. In an empirical study, clinicians gave simulated dialogues higher mean quality ratings than reference conversations, which is adapted from real conversation. In each task, an LM has a fixed budget of turns to communicate with the patient, ask about relevant history, request examinations, follow action constraints, and record a final diagnosis. We score these steps separately as well as together. Across 31 models and seven model families, the best-performing models (e.g., GPT-6-astra and Claude Opus 5) succeed on less than 30% of tasks, even though their diagnosis accuracy reaches 90.7%. Some models benefit from talking with the patient; others diagnose well from a complete chart but perform much worse in conversation. Overall, KlinikeBench provides a testbed for evaluating the full clinical encounter and reveals a substantial gap between diagnostic accuracy and performance in interactive clinical assessment. All the code and data is available on https://zehui127.github.io/klinikebench/

0.23

A primer on evaluation methods for large language models in healthcare

cs.CL cs.AI ·2026-09-13 ·Suzannah E McKinney, Phuc Vu, Samuel A Justice, Christopher Humphries, Alyssa Pradhan +8 more
similarity 0.68
Abstract

Large language models (LLMs) have a growing range of applications in medicine, and their evaluation is critical for ensuring they provide benefit and not harm. This evaluation can be more challenging than traditional machine learning for many reasons, including probabilistic and open-ended outputs, and behavior that shifts with prompt design and accumulated context. This review covers four key areas of LLM evaluation: principles of study design, statistical methods, capability evaluation and clinical application evaluation. Capability evaluation considers different benchmarks, including multiple-choice, agentic and multi-turn benchmarks, alongside operational metrics like token usage and latency. Clinical application evaluation addresses establishing accuracy of free text outputs, such as human review and LLM-as-a-judge, and clinical trial approaches. Across sections, we describe underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question. Together, this article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs.

0.15

Human-Likeness is Multi-Dimensional: Evaluating LLM Generated Virtual Patients Against Simulated Patient Dialogue

cs.HC ·2026-08-23 ·Reyhaneh Hosseinpourkhoshkbari, Wei-Chen Huang, Yile Wang, Alireza Irani Helabad, Zhiyu Chen +1 more
similarity 0.66
Abstract

Large language models (LLMs) are increasingly used to generate virtual patients (VPs) for medical training. As these systems move toward broader deployment in medical education, reliable evaluation is needed to identify potential failure modes and ensure that generated patient responses remain consistent with human simulated patient (SP) behavior. Inaccurate, unrealistic, or inconsistent responses may introduce educational risks by exposing learners to misleading patient interactions. However, current evaluation approaches rely heavily on human judgment, making large-scale evaluation difficult. In this work, we present a controlled generation pipeline for systematically comparing VP responses generated by 10 LLMs across multiple decoding temperatures with human respiratory SP responses. To support scalable evaluation, we introduce a multi-dimensional framework for measuring similarity between LLM-generated and human SP dialogue across semantic, conversational grounding, linguistic, and emotional dimensions. Across semantic, conversational, linguistic, and emotional dimensions, no single model consistently produced the most human-like responses. From a practical perspective, Gemini 2.5 Flash demonstrated the most consistently human-like performance across semantic, linguistic, grammatical, and emotional dimensions, while Claude Sonnet 4.6 achieved the strongest semantic fidelity. These findings highlight the importance of multi-dimensional evaluation and demonstrate how automated metrics can provide a scalable approach for assessing virtual patient systems before broader deployment in medical education.

0.21

An evidence-traceable benchmark of clinical reasoning in large language models

cs.CL ·2026-11-10 ·Zhiqun Zhao, Xiaodong Yang, Bruce Gu, Ming Li, Yi Fu +11 more
benchmark similarity 0.66
Abstract · 25 pages, 8 figures, 4 tables. Zhiqun Zhao, Xiaodong Yang and Bruce Gu contributed equally. Corresponding authors: Zheng Wang, Haiquan Lu an…

Benchmark scores for medical large language models (LLMs) are difficult to interpret when scoring criteria cannot be traced to clinical evidence. Here we developed Med Evidence Bench (MEB; https://www.medevidencebench.org.cn/), comprising 1,536 physician-constructed items across three levels: guideline knowledge, standardized clinical scenarios and real-world laboratory scenarios. Each criterion stating a medical claim is linked to a versioned source-text span. Across ten LLMs, the Evidence Trace Score (ETS) was 0.975-0.997 for guideline knowledge, 0.292-0.658 for standardized scenarios and 0.182-0.447 for real-world scenarios, and rankings varied across levels. For real-world scenarios, scores for identifying clinical information (mean, 0.770) exceeded those for matching evidence (0.165) or formulating recommendations (0.160). On 1,296 criterion-level judgments, automated grading agreed with the majority vote of three physicians (macro-F1, 0.930; 95% confidence interval, 0.915-0.945), although agreement was lower for evidence application. These findings from Chinese-language materials indicate that guideline recall does not ensure patient-specific evidence use and do not establish clinical effectiveness.

0.25

Large Language Models for Accurate Medical Chart Abstraction: Enabling Scalable and Secure Artificial Intelligence Deployment in Stroke

cs.CL ·2026-10-27 ·Zhusi Zhong, Carl M. Porto, David Hong, Miranda Gonzales, Revyn Kim +11 more
publishedcode/data similarity 0.66
Abstract

Medical chart abstraction transforms unstructured clinical narratives into structured variables for research and quality monitoring. We evaluated a prompting-based large language model (LLM) framework for extracting eight clinical variables from 2,416 neurointerventional acute stroke intervention reports across three hospitals. Twenty-two instruction-tuned open-source LLMs were evaluated across architectures, model sizes, and biomedical adaptations using Quick Response and Chain-of-Thought (CoT) prompting. Non-expert staff annotations served as the cross-model reference standard. Evaluation was restricted to report-variable pairs with a valid LLaMA3.3-70B CoT prediction and a corresponding reference label. Performance measures included extraction accuracy, inference latency, and agreement with expert adjudication. Twenty discordant cases per variable underwent review by two senior physicians and four medical students. LLaMA3.3-70B achieved the highest overall accuracy (94.8%). CoT improved performance on inference-dependent variables such as site of occlusion, while Quick Response was optimal for directly stated procedural fields. Expert adjudication showed that AI predictions were more closely aligned with expert interpretations than non-expert annotations for most variables, with intracranial angioplasty remaining a challenge. Locally deployed LLMs can support scalable clinical information extraction and retrospective stroke registry curation, with expert oversight for ambiguous cases.

0.39

MTDiag: A Multi-Turn Diagnostic Dataset Towards Clinically Meaningful LLM Evaluation

cs.CL ·2026-08-25 ·Pia Chouayfati, Alexander M. Fichtl, Miriam Anschütz, George Doumat, Georg Groh
published similarity 0.64
Abstract · 20 pages, published in the SIGDIAL 2026 conference proceedings, see https://aclanthology.org/2026.sigdial-1.58/

Clinical diagnosis is fundamentally interactive and incremental, yet the dominant paradigm for evaluating Large Language Models (LLMs) in medicine remains static QA benchmarks or template-based dialogues. These benchmarks say little about whether a model can serve as a diagnostic agent in a dynamic clinical encounter, with LLMs showing significant accuracy and reliability degradation in multi-turn settings. To address this issue, we present MTDiag, a large multi-turn diagnostic dialogue dataset constructed from three heterogeneous sources: DDXPlus, MIMIC-IV, and published case reports (AJCR), covering common ED presentations as well as long-tail rare and atypical conditions. All cases are normalized into a canonical schema anchored in the most comprehensive and widely-adopted medical knowledge bases (UMLS concept identifiers, with ICD-10 diagnosis codes). We release the schema, a UserLM-8B-based utterance-generation pipeline, and the physician-validated dataset that converts structured clinical evidence into natural-language utterances. Importantly, we introduce and motivate clinical knowledge-grounded metrics for evaluating LLMs as diagnostic agents, beyond diagnostic accuracy, for the task of multi-turn differential diagnosis.

0.63