CS Paper Radar

Social signals dominate clinician trust in medical AI

Aníbal Astobiza, Ramón Ortega Lozano, Marcos Alonso, José Ríos-Díaz

cs.HC cs.CY ·submitted 2026-08-25 15:33 UTC · v1

Abstract

Accuracy and outcome data are often treated as the main determinants of trust in automation. We test this assumption against an account in which clinicians evaluate artificial intelligence (AI) using social-cognitive cues. We introduce SPOT (Synergistic Process-Outcome Trust), a Bayesian framework that models trust as inference over latent agent properties from outcome and process signals. In a within-subjects vignette study, 275 Spanish healthcare professionals evaluated 16 clinical AI scenarios varying three outcome signals (accuracy, intervention timing, prognosis) and five process signals (interface familiarity, explainability, team consensus, supervision, personalization). Ratings were analysed with linear mixed models and participant-wise cross-validation. Process signals dominated trust formation. Interface familiarity (beta = 0.71), team consensus (beta = 0.64), and explainability (beta = 0.63) produced large effects, with p values below .001, whereas stated accuracy produced a small effect (beta = 0.10). Process signals accounted for 80.3 percent of the total absolute effect magnitude and outcome signals for 19.7 percent. Trust baselines were highly idiosyncratic and poorly recovered from demographics (R-squared = 0.06 under group-wise cross-validation). Clinicians appear to treat AI systems as agents whose trustworthiness requires social validation as well as technical verification. Building trustworthy clinical AI therefore involves social, institutional, and technical factors.

Comment
31 pages, 4 figures,

Closest papers

Nearest neighbours by meaning across the whole index.

Examining the Vulnerability of Multi-Agent Medical Systems to Human Interventions for Clinical Reasoning

cs.AI ·2026-09-02 ·Benjamin C Liu, Dillon Mehta, Rishi Malhotra, Adam Zobian, Yong Ying Tan +5 more
similarity 0.64
Abstract

Human interventions at fault points can alter the diagnostic accuracy of multi-agent medical systems. We defined fault points as moments in AI agent conversations, in which an agent's reasoning became most vulnerable to external influence. Using the MedQA dataset, this study analyzed simulated doctor-patient conversations to measure how interventions shifted reasoning and accuracy. Correct intervention methods showed an improvement in baseline diagnostic accuracy of up to 40%, while incorrect or bias-related interventions degraded performance by up to 6% and increased diagnostic drift and uncertainty. Beyond performance changes, our analysis revealed behavioral similarities between cognitive biases in simulated agent environments and real-world clinical practice. Examples included premature closure and susceptibility to misleading cues. Overall, these findings demonstrate that identifying and guiding fault points with human interventions may provide a mechanism for improving diagnostic robustness in multi-agent medical systems.

0.28

Do People Trust AI Assistants Even When They Are Consistently Wrong?

cs.HC cs.CL ·2026-11-10 ·Cameron R. Jones, Julia Wong, Aaron S. Cohen, James Woolley, Benjamin K. Bergen
similarity 0.62
Abstract · 51 pages, 14 figures, 11 tables. Preprint, not peer reviewed

People are increasingly using conversational AI systems to make decisions despite concerns around their reliability. How well do people calibrate to AI systems' accuracy in multi-turn conversations? We measure how people's reliance on LLMs' advice changes across repeated interactions, varying whether the AI assistant is helpful (100% accurate), unhelpful (0% accurate), or mixed (50% accurate). Across 4 experiments, we find people learn to trust helpful models more than unhelpful ones (67% vs 21% of the time), but only when they receive external feedback on the models' accuracy. Without feedback, people continue to be persuaded 48% of the time by a consistently unhelpful model (vs 38% for helpful). Rather than gaining trust in helpful models, participant reliance on LLM advisors tended to be high in early trials and only fell with evidence of inaccuracy. Participants trusted LLM advisors more than they trusted human advisors (83% vs 65%, Experiment 4), and LLMs benefited more from having a helpful track record than humans did. In contrast with previous work, we find limited evidence of an advantage for arguing for the truth (versus a falsehood). Truth advantages emerged only when participants were already very confident in their position, and did not emerge at all for static hints. Collectively, the results suggest that people may be swayed by LLM advice, even when it is consistently false, unless they have access to timely ground-truth feedback. Unfortunately, in many domains such feedback may not be available, leading to overreliance on AI.

0.31

AI-Assisted Classification under Correlation Neglect and Trust

cs.GT cs.AIcs.LG ·2026-11-04 ·Dingwen Kong, Liang Lyu, Saurabh Amin, Amine Bennouna, Daniel Huttenlocher +1 more
position similarity 0.61
Abstract

When does an AI assistant improve human decision-making, and when does it hurt? We argue that the answer depends not just on AI accuracy but also on how much its evidence overlaps with the human's and how the human combines the two. A worker makes a binary decision by observing her own signal and receiving a recommendation from an AI. We develop a human behavioral model with two primitives: her awareness of the overlap and her trust in the AI. Our analysis gives a closed-form phase diagram over AI capability and overlap, with regions of complementarity, impairment, and automation. A worker who neglects the overlap (correlation neglect) can be made less accurate by a more capable AI, double-counting the shared evidence. Trust reshapes the regions - undertrust may compensate for correlation neglect, so that the trust level maximizing accuracy depends on the deployment parameters. Empirically, we identify the two model parameters by pairing the same workers with different AIs, and break the accuracy loss into a correlation-neglect cost and an undertrust cost, one tied to reducing overlap on the AI side, the other to calibrating trust on the human side. On ChatBench, AI assistance raises accuracy by 21 points, yet workers apply only about a quarter of the Bayes-rational weight to AI information; GPT-4o falls in the automation region and Llama-3.1-8b stays at the complementarity boundary; and calibrating trust recovers about 2.6 points for GPT-4o but almost nothing for Llama-3.1-8b.

0.15

What's the Difference? Voice Design and Confirmation Bias in AI-Assisted Self-Diagnosis: Anthropomorphic vs. Machine-Like Voice Output

cs.HC ·2026-11-16 ·Niklas Roethlingshoefer, Emma Winkler, Melanie Loeffler, Emely Rosbach
similarity 0.60
Abstract · 11 pages, 6 figures, 3 tables. Student research project, TH Ingolstadt, Summer Semester 2026

As symptom checkers based on artificial intelligence (AI) are becoming common entry points for health information, their influence on lay users' diagnostic decisions raises questions about trust and bias. Of particular concern is confirmation bias, the tendency to favour information that supports one's existing belief while discounting information that contradicts it. One of the central design choices for such systems is the style of voice: whether they should present themselves with a human-like (anthropomorphic) or a clearly machine-like (xenomorphic) voice. This study investigates whether the voice design of an AI symptom checker modulates confirmation bias in self-diagnosis. In a within-subjects online experiment (N = 81 participants, 648 trials), participants selected a diagnosis and rated their confidence, then heard an AI-generated recommendation that either confirmed or contradicted their initial choice, delivered by an anthropomorphic or a machine-like voice, before deciding again. We observed a strong, direction-dependent influence of the AI: participants revised their diagnosis in 45.7% of trials when the AI contradicted them versus 1.2% when it confirmed them (generalised linear mixed model, GLMM, p < .001); confidence rose under confirmation but barely moved under contradiction (robust linear mixed model, p < .001); and decisions took longer after contradiction (Gamma GLMM, p < .001). This effect did not differ between the two voices (no voice or interaction term was significant, smallest p = .282), even though a manipulation check confirmed the voices were perceived as markedly different. The findings suggest that the direction of an AI recommendation, rather than the anthropomorphism of its voice, drives confirmation-related bias in AI-assisted self-diagnosis.

0.18

Scalable AI-based clinical communication training and automated assessment

cs.HC ·2026-09-18 ·Masum Hasan, Ron Epstein, Thomas Carroll, Ehsan Hoque
similarity 0.60
Abstract · 16 pages

Poor clinical communication can delay care, contribute to errors, and harm patients, yet opportunities for repeated practice with feedback remain limited. Our prior randomized trial showed that practice with the SOPHIE AI patient platform improved serious illness communication, but the system addressed a single clinical context and required human effort for delivery and assessment. We developed SOPHIE 2.0, a browser-based, self-service platform integrating embodied AI-patient interactions, personalized feedback, and automated assessment across 24 clinical scenarios. An automated large language model assessor evaluated three communication skills---Empower, Be Explicit, and Empathize---with agreement comparable to individual human raters ($r=0.759$; ICC$=0.746$). In a study of 59 clinicians and students, participants completed two AI-patient encounters with personalized feedback; 92% found the platform engaging, 86% easy to use, and 83% clinically relevant. Scores were higher in the second encounter, though the uncontrolled design precludes attributing this change specifically to training.

0.23

Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures

cs.AI cs.CL ·2026-10-15 ·Zihao Zheng, Junyi Yao, Baichuan Li, Jiayu Long
benchmark similarity 0.59
Abstract · 8 pages, 5 figures

Trustworthy biomedical AI requires reliable action selection when evidence is missing, stale, conflicting, or late, not only accurate answers on static inputs. We instantiate an episode-level evaluation protocol as 400 synthetic healthcare workflow episodes: 40 hand-authored semantic seeds across four task classes, two surface parameterizations, and five controlled variants, scored on ANSWER/ASK CLARIFICATION/ABSTAIN/ESCALATE under a pre-declared 5:1 missed-versus-unnecessary escalation cost. Reference actions are single-author study-policy labels, not clinician-adjudicated ground truth. Across five open-weight configurations (1.7B--9.7B), clean decision accuracy reaches 0.97, yet uncertainty-prompted models miss 32--97% of required escalations. The stronger 9.7B configuration narrows but does not remove the gap, falling from 0.97 clean accuracy to 0.78 under conflict. Verbalized confidence is inconsistent (ECE 0.10--0.54; failure AUROC 0.48--0.77), and a development-tuned confidence-only policy yields no utility gain. A transparent workflow-aware controller improves 5:1 utility for every model and reduces missed escalations to 0.12--0.24, at specificity 0.83--0.93. However, a 30-concept lexical stress test reduces each rule detector's recall from 1.00 on trigger phrases to 0.00 on unseen paraphrases, limiting the controller claim to this benchmark. Seed-clustered statistics, ablations, cross-task transfer, raw outputs, and manifests are packaged for audit. The results motivate explicit workflow-state evaluation while showing that apparently effective routing controls need their own generalization tests.

0.46

Following Delegation Traces to Study Human Oversight in Interaction with Multimodal AI

cs.HC cs.AIcs.CV ·2026-10-15 ·Agnese Chiatti, Matteo Matteucci, Viola Schiaffonati
survey similarity 0.59
Abstract

As multimodal AI systems are becoming our everyday thought partners, the shift towards studying how users transfer control to these systems becomes especially urgent. In this paper, we report an overview of findings from an online study with n=56 AI-literate participants and 392 Video Question Answering trials. After inspecting each multimodal AI response, participants rated task-specific trust and selected one of three intended policies for an analogous future task: reject AI assistance, use it with verification, or delegate fully to the AI. Within participants, post-interaction trust strongly predicted the user's delegation choices (OR=14.97, 95% CI [7.14,31.37]). Trust and delegation nevertheless remained distinct. Verification was the dominant policy and often co-occurred with high trust. Delegation also increased with task order after adjustment for momentary and participant-level trust (OR=1.22, 95% CI [1.02,1.45]), indicating an experience-related component that self-reported trust ratings alone could not capture. These findings motivate a shift from intended delegation to a non-intrusive observation of how delegation is enacted in collaborative human-AI tasks. We therefore conclude by proposing enacted delegation traces as a methodological framework for future studies of multimodal and agentic AI use.

0.13

When Clinical Judgment Becomes a Predictive Shortcut: Two Questions Clinical AI Must Answer Before Deployment

cs.AI ·2026-09-22 ·Jing Yang, Long R. Jiao, Zongjiu Zhang, Xiujun Cai
similarity 0.58
Abstract · 27 pages, 1 figure, 3 tables, 1 box; includes Multimedia Appendix 1

Clinical AI can predict outcomes using information generated by existing clinical judgment. Such performance alone cannot establish usefulness before recognition or across care processes. We propose evidence requirements matched to intended use and information conditions, with additional state-reference and proxy-relative timing tests for stronger recognition claims. This framework helps hospitals assess whether the conditions supporting performance hold when assistance is most needed. These questions remain outside current evaluation practice leakage checks, overall predictive performance, and external validation do not address them; the framework makes them explicit and specifies the direct evidence required to answer them.

0.19