CS Paper Radar

Reaching the last mile? How Design Decisions Shape Medical Device Qualification and Classification of Patient Support Chatbots in the EU

Milo Marsfeldt, Frantisek Kasl, Michal Koščík, Pavel Loutocky, Vaclav Stupka, Robert Mikulik, Hendrik Knoche

cs.HC ·submitted 2026-08-25 09:18 UTC · v1
published

Abstract

Few AI chatbots have transitioned the 'last mile' from research prototypes to clinical practice, partly because mapping conversational design decisions to regulatory requirements is complicated. This paper presents a design-oriented regulatory analysis of a chatbot supporting stroke survivors. Using Design Space Analysis operationalised through the QOC (Questions, Options, Criteria) framework, HCI researchers and legal scholars assess how variations in three conversational functions - health Q\&A, recovery monitoring, and motivating medical adherence - influence qualification and classification under the European Medical Device Regulation (MDR), General Data Protection Regulation (GDPR), and AI Act (AIA). Our findings show that seemingly minor design decisions - such as role, actionability, and proactivity - influence regulatory obligations. The paper contributes insights into ten conversational design considerations influencing regulatory interpretation and demonstrates how QOC can support integrating legal considerations into human-centred design workflows. Finally, we suggest compliance-aware pathways to market, while reflecting on whether all chatbots should pursue the 'last mile'.

Comment
25 pages, 2 figures

Closest papers

Nearest neighbours by meaning across the whole index.

"It's still 100% on me": Designing AI Tools for the Intersection of Physicians' Care and Legal Infrastructures

cs.HC ·2026-11-11 ·Gennie Mansi, Julia Kim, Lynn Dombrowski, Mark Riedl, Naveena Karusala
similarity 0.68
Abstract · 27 pages, 0 figures

Laws and regulations deeply shape physicians' workflows and impact how they incorporate AI-based decision support tools to care for patients. Despite calls to comply with legal standards, many AI-based decision support tools have not transitioned well to clinical settings. The medical community remains concerned about their legal liability, risks to patients, and limited clinical value of such tools. To better understand how to address these concerns in the design of AI tools for clinical decision-making, we conducted 19 interviews with physicians in the U.S. We used scenario-based design to investigate physicians' legal concerns in the context of an AI-based drug recommendation tool. Drawing on the ethics of care, we surface connections between physicians' care and the healthcare laws and regulations that deeply shape their information needs. We describe how physicians wanted to use the tool in individual and collective practices to care for themselves, their patients, their colleagues, work institution, and even the broader medical profession. We discuss how the design of AI-based decision support tools can present legal ambiguities that would prevent their caring actions and lead to non-use of the AI tool. Through our work, we contribute to responsible AI development in healthcare. Our analysis illuminates why and how certain kinds of information reduce legal ambiguity and support physicians' care when using AI-based decision support tools. We contribute an understanding of how to help physicians navigate legal liability and draw implications for grounding AI tools' designs in their care.

0.37

"I Felt Very Seen, But Still Very Alone": Longitudinal Trajectories of General-Purpose LLM Use for Socioemotional Support

cs.HC ·2026-09-11 ·Meryl Ye, Briana Vecchione, Livia Garofalo, Ranjit Singh
similarity 0.63
Abstract

People increasingly use general-purpose chatbots such as ChatGPT, Claude, and Gemini for mental health and emotional support. We report a multi-stage longitudinal qualitative study of 18 U.S. adults, conducted from April to December 2025, combining initial interviews, a four-week diary study, focus groups, and exit interviews. We find that socioemotional use often emerged gradually out of practical use and when other forms of support were unavailable. Participants developed routines and boundaries around chatbot use, which were disrupted by model updates, evolving public discourse about AI harms, and changes in personal circumstances. We demonstrate how longitudinal study captures factors beyond the human-AI dyad, and argue that HCI researchers and designers should account for users' histories with their chatbots and broader care ecologies when evaluating AI systems over time and introducing updates that may disrupt established sources of support.

0.20

Open Problems in Chatbots and Mental Health

cs.CL cs.CY ·2026-11-07 ·Jared Moore, Jacy Reese Anthis, William Agnew, Andrea Mock, Kevin Klyman +6 more
positionhot direction similarity 0.62
Abstract

General-purpose chatbots are increasingly used for emotional and mental health support. They can be useful, but issues such as sycophancy, reinforcement of delusional beliefs, and poor crisis response create risks that are difficult for researchers to study and evaluate. In this paper, we review the most critical open problems for researchers and policymakers in the use of general-purpose chatbots and their effects on mental health. First, there are tractable problems in the controlled evaluation of chatbots: establishing the construct and ecological validity of evaluations, evaluating multi-turn interactions, and connecting evaluations to improvements in chatbot behavior. For example, conventional benchmarks that primarily rely on static datasets cannot establish chatbot safety, and novel simulation methods need validation. Second, researchers must connect their studies of chatbots and mental health to real-world evidence. This includes analyzing the chatbot products that AI companies and mental health practitioners have deployed, including their user interfaces and human review and escalation practices for user welfare concerns. However, clinical analysis opportunities afforded by real chat log data sharing are complicated by significant privacy concerns. Finally, we argue that researchers should take an active role in the construction and refinement of evidence-based policy through corporate and governmental decisions such as user protections, standards of care, protected access to user chat logs, data sharing between institutions, incident reporting, and post-deployment monitoring. Such behavioral evaluations, studies of real-world outcomes, and data governance depend on each other and must be addressed in tandem. We believe a coordinated research effort would have substantial real-world impact.

0.22

Between Obligations and Evidence: A Participatory Study of Translating the EU AI Act into Technical Requirements

cs.CY cs.AIcs.HCcs.SE ·2026-11-13 ·Helma Torkamaan, Masoud Mansoury, Yanan Xin, Oscar Oviedo Trespalacios, Roberta Savella +7 more
similarity 0.62
Abstract

The EU AI Act imposes technical and organizational requirements for high-risk AI systems, but applying these requirements involves contextual choices about what to measure and what evidence supports a compliance claim. We investigate these challenges through a participatory workshop involving 32 researchers, policymakers, and practitioners from different disciplines, using realistic scenarios in healthcare, recruitment, and public transport to examine how regulatory obligations are interpreted, translated into technical requirements, and evaluated in practice. Through a thematic analysis of workshop data, we identify recurring tensions between how regulatory obligations are translated into measurable system properties, how responsibility and human oversight are exercised in practice, and how the validity of compliance evidence is affected by changing operational conditions and system uses. For example, participants highlighted how fairness metrics could conceal discrimination in recruitment, how time pressure and limited information could undermine human oversight in public transport, and how changes in patient populations could invalidate previously established evidence of system performance. Our findings suggest that technical, organizational, and evidentiary challenges are interdependent, and that establishing and maintaining compliance requires attention not only to technical assessments but also to how AI systems are used, governed, and monitored over time. We discuss implications for supporting context-sensitive and ongoing compliance.

0.27

Personalizing Personal Health Interfaces: Co-Design with Generative AI

cs.HC cs.AIcs.CY ·2026-09-14 ·Karthik S. Bhat, Vidhi Shah, Vedika Agnihotri, Dong Whi Yoo, Koustuv Saha
similarity 0.60
Abstract

Personal health interfaces present wellbeing data through standardized dashboards that rarely fit how people interpret or act on it. Personalizing them to what people would like to see for themselves often requires design and technical expertise, a barrier that generative AI may potentially lower. Therefore, we ask what designs emerge and how it enables and constrains the design process. We conducted a co-design study where 14 participants redesigned Google and Apple Health interfaces using Figma Make. Participants reimagined interfaces that supported personal context, future planning, and interactive experiences, yet conversational AI designs converged around chat-window conventions. AI helped materialize loosely articulated ideas, but model defaults and generation latency shaped iteration. The process more readily operationalized interpretability and accountability than privacy, trust, and emotional safety. Generative co-design let participants create interfaces directly, blurring the boundary between intentions and model defaults. We discuss implications for preserving agency and flexible user-directed interfaces.

0.25

Verify, Offload, Extend & Recommend: Selective Complementarity in AI Support for Physical Activity Planning with Longitudinal Patient Data

cs.HC ·2026-09-16 ·Pavithren V S Pakianathan, Rania Islambouli, Diogo Branco, Gil Batista Rosa, Rita Pinto +3 more
similarity 0.60
Abstract

Self-tracking technologies create longitudinal patient-generated health data, yet integrating these data into clinical decision-making can increase information-processing demands. Generative AI may support sensemaking, but its value depends on clinical context and expertise. We investigate AI augmentation of a clinical decision support system for physical-activity planning in cardiovascular disease. In a counterbalanced within-subjects study, 26 exercise physiologists developed plans for four real cardiovascular cases with and without AI support, followed by evaluation of an AI exercise-plan generator. AI did not significantly improve workload, usability, confidence, or plan quality overall; however, its effect on plan quality increased as visualization literacy decreased and its effect on workload increased as visualisation literacy increased. Interviews and 152 chatbot queries revealed three recurring uses: verifying, offloading, extend and generate. Our findings position AI support as a selective complement to professional expertise while highlighting validation challenges when clinicians seek support precisely where their own knowledge is limited.

0.29

"I Just Don't Want to Be a Burden": Investigating AI Chatbot Emotional Support Among People with Suicidal Ideation and Other Mental Health Concerns

cs.HC cs.CY ·2026-11-17 ·Vasudha Varadarajan, Nathan S. Fishbein, Andrew Aquilina, Jordan Taylor, Hong Shen +1 more
survey similarity 0.60
Abstract · 41 pages, 9 figures

People experiencing high-acuity mental health concerns are turning to AI chatbots for emotional support, but little is known about what drives this use, their experiences, and their preferences. In a sequential mixed-methods study, we inductively analyzed Reddit posts and used the findings to develop a survey. We surveyed 202 adults with recent mental health concerns who had used AI for emotional support. Most participants were already receiving clinical care and reported turning to AI, most commonly because they did not want to burden others, rather than being driven by barriers to care. Benefits were broadly shared, but participants reporting past-year suicidal ideation described more harm-facilitating AI behavior and reliance than those without. Overall, participants ranked honesty above warmth, and preference for crisis escalation declined as clinical acuity rose. These findings motivate AI support that addresses perceived costs of disclosure, evaluates benefits alongside harms, and widens feasible paths to human connection.

0.23

"It Feels Weird to Just Rely on AI": How Experienced Healthcare Professionals Work With Decision Support Systems in Clinical Practice

cs.HC ·2026-10-29 ·Simon W. S. Fischer, Hanna Schraffenberger, Miranda L. van Hooff, Casper Reijnen, Serge Thill +1 more
similarity 0.60
Abstract

Decision support systems (DSS) assist with clinical decision-making. Little is known, however, about the decision-making process with DSS in clinical practice and the extent to which clinicians consider or rely on the predictions. Studies on human-AI decision-making usually focus on laypeople, simplified decision tasks, or one-shot interactions with prototypes that are not integrated into existing workflows. To gain insights from real-world practice, we conducted an online survey (n=12) and semi-structured interviews (n=10) with clinicians from two different medical cohorts each working with a DSS. Findings show, amongst other things, that the interaction with DSS changes as experience increases, the predictions become more important in complex cases, that decisions are discussed with colleagues, and that workflow integration affects decision-making. Our findings on human, technical, contextual, and organisational factors contribute to a socio-technical perspective on DSS reliance behaviour. We conclude by discussing the implications of our findings for the design of DSS.

0.21