CS Paper Radar

Designing PULSE: A Realtime Annotation Tool to Support Simulation Debriefing

Caleb Vatral, Jennifer Hunt, Mary Ann Jesse, Gautam Biswas

cs.HC ·submitted 2026-08-10 15:18 UTC · v1

Abstract

Debriefing is central to effective simulation-based education. However, effective debriefing is challenged by high instructor workloads and limited engagement of observing students. A real-time annotation tool to support debriefing, called PULSE, was co-designed with nursing educators. A field study comparing three standard simulation debriefings with three debriefings using PULSE was conducted as a preliminary evaluation. Outcomes were assessed using the Debriefing Assessment for Simulation in Healthcare (DASH) student survey and a follow-up instructor interview. PULSE significantly improved overall DASH scores (t(4) = 4.03, p = 0.027, Cohen's d = 2.05). Survey findings suggested improvements in debriefing organization and depth of reflection. Interview data indicated that the student-generated annotations enhanced engagement and stimulated more interactive discussions. PULSE shows promise as a support tool for debriefing, particularly by facilitating reflection of observing students. However, larger and more diverse studies are needed to confirm effectiveness and refine the system for broader implementation.

Comment
22 pages, 2 figures, 1 table, Currently Under Review

Closest papers

Nearest neighbours by meaning across the whole index.

Iterative Design and Evaluation of a VR Nursing Training System: Insights from Clinicians and Student Learners

cs.HC ·2026-10-13 ·Danhua Zhang, Cynthia Bradley, Christopher Kissell, Courtney Hutton Pospick, Seraphina Yong +3 more
similarity 0.58
Abstract · 20 pages, 6 figures. Preprint; under review

This paper presents the iterative design and evaluation of a virtual reality (VR) nursing training system developed in collaboration with nursing education specialists to enhance procedural skill acquisition, clinical decision-making, and critical thinking among pre-licensure nursing students. Two clinical scenarios, chest pain and oxygen therapy, were created to simulate acute care settings, with design refinements informed by an expert evaluation study (n=8) and further assessed through an effectiveness study involving nursing students (n=17). Results demonstrate high user acceptance, perceived usefulness, and significant post-simulation increases in learner confidence. The VR system was found to be an accessible supplemental training tool that can foster self-directed learning. Finally, we present design recommendations for developing VR simulations for nursing education, emphasizing collaboration with domain experts, accommodating diverse learning goals, and simulating complex clinical workflows.

0.29

Micro-Randomized Trial of an AI-Simulated Practice Tool for Therapeutic Skills

cs.HC cs.CY ·2026-08-25 ·Ryan Louie, Ellen Converse, Diyi Yang, Emma Brunskill
published similarity 0.54
Abstract · Appeared in the Impactful and Responsible AI Systems for Education workshop at the Festival of Learning (PMLR RAISE) 28 June 2026. 27 pages …

Though LLM-simulated practice may support psychotherapy education, open questions remain about which system features provide the most educational value. We developed an AI-simulated practice tool for use in psychotherapy classrooms that provides speaking practice with LLM-based patients, followed by a post-practice activity that includes feedback from a fine-tuned LLM and written reflection exercises. We deployed the tool in a graduate psychotherapy course (n=25, 5 weeks) using a micro-randomized trial (MRT) - a method which can estimate the causal excursion effect of an intervention by leveraging longitudinal repeated randomization with participants. Testing four conditions crossing AI feedback (present/absent) with reflection granularity (utterance/session-level), we found surprising interaction effects on student engagement and educational value: utterance-level reflection paired with AI feedback significantly outperformed all conditions. Students valued comparing the AI suggestions to their own rather than passively accepting them. As the first MRT in a health education setting testing an LLM-simulation system, this work demonstrates that MRTs, a method mostly used in mobile intervention research, offers a viable evaluation paradigm for AI systems deployed in health education settings.

0.25

Critical Thinking with Generative AI: A Constraint-First Design Pilot of a Thinking-Partner Intervention

cs.HC ·2026-09-29 ·Fatima Tuz Zahra, Jiangen He, David M. Bowers, Wei Wang
similarity 0.54
Abstract · 53 pages, 5 figures

Generative AI (GenAI) tools entered higher education classrooms faster than the field was able to study their effects on learning. One concern is that GenAI may displace the critical thinking and AI literacy that students will need after graduation. This paper reports a Design-Based Research pilot of a GenAI-assisted critical thinking framework, in which ChatGPT was used as a thinking partner in an undergraduate research methods and statistics course during Spring 2025 (N = 14). The mixed-methods design combined pre- and post-intervention measures of statistical learning (AASCDM), AI literacy (MAILS), and critical thinking (WGCTA) with instructor field notes, student artifacts, and student-AI interaction logs. Pre-post tests showed gains on every AASCDM dimension and on eight of nine MAILS dimensions, while WGCTA percentiles did not change. Qualitative analysis identified four themes: the ways students positioned the LLM (as answer generator, validator, or co-thinker); the depth of student engagement (procedural vs. conceptual); occasional humanizing of the tool; and the role of curriculum design in shaping each of the prior three. Read together, the findings indicate that one semester of GenAI-assisted instruction can move domain learning and self-reported AI literacy but does not move standardized critical thinking, and that the modal student-LLM relationship is one of validation instead of dialogue. We end with design principles for the next iteration of the framework and implications for research on adaptive and personalized learning.

0.20

When Attention Guardrails Become Barriers to Learning: Towards the Tipping Point

cs.HC cs.CY ·2026-09-30 ·Meenakshi V., Pavani Ayinampudi, Aditya B. M. V., Jinal Gupta, Prakash Hegade +3 more
position similarity 0.53
Abstract · 10 pages, 2 figures, 4 tables

Online learning offers flexibility but lacks the structure of a classroom, where a teacher's presence guides attention. The platform we study restores that structure by monitoring the learner through the webcam during ordinary coursework, interrupting or restarting a video when the learner appears distracted. What such monitoring does to a learner across a whole course, rather than in a single examination, is largely unexamined. We report a convergent mixed-methods study of one monitored course pipeline in a summer internship program. We read two free-text surveys alongside three platform channels. The dropout exit survey yielded 15 analyzable responses; the persisting-learner reflection survey, 36. The channels are camera-verification telemetry (14,529 flags from 448 students), an in-video emotion widget (615 submissions from 273 students), and a mandatory end-of-course survey (up to 634 respondents per item). Neither survey named monitoring, so every mention analyzed here was raised by the respondent. Focus-monitoring was raised by 18 of the 51 free-text respondents: 7 of 15 dropouts and 11 of 36 persisting learners. Among the dropouts who raised it, focus-monitoring was the stated primary cause of departure in 4 of 7 cases. None of the 18 questioned being observed in principle. What learners contest is the misreading of ordinary actions, drinking water or moving the head, and the severity of what follows a flag: a video already watched returns to the start of its segment. These findings identify two targets for redesign: the severity of the response to a flag, and the environment check, which can flag a learner before any content has been seen. We argue that the proportionality of that response marks the point at which an attention guardrail becomes a barrier to learning, the tipping point this study approaches.

0.33

Evaluating Scaffolding-Oriented Multi-Agent Large Language Model System for Clinical Interview Training

cs.MA cs.AIcs.HC ·2026-09-10 ·Luming Yang, Haoxian Liu, Siqing Li, Rong Jia, Yue Xiao +2 more
dataset similarity 0.53
Abstract

Clinical education must prepare medical students to conduct safe and coherent patient interviews under conditions of uncertainty. Traditional standardized patient (SP) training is resource-intensive and difficult to scale. We developed a scaffolding-oriented multi-agent Large Language Model (LLM) AI Standardized Patient (AI-SP) training platform1. The system includes a patient agent for simulated dialog, a tutor agent providing Socratic prompts without disclosing diagnostic information, and a turn-level evaluator agent that monitors clinical progress without revealing summative scores. In a randomized controlled study (N = 100 medical students), participants were assigned to either a multi-agent (MA) scaffolding condition or a control condition. All students completed two learning sessions under their assigned condition followed by an examination conducted in a patient only environment. Performance was assessed using a standardized Objective Structured Clinical Examination (OSCE) based rubric. While no significant difference was observed in final diagnostic accuracy between groups, the multi-agent AI standardized patient system improved final examination scores compared to the control group utilizing structured progressive information disclosure; the most substantial and consistent improvements were observed in communication, the expression of empathy, and specific history-taking behaviors. These findings suggest that specialized LLM agents enhance the process quality of simulated clinical interviews without artificially inflating examination outcomes. To support future research, we release a multi-expert annotated dataset comprising transcripts, checklist annotations, turn-level evaluations, and OSCE-aligned scoring outcomes. This resource aims to facilitate the development of pedagogically grounded AI-SP systems and advance research on AI-supported clinical reasoning training.

0.09

Critsly and StudioCrit: An Artefact-Aware AI Critique Workspace and Simulation-Based Readiness Study for Design Education

cs.HC ·2026-09-07 ·Nizam Kadir
similarity 0.52
Abstract · 13 pages, 8 figures, 4 tables. Technical report adapted from an SMT 99.580 research project submitted on 23 July 2026. Documents StudioCrit …

Critique in design education depends on interpreting work in progress, articulating intentions and translating feedback into revisions. This technical report presents Critsly, an artefact-aware AI critique workspace, and StudioCrit, its architecture-studio research mode. Critsly combines a visual board, design-intention fields, guided reflection, perspective-based critique and action planning. StudioCrit adds studio/class organisation, role-based access, cognitive and architectural classification, educator analytics and exportable evidence. The report consolidates implementation and simulation evidence recorded in a research project submitted in July 2026. Three simulated studio scenarios yielded 109 classified evidence rows, including 85 assigned to higher-order Bloom categories. A separate rehearsal using 50 disposable learner accounts yielded 56 evidence rows, including 46 assigned to higher-order categories. A subsequent hardening rehearsal recorded 50 completed sessions, 50 successful board pulls and 50 denials of student access to analytics. These are software and synthetic-trace observations, not measurements of learning gains or human cognitive performance. Automated classifications remain provisional, and the source report does not establish classifier accuracy or inter-rater reliability. The contribution is an implemented critique-to-evidence workflow and a bounded account of its readiness for further controlled evaluation.

0.43

When Should a Virtual Patient Ask? RISP-Guided Proactive Information Seeking in Multi-Agent Patient Simulation

cs.MA ·2026-11-13 ·Jingyao Cai, Shuaijun Liu, Abdul Rehman, Xiaowan Li, Qintian +3 more
similarity 0.52
Abstract

Virtual patient (VP) systems support nursing communication training by providing simulated encounters in which learners practise explaining care and responding to patient concerns. Although large language models (LLMs) support flexible dialogue, current approaches give less explicit attention to how patients' information needs develop and guide the content and timing of their questions. We draw on the Risk Information Seeking and Processing (RISP) model, which relates information seeking to perceived risk and the gap between perceived knowledge and desired understanding. Guided by this perspective, we propose the RISP Multi-Agent System (RISP-MAS), a VP system for patient-initiated information seeking. The system links ongoing assessment of the patient's perceptions and information needs to coordinated decisions about whether to seek information, when to ask and which uncertainty to address. Evaluation of nursing student-VP interactions suggested that RISP-MAS exhibited plausible patient information-seeking behaviour, as indicated by expert assessments, while student feedback reflected positive interaction experiences. This work extends patient simulation by connecting a psychological account of information needs to explicit decisions about question initiation. The approach offers nursing students opportunities to practise recognising unresolved patient concerns and tailoring explanations to patients' information needs.

0.17

Inspectable LLM-Simulated Participants for Qualitative Interview Rehearsal

cs.HC ·2026-10-19 ·Hangyue Zhang, Qiran Hu, Ziyi Zhang, Hyanghee Park, Yun Huang
similarity 0.52
Abstract

LLMs are increasingly used across research workflows, including research ideation, qualitative analysis, and participant simulation. Within work on participant simulation, prior studies have largely evaluated simulated participants by how realistic or human-like their responses appear, leaving less known about how researchers use and critique these materials when revising interview protocols before fieldwork. We present Rehearsim, an inspectable research probe that treats LLM-simulated participants as materials for methodological rehearsal rather than substitutes for human participants. Rehearsim combines literature-grounded participant construction, interactive interview rehearsal, and traceable thematic review while keeping protocol decisions under researcher control. In a study with 21 qualitative researchers, participants used simulated materials to surface assumptions and gaps, reconsider participant populations, revise target populations, and develop follow-up probes. They also identified risks, including the concealment of weak questions, dismissal of unexpected synthetic details, and interpretive anchoring from generated themes. We derive an inspectable-rehearsal framework emphasizing traceability, contestability, researcher control, and epistemic boundaries.

0.20