CS Paper Radar

NeuroClick: Preserving Surgeon Autonomy through Hands-Free Earable Tooth-Click Control in Neurosurgery

Jonas Hummel, Maximilian Burzer, Clara Sayffaerth, Valeria Zitz, Amir El Rahal, Michael Küttner, Tobias Röddiger, Jürgen Beck, Michael Beigl

cs.HC ·submitted 2026-09-11 14:36 UTC · v1

Abstract

Neurosurgeons frequently interact with operating room (OR) technologies while sterility and occupied hands constrain control. We introduce earables as a direct, hands-free control platform for neurosurgery using tooth-click input. Formative OR observations and interviews with 10 domain experts grounded the design. Using OpenEarable 2.0 data from 12 participants, we developed a real-time recognition pipeline whose classifier achieved a median macro F1-score of 98.6% under leave-one-subject-out cross-validation. We evaluated the technique with 20 neurosurgeons during a simulated resection task in a neurosurgical OR. Participants reported few focus shifts and rated Earable favorably for workflow integration and perceived safety. Autonomous microscope control was rated significantly higher with Earable than Delegation, whereas Delegation enabled faster task completion under continuous assistant availability. Workload, usability, and task errors showed no significant differences. Preferences depended on training, reliability, context, and assistant availability. Earables thus add a direct, hands-free option for controlling selected functions alongside established workflows.

Closest papers

Nearest neighbours by meaning across the whole index.

SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition

cs.RO ·2026-10-18 ·Parham M. Kebria, Saeid Nahavandi, Mehrdad Saif, Fernando Bello
similarity 0.63
Abstract

The assessment of surgical skills in robotic surgery training has traditionally relied on subjective evaluations by human experts, creating bottlenecks in surgical education and limiting scalability. We present a novel foundation model that automatically analyzes robotic surgical videos to provide comprehensive skill assessment and gesture understanding. Combining a visual encoder with a transformer architecture, the model performs three key tasks simultaneously: 1) surgical task classification (knot tying, needle passing, suturing), 2) skill level assessment on a standardized scale, and 3) natural language generation of surgical gesture descriptions. The model processes video sequences from the da Vinci surgical robot and generates human-readable text descriptions of surgical actions, similar to how language models generate text from prompts, using a multi-task learning framework with adaptive uncertainty weighting to optimize performance across all objectives. Our transformer-based architecture incorporates positional encoding for temporal understanding and attention mechanisms for spatiotemporal feature integration. Evaluated on the da Vinci Skills Simulator dataset, the proposed framework demonstrates state-of-the-art performance in surgical task recognition and provides interpretable skill assessments that correlate with expert evaluations, with the model's ability to generate natural language descriptions of surgical gestures opening new possibilities for automated surgical education, real-time feedback systems, and objective skill assessment in surgical training and certification programs.

0.06

Semantic RGB--Depth Based Surgical Skill Assessment in Microscopic Stereo Videos

cs.CV ·2026-10-01 ·Jecia Z. Y. Mao, Sue M. Cho, Francis X. Creighton, Deepa Galaiya, Russell H. Taylor +1 more
similarity 0.60
Abstract

Objective assessment of microsurgical technical skill is essential for competency-based training and quality assurance, yet existing video-based approaches predominantly rely on RGB images and therefore overlook the 3D spatial relationships that characterize instrument-anatomy interactions. Although stereo operating microscopes provide complementary depth information, conventional stereo matching algorithms can produce sparse and unreliable depth estimates under high-magnification imaging conditions, limiting their use for automated skill assessment. This work presents a semantic RGB-Depth framework for surgical skill assessment from microscopic stereo videos. A regression-based depth fusion method combines sparse metric stereo depth with dense monocular depth estimates to generate a dense geometric representation of the surgical scene. This representation is integrated with semantically decomposed RGB streams corresponding to individual surgical instruments and surrounding anatomy. A hierarchical attention architecture jointly encodes these streams to capture discriminative patterns of instrument use and instrument-anatomy interaction across surgeons at different training levels. The framework was evaluated on 33 ex vivo transoral microlaryngeal procedures performed by six surgeons, comprising attending surgeons and surgical residents, using leave-one-surgeon-out cross-validation. The proposed semantic RGB-Depth model achieved an F1 score of 0.938 for skill-level classification, compared with 0.696 for semantic RGB and 0.929 for semantic depth. These results suggest that geometric information can improve automated surgical skill assessment from microscopic stereo videos. The learned spatial, temporal, and semantic attention patterns also support qualitative examination of the scene regions, video segments, and semantic streams emphasized by the model.

0.17

A Systematic Analysis of Live Intraoperative Surgical Expert Narrations

cs.CV cs.CL ·2026-10-05 ·Arda Mamur, Matthias Seibold, Christoph Laux, Florian Wanivenhaus, Philipp Fürnstahl
published similarity 0.59
Abstract · Accepted as abstract at the SafeSurg Workshop at MICCAI 2026

Manual annotation remains the principal bottleneck in surgical video understanding. Live intraoperative narration by the operating surgeon is a scalable alternative that requires no manual annotation, but the information it contains has not been characterized. We analyzed 13 endoscopic lumbar spine decompression procedures (13.2 h) by two senior spine surgeons, using time-synchronized close-mic German audio. An annotation-free pipeline (word-level speech recognition, deterministic correction of surgical-term errors, utterance fusion, LLM-based translation to English, and relevance filtering) converted speech into time-stamped procedural cues labelled with the surgical entities they reference and their speech act. Of 3,171 utterances, 1,201 (37.9%) were retained as procedural cues, while 57.6% were non-procedural and removed. Retained cues referenced instruments (60.7%), anatomy (48.0%) and surgical actions (57.1%), and 98.5% carried at least one of these. 39% of cues (472 of 1,201) were directives, mainly instrument requests and commands. All 467 manually verified instrument requests preceded the instrument's appearance in the field, by a median of 8.2 s. Domain-term transcription corruption affected 18.0% of retained cues. Live narration is thus a dense, annotation-free supervisory signal describing instruments, anatomy, actions and team coordination, but it is noisy and temporally displaced from the events it names, and this displacement must be modelled before safe downstream use.

0.32

A Controlled Comparison of Manual and Teleoperated Intraocular Instrument Motion for an Input Device

cs.RO cs.HC ·2026-09-08 ·Korab Hoxha, Mirza Imamovic, Angelo Henriques, M. Ali Nasseri
similarity 0.59
Abstract · This work has been submitted to IEEE Access for possible publication

Input devices for robotic microsurgery are frequently described as preserving the surgeon's trained technique, but the claim is rarely measured. We compared manual and teleoperated intraocular instrument motion with the trocar constraint, the instrument, the eye model and the tracking source common to both conditions, so that the control interface was the only factor varied. Prior comparisons cannot hold the instrument fixed, because a robotic instrument is not the tool used manually. Sixteen participants performed a navigation task on a commercial ophthalmic simulator by hand and through a three-degree-of-freedom input device commanding a five-joint robot. Task outcome was equal but at ceiling: every participant acquired all five targets under both interfaces with no retinal or lens injury. Execution differed on every measure. Teleoperated trials took three times as long at a quarter of the median speed, covered less than half the angular working range, and were broken into 3.5 times as many separate movements. Completion time and movement fragmentation improved substantially across four trials of practice and had not plateaued; the measures set by the configured rate ceiling and joint limit changed the least. Finger activity doubled and pinch variability tripled, so reducing instrument degrees of freedom redistributed manual effort rather than reducing it. The interface preserves the outcome and reshapes the execution.

0.30

Scene Graph-Driven Haptic Feedback for Safety Enhancement in Robotic Ophthalmic Surgery via Physically Simulated iOCT

cs.RO cs.CV ·2026-09-07 ·Danial Arbabi, Korab Hoxha, Angelo Henriques, Mirza Imamovic, M. Ali Nasseri
published similarity 0.59
Abstract · 7 pages, 9 figures, 2 tables. Accepted at the 2026 IEEE RAS/EMBS 11th International Conference on Biomedical Robotics and Biomechatronicsnic…

Robotic ophthalmic surgery offers high precision but introduces a "sensory gap" by decoupling the surgeon from their instrument, resulting in a loss of tactile feedback. This paper presents a novel haptic feedback system for subretinal injection tasks leveraging Scene Graphs (SG). The system bridges the sensory gap by analyzing a physically simulated intraoperative Optical Coherence Tomography (iOCT) feed to construct a real-time surgical SG. The SG serves as a semantic abstraction layer for the surgical scene, which is then utilized by a deterministic, rule-based engine to generate state-dependent haptic feedback on a robotic input device. The system was evaluated in a user study (N=16) using an anthropomorphic head phantom and a custom-built surgical robot. Results demonstrate that the SG-driven haptic feedback improved surgical precision, reducing needle alignment error by 14% (p = 0.044) and improving System Usability Scale (SUS) scores by 8% (p = 0.015), while maintaining comparable task completion times. A needle trajectory analysis revealed the emergence of a safer "Align-then-Approach" strategy, in which our haptic negative reinforcement prompted users to fine-tune the tool's trajectory before approaching the retinal target. This work suggests that SGs can effectively serve as the direct computational foundation for real-time, safety-enhancing context-aware haptic feedback in robotic microsurgery.

0.40

Constructing Predictive Surgical Path for AI-based Capsulorhexis Skill Transfer

cs.RO cs.AIcs.LG ·2026-08-18 ·Mohammad Javad Ahmadi, Hamid D. Taghirad
hot direction similarity 0.59
Abstract

Automated training of surgeons is one of the most crucial factors that significantly minimize surgical training risks and expenses. With recent advances in artificial intelligence (AI) knowledge and available data from various surgeries, AI's involvement in surgical training is becoming very promising. It is recommended that at the early stages of AI development, it interferes in the surgery as a third agent alongside the trainer. As trust in AI increases, this process will lead to an AI agent acting as a trainer in the future. The first phase in which AI can intervene in the training process is to suggest an improved surgical path to the trainer. A platform must be constructed in the first step, to accomplish this task and to enhance the movement path of trainee surgeons. This paper introduces this platform along with an annotated capsulorhexis surgery dataset called the ARAS-Farabi dataset. In this research, a deep convolutional neural network is pre-trained with JIGSAWS and ARAS-Farabi surgical datasets that can extract surgical skill characteristics from surgery tool tip motion data. The proposed platform develops a reference model from the feature space of an expert surgeon's movement trajectory and proposes an improved path to enhance the skill of a novice surgeon. An optimization with two loss functions is utilized to create a path that raises the skill level of the novice surgeon's path while simultaneously predicting and preserving his/her intent. The results of this study reveal that, with the assistance of an AI agent, the trainee surgeon's movement path can be enhanced by at least 20 percent while maintaining his intentional objective. In addition to the recommended deep network, various tangible indicators have also been developed in this research to verify the level of trainee improvement.

0.27

Video Based Assessment of Surgical Skills Using Frozen Pretrained Video Foundation Models

eess.IV ·2026-09-17 ·Sangrock Lee, FNU Rahul, Suvranu De
publishedbenchmark similarity 0.59
Abstract · Accepted for publication in npj Digital Medicine

Automated video-based surgical skill assessment has advanced rapidly, yet rigorous evaluation of continuous standardized score prediction under participant-level generalization to unseen trainees remains limited. We introduce VBA-Net+, a video-only framework for Fundamentals of Laparoscopic Surgery (FLS) score regression and pass-fail classification using pretrained video foundation models as frozen feature extractors. We evaluate two FLS datasets, suturing and pattern cutting, using VideoPrism, V-JEPA2, and VideoMAE v2, with a frame-level SimCLR baseline. A lightweight fully convolutional head is trained on embeddings offline and evaluated using participant-level leave-one-user-out (LOUO) cross-validation within the standardized assessment protocol. For continuous score prediction, the best representation achieves $R^2$=0.6367 for suturing and 0.9261 for pattern cutting. For pass-fail classification at official FLS thresholds, area under the receiver operating characteristic curve (AUC) reaches 0.9073 and 0.9906, respectively. Frozen video-encoder pipelines generally outperformed the frame-level SimCLR pipeline, particularly for suturing, providing a benchmark for video-only FLS assessment.

0.24

From Instrument-Mounted Demonstrations to In-Vivo Execution: Learning Bimanual Laparoscopic Appendectomy Without Robot-Collected Demonstrations

cs.RO ·2026-09-22 ·Dongho Yee, Juahn Oh, Jinseok Lee, Jiyul Lee, Yechan Seo +8 more
code/data similarity 0.59
Abstract · Submitted to IEEE ICRA 2027. Project page & Dataset: https://rosota-research.github.io/instrument-mounted-demonstrations-in-vivo/

Most minimally invasive surgery is still performed with hand-held laparoscopic instruments, and the surgeon's instrument kinematics are lost when the operation ends; only the endoscope video is kept. This paper presents an end-to-end pipeline that captures this motion in the operating room and uses it to train a surgical robot policy, validated on live animals. We introduce a surgical instrument-state logger that mounts on the shaft of a standard laparoscopic instrument and recovers its pose and jaw state from an inertial sensor, a time-of-flight sensor and a Hall sensor, with no external camera or tracker. A data pipeline measures the latency of every sensor channel against a robot ground truth and aligns the channels before forming observation-action pairs. On these demonstrations we train a diffusion policy with a fine-tuned DINOv3 backbone, selecting its design by closed-loop rollouts in a physics simulator reconstructed from depth maps of an ex-vivo rabbit appendix. The policy is then retrained on 849 in-vivo demonstrations from four live rabbits and deployed on four additional live rabbits with electrosurgery armed. With the surgeon selecting the surgical phase, the policy completed the appendectomy in three of the four animals. The results show that demonstrations recorded from a surgeon's own instruments are sufficient to train, select and deploy a bimanual surgical policy in vivo. The robot serves only as the timing reference for sensor calibration and as the executor, and collects no demonstrations. Both demonstration corpora are released to support future surgical robot learning research.

0.34