CS Paper Radar

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

Chaohui Dang, Zheheng Jiang, James Glasbey, David Luke, Theodoros Arvanitis, Le Zhang

cs.CV ·submitted 2026-08-31 14:32 UTC · v1
benchmark

Abstract

Objective assessment of surgical technical skill is important for surgical training and structured feedback, but current workflows remain dependent on labor-intensive expert review. Existing automated approaches primarily focus on visual inputs and provide limited support for jointly studying operative performance, structured skill scores, and evaluator feedback. We introduce SurgSkill-Bench, an initial video-score-text benchmark-style dataset containing 214 surgical training simulation videos, six-dimensional OSATS scores, and expert free-text comments. We define two evaluation settings: video-only OSATS prediction for automated assessment and post hoc expert-comment-assisted prediction, where evaluator comments are available as auxiliary information. We provide controlled baseline experiments using representative frozen visual backbones, content-adaptive key-frame sampling, and a simple video-text co-attention fusion module. Under internal video-level validation, content-adaptive sampling improves video-only performance in this dataset, while evaluator comments provide additional score-related signal in the assisted setting. The best mean AUROC reaches 0.88 under dataset-specific median dichotomization. We further discuss evaluation constraints related to dataset scale, metadata completeness, and the interpretation of comment-assisted prediction. Code will be released publicly at a later date.

Closest papers

Nearest neighbours by meaning across the whole index.

SurGPT: A Multi-Task Foundation Model for Robotic Surgical Skills Assessment and Gesture Recognition

cs.RO ·2026-10-18 ·Parham M. Kebria, Saeid Nahavandi, Mehrdad Saif, Fernando Bello
similarity 0.78
Abstract

The assessment of surgical skills in robotic surgery training has traditionally relied on subjective evaluations by human experts, creating bottlenecks in surgical education and limiting scalability. We present a novel foundation model that automatically analyzes robotic surgical videos to provide comprehensive skill assessment and gesture understanding. Combining a visual encoder with a transformer architecture, the model performs three key tasks simultaneously: 1) surgical task classification (knot tying, needle passing, suturing), 2) skill level assessment on a standardized scale, and 3) natural language generation of surgical gesture descriptions. The model processes video sequences from the da Vinci surgical robot and generates human-readable text descriptions of surgical actions, similar to how language models generate text from prompts, using a multi-task learning framework with adaptive uncertainty weighting to optimize performance across all objectives. Our transformer-based architecture incorporates positional encoding for temporal understanding and attention mechanisms for spatiotemporal feature integration. Evaluated on the da Vinci Skills Simulator dataset, the proposed framework demonstrates state-of-the-art performance in surgical task recognition and provides interpretable skill assessments that correlate with expert evaluations, with the model's ability to generate natural language descriptions of surgical gestures opening new possibilities for automated surgical education, real-time feedback systems, and objective skill assessment in surgical training and certification programs.

0.06

SurgQ-VLM: Question-Conditioned Dual-Stream Compression and Answer-Type-Aware Reinforcement for Surgical Video Question Answering

cs.CV ·2026-11-01 ·Dhruv Makwana, Ananya Mantravadi
publishedcode/data similarity 0.76
Abstract · Accepted to WACV 2027 (Applications track). 16 pages: 8-page main paper, references, and supplementary material. Code: https://github.com/Dh…

Reasoning about surgical video -- which phase is underway, which instruments act, and when events occur -- underpins operative documentation, workflow analysis, and skill assessment. Yet surgical visual question answering is still largely built on single frames and closed labels that cannot express a duration, an ordering, or a count, and the video models that lift the frame limit typically compress every clip the same way regardless of the question, discarding the temporal evidence the answer needs. Worse, the answers are structurally heterogeneous, spanning five incompatible structures -- binary, classification, multi-label, counting, and sentence-temporal -- that no single training signal scores uniformly well. We propose SurgQ-VLM, a two-stage model for surgical video question answering. Stage 1 trains a question-conditioned dual-stream (slow/fast) compressor for a LoRA-adapted Qwen2.5-VL-7B: cross-attention conditions both streams on the question, and each clip's visual tokens shrink 49x from the raw ViT patches (11x vs. post-merge backbone tokens). Stage 2 post-trains with GRPO under an answer-type-aware reward that scores each question on the metric matching its answer structure (micro-F1 for label sets, normalised error for counts, a content-timing blend for temporal windows). On a sealed 21,840-question test split over Cholec80 and CholecT50, SurgQ-VLM reaches 0.779 keyword accuracy and a 10x BLEU-4 gain over zero-shot; GRPO adds 2.5 pp K-Acc (2.0 pp attributable to the answer-type-aware reward), concentrated in the structurally hard types, with answer format unchanged (ROUGE-L). Code is available at https://github.com/DhruvMakwana/SurgQ-VLM.

0.37

Toward Cross-Rubric Surgical Skill Assessment via Joint Embedding Prediction

cs.CV ·2026-11-11 ·Jiajun Cheng, Haidar M. Abdul-Muhsin, Juha Choi, Shan Lin
benchmark similarity 0.76
Abstract

Expert review of surgical videos is essential for surgical training but labor-intensive, motivating AI-assisted skill assessment to provide feedback at scale. Although surgical assessment rubrics may use different language to describe similar skills, existing automated methods commonly predict predefined skill levels or numerical scores tied to fixed rubrics, limiting their ability to transfer across semantically equivalent assessment criteria. We propose Predictive Rubric Alignment (PRA), a joint embedding prediction approach that scores surgical videos using natural-language descriptions of skill levels. Rather than matching level probabilities, PRA predicts a rater-weighted combination of skill-level description embeddings. Adjacent description features define a score-parameterized path, and the predicted embedding is projected onto this path to read the score. We introduce two variants: \textit{PRA-item}, which uses learned queries for predefined assessment items, and \textit{PRA-cond}, which generates queries from textual item names to enable assessment of unseen items without retraining. On AIxSuture, a dataset of expert-rated suturing videos, PRA outperforms published baselines in skill classification. We further evaluate zero-shot transfer to the smaller surgical skill assessment benchmark JIGSAWS. Trained on only two AIxSuture rubric items, PRA-cond assesses unseen JIGSAWS items without further training, yielding positive score correlations and greater Grad-CAM overlap with instruments and suture than a description-matching baseline. These results support predictive rubric alignment as a promising direction for flexible and explainable automated surgical skill assessment.

0.05

Video Based Assessment of Surgical Skills Using Frozen Pretrained Video Foundation Models

eess.IV ·2026-09-17 ·Sangrock Lee, FNU Rahul, Suvranu De
publishedbenchmark similarity 0.75
Abstract · Accepted for publication in npj Digital Medicine

Automated video-based surgical skill assessment has advanced rapidly, yet rigorous evaluation of continuous standardized score prediction under participant-level generalization to unseen trainees remains limited. We introduce VBA-Net+, a video-only framework for Fundamentals of Laparoscopic Surgery (FLS) score regression and pass-fail classification using pretrained video foundation models as frozen feature extractors. We evaluate two FLS datasets, suturing and pattern cutting, using VideoPrism, V-JEPA2, and VideoMAE v2, with a frame-level SimCLR baseline. A lightweight fully convolutional head is trained on embeddings offline and evaluated using participant-level leave-one-user-out (LOUO) cross-validation within the standardized assessment protocol. For continuous score prediction, the best representation achieves $R^2$=0.6367 for suturing and 0.9261 for pattern cutting. For pass-fail classification at official FLS thresholds, area under the receiver operating characteristic curve (AUC) reaches 0.9073 and 0.9906, respectively. Frozen video-encoder pipelines generally outperformed the frame-level SimCLR pipeline, particularly for suturing, providing a benchmark for video-only FLS assessment.

0.24

EndoComparer: Comparative Surgical Video Question Answering with Pseudo-Siamese Visual Adaptation and Perceive-then-Compare Reasoning

cs.CV ·2026-11-14 ·Shi Li, Lalith Sharan, Joël L. Lavanchy, Kun Yuan, Lorenzo Arboit +6 more
benchmarkcode/data similarity 0.75
Abstract · 33 pages, 14 figures, 8 tables

Purpose: Surgical training benefits from explicit comparison between cases: in patient condition, in technique and in skill proficiency. Surgical vision-language models (VLMs) offer a promising and flexible interface for this purpose. Yet current VLMs typically see one video at a time, and they often fall short when asked to compare videos. Methods: We propose EndoComparer, a VLM for comparative surgical video question answering with two components. (a) A Pseudo-Siamese Visual LoRA (PSVL) gives each video slot its own low-rank adapter over a shared frozen vision encoder, separating the two videos in the visual pathway at the cost of a single encoder pass. (b) A Perceive-then-Compare chain of thought (PtC-CoT) trains the model to first state what each video shows, then derive the answer from these statements. We further curate EndoDiffVQA, a benchmark of nearly 11K multiple-choice questions on video-pair over cholecystectomy and gastric bypass procedures, covering anatomy, tool, action and skill. Results: EndoComparer outperforms strong generalist and surgical VLMs on detecting and specifying difference between videos. Ablations show that two proposed components contribute complimentarily, and the model remains more robust under a video-swap control that exposes confusion between the two videos. Conclusion: Providing a dedicated visual reasoning path through PSVL and PtC-CoT makes cross-case comparison of surgical videos more robust for VLMs. EndoDiffVQA and EndoComparer provide a benchmark and a strong baseline for this task. Project page: https://camma-public.github.io/endocomparer

0.19

Explainable AI-Powered Framework for Video-Based Skill Assessment in Cataract Surgery

cs.CV cs.AI ·2026-08-18 ·Mohammad Javad Ahmadi, Hamid D. Taghirad
dataset similarity 0.74
Abstract

Persistent shortages in the surgical workforce and inherent limitations of traditional training methods highlight the necessity of automated, data-driven approaches in surgical education. This study addresses these challenges by introducing a novel, explainable AI-powered framework for automated skill assessment, specifically focusing on cataract surgery. We present the world's largest dataset of cataract surgery videos, comprising 2,000 recordings. Additionally, we propose an AI-powered analytical framework that employs advanced computer vision and signal-processing techniques to automatically evaluate surgical videos to derive objective, quantitative performance indicators that complement or potentially replace subjective scoring methods. A significant advantage of our framework over previous methods lies precisely in its explainability of outputs, elevating it beyond merely an opaque skill classification tool. Through experimental analysis of 83 cataract surgery videos, we demonstrate that the automatically computed metrics exhibit strong correlations with expert-based subjective evaluations, achieving up to 87% accuracy in surgical skill assessment. Each metric was individually examined, and expert surgeons provided subjective ratings using the newly introduced Capsulorhexis Skill Assessment System (CSAS). These subjective assessments were compared with ten objective motion-based metrics extracted through our framework. The results indicated a robust correlation between subjective ratings and automated indicators, underscoring the framework's capacity to accurately model surgical expertise.

0.20

HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding

cs.CV ·2026-10-07 ·Leon Mayer, Lucas Luttner, Patrick Godau, Kai Fritzsche, Annika Reinke +40 more
benchmarkdatasetcode/data similarity 0.73
Abstract · 28 pages, 9 figures, 5 tables. Code: https://github.com/IMSY-DKFZ/orena-focus

Recent advances in Vision-Language Models (VLMs) have led to rapid progress in video understanding across a wide range of benchmark tasks. However, existing evaluations largely focus on short-term reasoning, failing to assess a critical capability: maintaining cumulative temporal consistency over extended time horizons. To close this evaluation gap, we introduce HeiCo-FOCUS, a clinically grounded dataset for evaluating long-context video understanding through the task of Foreign Object Contextual Understanding in Surgery. Built on a dataset of Heidelberg Colorectal surgeries, this task requires models to continuously track multiple objects as they are inserted, manipulated, occluded, and removed over procedures lasting up to hours. HeiCo-FOCUS comprises 30,000 visual question answering (VQA) pairs covering five core capabilities: object recognition, temporal grounding, aggregation, event and procedural understanding, and complex reasoning. The dataset was constructed through a rigorous multi-stage annotation pipeline involving large-scale crowd annotation and 39 surgical domain experts to ensure high quality and clinical relevance. To systematically probe model behavior, we introduce a multi-track evaluation framework that progressively increases temporal and contextual demands from single frames to full procedures. Experiments with ten frontier VLMs show that HeiCo-FOCUS tasks are far from solved: only around half of the models clearly outperform a text-only baseline. Across the video tracks, models perform best on event and procedural understanding (mean Accuracy: 56.5% across all models), while temporal grounding remains particularly challenging for all evaluated models (mean Accuracy: 19.7%). We therefore expect HeiCo-FOCUS to serve as a catalyst for the development of models capable of reliable, temporally consistent reasoning over hours-long videos.

0.27

How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding

cs.CV cs.AI ·2026-11-08 ·Gengyuan Zhang, Xiao Han, Shijie Zhou, Abdullah Koukash, Jingpei Wu +5 more
top venuebenchmarkcode/data similarity 0.72
Abstract · Accepted at NeurIPS 2026, Evaluations and Datasets Track. 64 pages including appendices. Project page: https://vitagent.github.io/

Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited due to scarce surgical data and fragmented evaluation protocols. We address these challenges through a two-stage evaluation framework for surgical scene understanding. In Stage I, we introduce SurgKnowBench, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-source MLLMs to map the boundary of pretrained surgical competence. SurgKnowBench reveals that current models retain partial visual priors but remain weak on relational and procedural understanding and workflow assessment. Building on this failure frontier and motivated by the scaffolded nature of surgical training, we introduce SurgAgentGym, an agentic environment where general-purpose MLLM agents can use bounded knowledge search over SurgWiki and general-purpose visual grounding for surgical tasks. SurgAgentGym evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. This opens a path toward future studies of agentic surgical understanding.

0.36