CS Paper Radar

Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation

Xinning Yao, Jingjing Wang, Jinghua Yue, Xiaoyan Luo, Fugen Zhou, Bo Liu

cs.CV ·submitted 2026-08-25 13:27 UTC · v1

Abstract

Surgical instrument segmentation (SIS) is fundamental for computer-assisted surgery, where reliable instrument masks enable precise scene understanding and clinical assistance. Recently, adapting foundation models like the Segment Anything Model (SAM) to the surgical domain via prompt-learning has shown encouraging results. However, the performance of these adapted models under challenging surgical conditions is constrained by suboptimal adaptation mechanisms. Specifically, optimizing prompts or prototypes purely via downstream segmentation loss tends to cause them to degenerate into task-specific parameters rather than serving as persistent, stable category memory, thereby degrading their robustness against complex intraoperative variations. Moreover, routing multi-scale visual cues through a single prompt pathway creates a bottleneck that hinders effective scale-matched coupling. To address these limitations, we propose HPMA, a Hierarchical Prototype-Memory Adaptation framework for SAM. Specifically, HPMA constructs a frozen, multi-scale visual prototype memory bank from annotated surgical scenes and integrates it into SAM's feature space using lightweight adapters to preserve stable category evidence. To maximize the utility of multi-scale cues, we introduce a scale-matched coupling mechanism where global prototypes calibrate class-level prompt features, structural prototypes guide decoder object queries, and local prototypes align high-resolution feature maps through a local alignment objective. Extensive experiments on the public EndoVis2017 and EndoVis2018 datasets demonstrate that our approach achieves state-of-the-art performance, outperforming existing foundation model adaptation methods.

Closest papers

Nearest neighbours by meaning across the whole index.

Rethinking Data Imbalance in Class Incremental Surgical Instrument Segmentation

cs.CV ·2026-10-26 ·Shifang Zhao, Long Bai, Kun Yuan, Feng Li, Jieming Yu +6 more
publishedbenchmarkhighly cited similarity 0.70
Abstract

In surgical instrument segmentation, the increasing variety of instruments over time poses a significant challenge for existing neural networks, as they are unable to effectively learn such incremental tasks and suffer from catastrophic forgetting. When learning new data, the model experiences a sharp performance drop on previously learned data. Although several continual learning methods have been proposed for incremental understanding tasks in surgical scenarios, the issue of data imbalance often leads to a strong bias in the segmentation head, resulting in poor performance. Data imbalance can occur in two forms: (i) class imbalance between new and old data, and (ii) class imbalance within the same time point of data. Such imbalances often cause the dominant classes to take over the training process of continual semantic segmentation (CSS). To address this issue, we propose SurgCSS, a novel plug-and-play CSS framework for surgical instrument segmentation under data imbalance. Specifically, we generate realistic surgical backgrounds through inpainting and blend instrument foregrounds with the generated backgrounds in a class-aware manner to balance the data distribution in various scenarios. We further propose the Class Desensitization Loss by employing contrastive learning to correct edge biases caused by data imbalance. Moreover, we dynamically fuse the weight parameters of the old and new models to achieve a better trade-off between the biased and unbiased model weights. To investigate the data imbalance problem in surgical scenarios, we construct a new benchmark for surgical instrument CSS by integrating datasets. Extensive experiments demonstrate the effectiveness of the proposed framework, achieving significant performance improvement against existing baselines.

0.53

SIRA: Reasoning-Aware Surgical Instrument Segmentation via Query-Anchored Alignment

cs.CV ·2026-09-18 ·Zhibo Zhang, Qijie Wang, Zengqiang Yan
top venuebenchmarkcode/data similarity 0.70
Abstract

Surgical instrument segmentation (SIS) plays a critical role in robotic assistance and surgical workflow analysis. However, most existing SIS methods formulate segmentation as a category-driven localization problem, limiting their ability to capture procedural context and task-dependent semantics in surgical workflows. We introduce Reasoning-Aware Surgical Instrument Segmentation (RA-SIS), a task formulation that frames segmentation as query-conditioned inference under surgical context. To benchmark this setting, we construct SurgRS, a surgical reasoning segmentation dataset consisting of 41,000 image-text pairs, which aligns instance-level masks with structured query-answer supervision to enable semantic grounding at the pixel level. Based on SurgRS, we propose Surgical Instrument Reasoning and Segmentation Assistant (SIRA), a multimodal framework that disentangles target-level and query-level semantics and integrates them with visual features through query-anchored dual alignment. By aligning query semantics with spatial features and segmentation prompts, SIRA enhances semantic-visual consistency in mask prediction. Extensive experiments on SurgRS demonstrate improvements over existing reasoning-aware baselines. Code is available at https://github.com/linxir226/SIRA.

0.43

A Surgical Foundation Model Reveals Task-Dependent Label Efficiency

cs.CV cs.AI ·2026-09-25 ·Florian Philipp Stilz, Lorenzo Arboit, Vinkle Srivastav, CAMMA International Surgical Partners, Jacques Marescaux +4 more
code/data similarity 0.68
Abstract · 14 page, 6 figures

Developing label-efficient models is a central challenge in surgical AI due to the high cost and scarcity of expert annotation. While self-supervised foundation models adapt well to new tasks with minimal data, how label efficiency varies across different surgical tasks remains largely unexplored. Here, we introduce SURGE, a surgical foundation model trained on SurgSpectrum-30M+, the largest pretraining dataset comprising over 30 million frames, with checkpoints released to enable further research. We systematically evaluate label efficiency across 5 task categories and 15 benchmarks. These range from temporal and spatial scene understanding to fine-grained reasoning tied to instrument-anatomy interactions and safety-critical maneuvers. SURGE outperforms prior state-of-the-art on all benchmarks, even surpassing task-specific models on complex reasoning tasks. Crucially, we reveal a task-dependent scaling behavior: while scene understanding tasks saturate with minimal supervision, fine-grained reasoning tasks continue improving with substantially larger annotation budgets, providing a blueprint for allocating expert effort in complex domains. Code: https://github.com/CAMMA-public/SURGE

0.60

SurgStream: A Workflow-Anchored Proactive Streaming Framework and Benchmark for Real-Time Surgical Video Understanding

cs.CV cs.AI ·2026-11-01 ·Ruipeng Ma, Qing Xu, Jie Liang, Zhen Chen
benchmark similarity 0.67
Abstract

Surgical video understanding is essential for intraoperative assistance such as workflow monitoring, instrument preparation and procedural documentation. Delivering such assistance in real time requires a model to jointly perceive the ongoing scene, remember past surgical events, and respond before clinically relevant moments pass. Existing approaches provide limited support for jointly retaining procedural evidence and deciding when to respond in streaming surgical video. In this work, we propose SurgStream, a workflow-anchored proactive streaming framework for real-time surgical video understanding. We first introduce Workflow-Anchored Memory (WAM) to record structured instrument-action-target observations and fold them into phase-level summaries, providing compact and queryable surgical memory. Deadline-aware Transition Learning (DTL) is then devised to supervise response states within clinical windows, enabling timely responses before procedural deadlines. We further introduce SurgStream-Bench, comprising 11.7K streaming interactions across various tasks covering real-time perception, backward memory queries, forward proactive QA, proactive alerts and caption memory. Experiments on SurgStream-Bench show gains over state-of-the-art methods, especially in memory and forward proactive QA. The source code will be available.

0.20

CasCVS-Net: A Staged Multi-Task Cascade for Critical View of Safety Assessment

cs.CV ·2026-09-23 ·Bock-Zien Toh, Yuanchuan Ren, Tay Aw Yu, Ng Khee Ong, Zhehua Mao +1 more
similarity 0.67
Abstract · 10 pages, 3 figures, 3 tables

Automated assessment of the Critical View of Safety (CVS) in laparoscopic cholecystectomy requires both recognition of the three CVS criteria and anatomical grounding in small, rare, and often occluded hepatocystic structures. Learning-based methods differ in the anatomical information they use, from image-level classification to detection, segmentation, or graph-based reasoning, yet grounding the safety-critical anatomy remains the main bottleneck. We propose CasCVS-Net, a staged multi-task cascade that jointly performs object detection, semantic segmentation, and CVS assessment, trained on the Endoscapes dataset. The model couples the tasks through predicted anatomy: predicted boxes guide segmentation, and predicted masks provide region-level features for CVS classification, so CVS assessment at inference uses only model predictions rather than ground-truth annotations. To reduce optimisation instability in this coupled setting, training progresses from detection to detection-segmentation and then to the full three-task cascade, followed by task-wise fine-tuning. Evaluation on the public unseen test set shows that CasCVS-Net improves over matched single-task baselines on all three tasks, achieving 32.0 detection mAP, 46.8 semantic mIoU, 15.3 rare-anatomy mIoU, and 67.2 CVS mAP. It outperforms the state-of-the-art LG-CVS and SV2LSTG by 6.3% and 4.5% relative CVS mAP, respectively, corresponding to 4.0 and 2.9 mAP points. These results show that staged task coupling through predicted boxes and masks improves anatomical grounding for CVS assessment, particularly for rare hepatocystic structures.

0.28

CVP-BAA: Efficiently Adapting Segment Anything Model for Medical Image Segmentation

eess.IV ·2026-10-22 ·Mei Wang, Minghui Li, Libin Lan
code/data similarity 0.66
Abstract · Submitted to CVIU, 35 pages, 8 tables, 8 figures. Our code is available at \href{https://github.com/WangMei-wm/CVP-BAA}{here}

Various task-specific medical segmentation models have demonstrated strong performance, their heavy reliance on extensive annotations and highly specialized architectures limits their versatility, scalability, and clinical applicability. Recently, adapting vision foundation models, particularly the segment anything model (SAM), has emerged as a promising alternative. However, prevalent parameter-efficient fine-tuning (PEFT) strategies, such as prompt tuning and adapter tuning, often lack an explicit mechanism to embed local structural priors into the long-range dependency modeling of vision transformers (ViT), rendering them insufficiently sensitive to fine-grained details. To address these limitations, we introduce CVP-BAA, a novel hybrid PEFT method that improves SAM's sensitivity to domain-specific textures and local details, leading to superior boundary delineation. The proposed CVP-BAA adopts two key ideas. First, it introduces a convolutional visual prompting (CVP) strategy that leverages convolutional features as structural priors, prepending them to the key and value projections within the multi-head self-attention layers of strategically selected ViT blocks. Through a dedicated prompt generation-and-injection module, this mechanism effectively bridges local spatial details with global context semantics. Second, a boundary-aware adapter (BAA) is designed as a parallel branch to the feed-forward network within the same selected blocks. By explicitly computing and amplifying high-frequency components, the BAA enhances feature representations along tissue boundaries and fine-grained lesion structures. Extensive experiments on five benchmarks across three imaging modalities demonstrate that CVP-BAA achieves superior segmentation performance, surpassing existing task-specific models and PEFT-based SAM adaptations in both quantitative metrics and qualitative results.

0.38

OphBiWSSD: Scaling Temporal Action Localization in Ophthalmic Surgeries with Bidirectional Weight-tied State Space Duality

cs.CV ·2026-09-11 ·Yang Liu, Qionghong Ma, Joongwon Chae, Lihui Luo, Yibing Shen +8 more
benchmarkcode/data similarity 0.66
Abstract · Submitted to IEEE Transactions on Image Processing. Under review

High-frequency surgical maneuvers in ophthalmology necessitate high-fidelity temporal modeling, yet characterizing long-range procedural dependencies remains computationally prohibitive for attention-based architectures. Existing models often require aggressive temporal downsampling, which compromises the detection of fine-grained action boundaries and instrument-tissue interactions. To address these scalability constraints, we present OphBiWSSD, a framework that reformulates surgical temporal action localization leveraging Bidirectional State Space Duality. By employing a weight-tied selective scan mechanism that incorporates both preceding and succeeding surgical contexts, our approach facilitates the global synthesis of non-causal temporal cues with linear complexity. This streamlined architecture is well-suited to capture the bidirectional dependencies present in ophthalmic workflows, effectively bridging the gap between local boundary precision and long-range procedural context without incurring the quadratic memory overhead of traditional Transformers. Extensive experiments on the OphNet benchmark demonstrate that OphBiWSSD achieves state-of-the-art temporal localization performance, with mean Average Precisions of 44.42% on phases and 43.08% on operations, surpassing the baselines by 6.80% and 6.66%, respectively. Empirical validation indicates that our approach ensures precise temporal localization and offers a computationally viable pathway for deploying surgical intelligence systems in clinical environments. The code is publicly available at https://github.com/yo3nglau/OphBiWSSD.

0.30

Feature-Space Latent Forecasting for Endoscopic Surgical Video Understanding

cs.CV ·2026-11-11 ·Songtao Hu, Hang Su
similarity 0.66
Abstract

Surgical workflow assistance is inherently anticipatory: systems must reason about how procedures evolve, not just recognize the current frame. However, many prior approaches perform prediction in pixel space by synthesizing future frames, which primarily focuses on appearance fidelity rather than the semantic representations required for downstream reasoning. To address this limitation, we propose a feature-space semantic forecasting framework that models temporal dynamics directly within the representations of a frozen Vision Foundation Model (VFM). Specifically, we build a hierarchical target representation from multi-layer VFM features, replace future-frame feature tokens with learnable \texttt{[MASK]} tokens, and train a masked transformer to forecast the corresponding future latent features from observed context, thereby formulating procedural evolution as masked future feature prediction. Experiments on public laparoscopic benchmarks demonstrate consistent improvements in phase anticipation, multi-label tool presence prediction, and remaining-time estimation over RGB-based forecasting baselines, while enabling a plug-and-play representation interface for lightweight downstream heads.

0.11