CS Paper Radar

Ultra-Lightweight 3D Medical Image Segmentation via Knowledge Distillation in Label-Scarce Domains

Rahim Ullah, Hasan Abbas, Syed Waqar Nabi

cs.CV cs.AIcs.LG ·submitted 2026-11-05 13:20 UTC · v1
code/data

Abstract

Mamba is a neural network architecture that reduces computational complexity, and has been shown to be effective for 3D medical image segmentation. However, existing models still remain computationally expensive. We propose compressing Mamba by distillation while achieving competitive performance. Specifically, we demonstrate compression of a 139M-parameter SegMamba-V2 teacher into a lightweight TinyUNet3D student with about 0.96M parameters, achieving about 145$\times$ compression with a performance drop of just over one Dice point. We further investigate the utility of knowledge distillation in a range of data-labeling scenarios under identical experimental settings, and show that its advantage is most prominent when labeled data is scarce. To demonstrate the generalizability of our approach, experiments are conducted on three datasets: BraTS 2023, MSD Spleen and MSD Hippocampus. The code for TinyUNet3D is publicly available at: https://github.com/2802990u/tinyunet3d-dist-kd.

Comment
This is a four pages paper having 3 figures and is submitted to ISBI2027

Closest papers

Nearest neighbours by meaning across the whole index.

Uni-Light: An Ultra-Lightweight Framework via Uncertainty-Aware Knowledge Distillation for Brain Tumour Segmentation

cs.CV ·2026-09-06 ·Libing Kuang, Soren Salehi, Ziling Wu, Ahmad P. Tafti, Armaghan Moemeni
similarity 0.71
Abstract

Accurate 3D brain tumour segmentation from multi-modal Magnetic Resonance Imaging (MRI) is essential for clinical diagnosis and treatment planning. Existing brain tumour segmentation methods often suffer from heavy computational demands, while current lightweight architectures frequently lack the capacity to maintain segmentation fidelity in complex tumour regions. To address these issues, we propose a novel ultra-lightweight framework (Uni-Light) that achieves high-fidelity segmentation with substantially reduced computational overhead. It combines multi-scale convolutions with an uncertainty-aware knowledge distillation scheme that directs the student model toward hard-to-classify regions, complemented by a Signed Distance Field boundary loss for geometric constraints. Experimental results on BraTS2023-GLI and MSD-BTS datasets demonstrate that Uni-Light reduces parameters by 97.56%, floating-point operations (FLOPs) by 73.03%, and inference memory footprint by 81.58%, while surpassing the state-of-the-art model by an average of 1.47% in Dice score, offering a highly competitive trade-off between segmentation accuracy and computational efficiency in resource-constrained clinical settings. This work also advances data engineering for medical imaging by demonstrating that teacher model uncertainty can be exploited as a data-driven supervisory signal, re-prioritising the training data distribution without requiring additional annotation.

0.24

OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute

cs.CV ·2026-09-28 ·Brandon Leblanc, Charalambos Poullis
publishedcode/data similarity 0.70
Abstract · Accepted at ACCV 2026

Feed-forward 3D reconstruction models have achieved impressive performance by scaling model and dataset size, but their cost excludes most research groups and precludes edge deployment. Additionally, generating 3D supervision without sensors still relies on slow, unreliable Structure-from-Motion, as the community lacks a COLMAP-like system for neural 3D pseudo-label generation. We present OTT3R (RGB-Only Tiny Transformer for 3D Reconstruction), a knowledge distillation framework that addresses both problems on a single workstation equipped with 2 GPUs. Distilling $π^3$ (959M parameters) into a 102M-parameter student yields 9.4$\times$ compression and up to 7$\times$ faster inference, trained at 1.6% of VGGT's training compute. An integrated pseudo-label pipeline offers a reliable, high-throughput alternative to COLMAP, generating dense per-pixel point maps and SE(3) camera poses for a 667K-image corpus in 3.5 hours on two commodity GPUs and succeeding on every sequence we tested, including those where COLMAP fails. The general student tracks the teacher on in-distribution monocular depth and, zero-shot, outperforms COLMAP on 7-Scenes and on DTU completion, but it does not replace the teacher on out-of-distribution multi-view geometry. The deployable artifact is the domain-specialized student: after specialization at 0.2% compute, it is 4$\times$ more accurate than COLMAP on 7-Scenes at 980$\times$ throughput, with near-teacher completion. Code is available at https://github.com/TheFourthKaramazov/OTT3R

0.43

Computationally Efficient Pathology Segmentation using Knowledge Distillation from Foundation Models

eess.IV ·2026-09-03 ·Jiaqi Lv, Yijie Zhu, Saki Okada, Maxwell Stanley Renna, Robert Goldin +2 more
similarity 0.67
Abstract · Submitted to Medical Image Analysis

Automatic tissue segmentation is essential for large-scale analysis of histopathology whole-slide images (WSIs), but accurate pixel-level segmentation remains challenging. Pixel-level annotations are expensive to obtain, models pre-trained on natural images may transfer poorly to histopathology, and pathology foundation models, despite their strong representations, are computationally expensive to deploy at scale. We address these challenges with a foundation-model knowledge distillation framework for efficient tissue segmentation. We first train Virchow2-based segmentation teachers, including a LoRA-adapted variant, achieving state-of-the-art or highly competitive performance across four datasets: PUMA, IGNITE, BEETLE, and a private blood vessel segmentation dataset. We then transfer response-level and feature-level knowledge from these teachers into compact student networks. Distillation consistently improves student performance over supervised training alone, producing state-of-the-art or near state-of-the-art results with substantially fewer parameters and up to ten-fold higher inference throughput than foundation-model-based segmentation networks. These results show that foundation-model knowledge can be effectively transferred to efficient segmentation models for scalable deployment without compromising performance. Models trained on the public datasets will be released through TIAToolbox.

0.35

Task-Performance Routing for Multi-Teacher Distillation in CT Medical Image Segmentation

cs.CV ·2026-10-30 ·Jiaye Yang, Yao Liu, Die Dai, Hanwen Zhang, Peiyuan Jiang +3 more
top venuecode/data similarity 0.66
Abstract · Accepted at MICCAI 2026. 11 pages, 3 figures, 4 tables. Code: https://github.com/EdvinCecilia/TPR_MultiTeacher

Pre-trained models offer complementary strengths in CT organ segmentation, but their relative accuracy varies across organs and spatial regions. Multi-teacher knowledge distillation can combine these strengths in a single student, yet global or instance-level teacher weights overlook variation within a 3D volume. We propose Task-Performance Routing (TPR), which uses region-wise segmentation errors to determine teacher weights during training without learned routing parameters. In hard regions, TPR favors teachers with lower prediction errors; in easy regions, it uses uniform weights to combine their knowledge. A hierarchical spatial partition provides fine-grained routing in shallow encoder stages and coarse-grained routing in deep stages. Experiments on BTCV, MSD-Liver, and MSD-Pancreas with two teacher pairs and two backbone architectures show higher Dice scores than the individual teachers, adaptive ensembles, and evaluated multi-teacher distillation baselines. On MSD-Pancreas, TPR achieves Dice scores of 65.40% with SwinUNETR and 68.25% with U-Net, compared with 64.38% and 67.45% for the corresponding adaptive ensembles. Inference uses the student alone, with no routing or teacher overhead. Code is available at https://github.com/EdvinCecilia/TPR_MultiTeacher.

0.38

GAD-MambaUNet: Direction-Group Mamba with Gradient-Adaptive DINOv3 Distillation for Lightweight Medical Image Segmentation

cs.CV ·2026-09-22 ·Fang Wang, Huitao Li, Wenhan Chao, Zheng Zhuo, Xinxin Yang
similarity 0.66
Abstract

In this paper, we proposed GAD-MambaUNet, a lightweight medical image segmentation network that combines efficient local modeling, direction--group state-space interaction, and training-time foundation-model supervision. To improve contextual modeling in compact segmentation networks, we introduced Direction-Group Graph Selective Scan (DG-GSS), which treated scan-direction and channel-group responses as graph nodes and enabled structured information exchange before multi-directional fusion. We further incorporated DINOv3-GAD supervision, where a frozen DINOv3 teacher provided semantic guidance during training, and Gradient-Adaptive Distillation dynamically regulated the distillation strength. GAD-MambaUNet achieves a favorable accuracy--efficiency balance compared with representative lightweight and general segmentation methods. Ablation studies further verify the effectiveness of DG-GSS and training-time DINOv3-GAD supervision. In future work, we will explore more flexible teacher--student alignment strategies and extend the proposed framework to more diverse medical segmentation scenarios, such as multi-class and multi-modal segmentation tasks.

0.46

Confidence-Aware Teacher-Student Distillation for 3D Medical Segmentation

cs.CV ·2026-09-20 ·Georgios Triantafyllou, Dimitris K. Iakovidis
theory similarity 0.66
Abstract

Medical image segmentation models typically rely on large amounts of densely annotated volumetric data, limiting their scalability across tasks and imaging modalities. This work addresses the challenge of predicting entire 3D anatomical structures from extreme annotation sparsity. An annotation-efficient student-teacher framework is proposed for automatic 3D medical segmentation that requires only a set of point prompts on a single 2D slice per volume, as input. A foundation model serves as an offline teacher, utilizing the provided point prompts from the selected slice to full-volume pseudo-annotations alongside their corresponding spatial confidence scores prior to student training. To mitigate the error propagation of noisy pseudo-annotations, a task-specific 3D student network is trained using a confidence-aware optimization strategy. By leveraging the teacher's pre-computed confidence scores, this strategy explicitly excludes statically uncertain regions of the pseudo-annotations from the loss calculation, while simultaneously emphasizing regions with higher confidence. Evaluated on 3D cardiac MRI datasets, our framework outperforms state-of-the-art semi-supervised methods, improving segmentation performance by up to 43.6%. Furthermore, it drastically reduces the manual annotation burden to just a few positive point prompts per volume, while improving surface boundary precision by up to 14.7% over the teacher and successfully recovering up to 34.1% of the performance gap toward the fully supervised upper bound.

0.22

Promise and Limits of Cross-Architecture Knowledge Distillation for Lightweight Medical Vision Transformers (LiteMed-ViT)

cs.CV ·2026-10-24 ·Shuaibu Shehu, Elijah M. Flomo, Mohammad Safiya, Anurodh Kumar, Subodh Kumar
published similarity 0.65
Abstract · 6 pages, including references. Accepted at the International Conference on Intelligent Computing and Secure Communication Systems (ICICSCS 2…

Running deep models for medical image analysis at the point of care is challenging with standard architectures: convolutional neural networks (CNNs) are often too heavy for edge devices, while Vision Transformers (ViTs) require massive training datasets to compensate for missing inductive biases. This study investigates how far the dual-token distillation used to train data-efficient ViTs can be pushed when applied across architectures, from a CNN-based teacher into a lightweight ViT student. LiteMed-ViT is introduced as a dual-token ViT featuring a convolutional stem and a class-balanced distillation objective; at 0.234M parameters, LiteMed-ViT carries 47.8x fewer parameters than its ResNet18 teacher at a memory footprint of just 0.89 MB. Evaluated against three compact CNN baselines on three MedMNIST subsets of varying modality and class balance, LiteMed-ViT attains higher accuracy than all three baselines on BloodMNIST (95.05%) and two on PathMNIST (89.15%). Under MedMNIST-C corruptions, it is at least as robust as every compact CNN across all subsets, outperforming them on BloodMNIST and PathMNIST without declining in relative standing as severity increases. Particular attention is devoted to identifying the exact conditions where performance plateaus. On DermaMNIST, LiteMed-ViT's balanced accuracy remains up to 1.9 points below the strongest compact CNNs and does not benefit from higher input resolutions up to 224 pixels, where ResNet18 gains 12.6 points. Stronger teachers do not yield a more accurate student, and overconfident ones lower its DermaMNIST balanced accuracy unless calibrated.

0.27

Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learning

cs.CV ·2026-10-09 ·Junkai Liu, Le Zhang
publishedcode/data similarity 0.65
Abstract · NeurIPS 2026. Project page: https://volumeprobe.github.io/

Self-supervised learning on 3D medical images commonly relies on random sub-volume sampling during pretraining. However, random cropping implicitly treats all spatial regions as equally worth observing, despite anatomical priors and spatial redundancy that make regions inherently unequal in their value for representation learning. To this end, we first show through controlled crop-quality manipulations that better sub-volumes lead to better downstream transfer, revealing the observation policy as an overlooked bottleneck in 3D medical SSL. Motivated by this finding, we revisit sub-volume sampling as a where-to-look problem. We propose VolumeProbe, a plug-and-play framework that replaces random cropping with an offline coarse-to-fine probe bank guided by anatomy-awareness, informativeness, and diversity. We instantiate VolumeProbe as a lightweight Lite variant using geometric and image-statistical cues, and a diagnostic Feat variant using frozen visual features, to examine whether lightweight volume-intrinsic priors suffice or richer semantic cues reliably provide additional value. Extensive experiments across multiple 3D medical SSL backbones and downstream tasks demonstrate that VolumeProbe-Lite consistently improves transfer performance over random sampling. Mechanistic analyses further link these gains to more anatomically plausible and informative observations, reduced spatial redundancy, more efficient learning dynamics, and more structured representations.

0.22