CS Paper Radar

Lightweight Frame Compensation for Real-Time Polyp Segmentation under Frame Skipping

Weihao Gao

cs.CV ·submitted 2026-10-22 13:13 UTC · v2

Abstract

Real-time polyp segmentation in computer-assisted colonoscopy is hampered by the difficulty of deploying high-accuracy models under strict latency budgets. Frame skipping, which performs inference every K frames and reuses or interpolates the rest, improves throughput but introduces motion blur and lesion misalignment that degrade temporal consistency. To close this gap, we propose the Lightweight Frame Compensator (LFC), a 77K-parameter plug-and-play module inserted before a frozen segmentation backbone. Operating on a three-frame window, LFC fuses consecutive frames via spatio-temporal attention into a single compensated frame, and is trained only on pseudo-labels generated by a differently seeded model, without any expert annotation. Experiments on three polyp video benchmarks show that LFC recovers most of the accuracy degraded by direct frame skipping and, in several settings, reaches the accuracy of frame-by-frame inference, while retaining a 2.05X--2.55X throughput gain. We further evaluate the temporal stability of three backbones spanning CNN, pyramid Transformer, and global-attention Transformer architectures, revealing a spectrum of sensitivity to frame skipping that arises from how locally or globally each architecture aggregates misaligned input. The compensation gain of LFC follows this spectrum: it is largest for high-capacity Transformer backbones, which achieve the best static accuracy but suffer the most under skip-frame inference. LFC thus offers a lightweight, pluggable, and deployment-friendly route to real-time polyp segmentation.

Comment
Subsequent analysis revealed a flaw in the evaluation protocol of the proposed lightweight frame compensation. The original experiments did not sufficiently account for highly variable content dynamics, leading to over-optimistic conclusions. The authors are withdrawing this preprint to re-examine the findings under a more rigorous empirical setup.

Closest papers

Nearest neighbours by meaning across the whole index.

TRUE-Colon: Exposing a Consistent Transfer Asymmetry in Real-Time Polyp Detection

eess.IV cs.CVcs.LG ·2026-08-13 ·Sebastian Doerrich, Andreas Franz Schwab, Francesco Di Salvo, Shyam Nandan Rai, Hanh Huyen My Nguyen +1 more
publishedcode/data similarity 0.66
Abstract · Accepted to EndoLINA @ MICCAI 2026 (The International Workshop on Endoluminal Intervention Navigation and Autonomy)

Computer-aided detection (CADe) systems for colonoscopy promise to reduce clinical miss rates, yet reliable real-world deployment remains elusive. This translational gap stems in part from a structural flaw in model development: the reliance on curated datasets that under-represent the long negative stretches and procedure-related artifacts characteristic of routine examinations. Training and evaluating architectures strictly on these lesion-centric benchmarks creates an illusion of success, since such benchmarks cannot capture clinically crucial metrics. To expose this gap, we establish TRUE-Colon, a standardized benchmarking protocol that measures key deployment characteristics alongside localization accuracy, and evaluate four real-time architectures (Faster R-CNN, YOLOv8, YOLOv11, RT-DETR) across curated benchmarks (SUN, PICCOLO) and 60 unedited, full-length procedures (REAL-Colon). We observe a consistent transfer asymmetry: models trained strictly on curated clips suffer a severe performance collapse when evaluated on full procedures, whereas procedure-trained models substantially improve rejection of non-polyp content on REAL-Colon, and largely retain their accuracy on curated benchmarks. Beyond transferability, we find that the Transformer detector attains the strongest sensitivity and the earliest, most persistent detections, while the convolutional detectors stay competitive at a higher throughput. Together, these results indicate that both training and benchmarking for deployable CADe should shift from curated, lesion-centric clips toward full-procedure data and deployment-relevant operating points. Source code is available at https://github.com/sdoerrich97/true-colon.

0.38

WSPolypNet: Weakly Supervised Polyp Localization in Colonoscopy Videos

cs.CV cs.AI ·2026-09-08 ·Giseong Hwang, Minjae Jo, Yeonghyeon Park, Kyeonghun Kim, Seoyeon Han +7 more
similarity 0.64
Abstract

Because dense frame-level annotation of colonoscopy videos is costly, we propose WSPolypNet, a weakly supervised framework for polyp localization using only video-level labels. WSPolypNet employs a 3D convolutional neural network trained with video-level supervision to generate class activation maps (CAMs), which identify candidate polyp regions without requiring frame-level spatial annotations. The CAM-derived localization cues are further enhanced using a multi-view strategy and provided to MedSAM2 as point prompts. MedSAM2 then propagates segmentation masks across the video, refining the coarse localization cues according to polyp boundaries. WSPolypNet achieved CorLoc scores of 47.80%, 43.68%, and 35.01% at IoU thresholds of 0.3, 0.5, and 0.7, respectively, compared with 36.87%, 33.72%, and 27.94% in the single-view setting. For small polyps, the multi-view strategy improved CorLoc@0.5 from 16.01% to 30.97%. The framework also achieved a recall of 94.51%. These results demonstrate the potential of weakly supervised spatiotemporal learning to substantially reduce spatial annotation requirements for polyp localization in colonoscopy videos.

0.33

CSG-Mamba: A Convolutional Scoring Gating Vision State Space Network for Endoscopic Polyp Segmentation

cs.CV ·2026-08-14 ·Yuliang Wang, Jiaqi Wu, Jiaye Song, Shuxia Ren
published similarity 0.64
Abstract · 14 pages, 6 figures, and 5 tables. Accepted by ICONIP 2026

Accurate polyp segmentation is critical for computer-aided colonoscopy, yet endoscopic images often contain low-contrast boundaries, mucosal texture interference, specular highlights, and device-dependent appearance shifts. Vision State Space Models (SSMs) provide efficient long-range modeling with linear complexity, but existing Vision Mamba segmentation models typically convert 2D features into 1D scanning sequences, which may weaken local geometric continuity and over-smooth irregular contours. We propose CSG-Mamba, a convolutional scoring gating Vision State Space network for endoscopic polyp segmentation. Built on a VM-UNet-style asymmetric U-shaped encoder-decoder, CSG-Mamba inserts a Convolutional Scoring Gating (CSG) module at the semantically rich bottleneck. CSG generates a local spatial score map through pointwise and large-kernel depthwise convolutions and recalibrates state-space features by multiplicative gating. Experiments with three random seeds show that CSG-Mamba achieves 0.9220 Dice and 15.87 HD95 on Kvasir-SEG, and 0.7418 Dice and 0.6570 mIoU on CVC-ColonDB, outperforming the baselines on most overlap and recall metrics while maintaining competitive boundary accuracy.

0.28

Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models

cs.CV cs.LG ·2026-11-17 ·Rui Ning, Om Chabra, Hari Balakrishnan
benchmarkcode/data similarity 0.64
Abstract

Applications such as visual accessibility agents and robotics require answering questions about recent video content promptly, over a bounded window of past frames (e.g., the last two minutes). Hybrid vision-language models (VLMs), which interleave attention with linear recurrent layers such as Gated DeltaNet (GDN), are attractive for this setting, but maintaining a sliding window is expensive. The reason is that recurrent layers combine all past frames into a shared state, so expiring old frames normally requires re-running the entire window through the model. We observe that the recurrent update is affine in the state with input-dependent coefficients, so the effect of each frame can be captured as a transformation that composes associatively. We present Spool, a training-free framework that caches per-frame state transformations and composes them efficiently as the window advances, removing expired frames from each recurrent layer's state without replaying frames through the backbone. Removal is exact within each layer; as with key-value eviction in attention, expired frames can retain residual influence through the cached inputs computed by earlier layers. We design a dedicated data structure and GPU kernels to achieve low caching and composition overhead. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool achieves the lowest p95 time-to-first-token in all 20 model-benchmark tests, reducing it by a geometric mean of $6.6\times$ over the faster of two recent vLLM baselines, while improving accuracy by 2.3 percentage points on average.

0.32

MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

cs.CV ·2026-08-18 ·Bonan Zhang, Shiyu Dong, Quan Hung Tran, Katharina Gschwind, Shuqi Yang +7 more
top venuecode/data similarity 0.64
Abstract · Accepted to ECCV 2026

Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.

0.63

GramLoop: Training-Free Gram-Gated Replay for Robust Dense Prediction

cs.CV ·2026-08-29 ·Yang Chen, Canyu Shen, Xinzhe Rao, Yuanyi Yan, Yunlu Chen +3 more
code/data similarity 0.64
Abstract · Code will be released at https://github.com/cheyan9/GramLoop

We aim to improve frozen DINOv3 dense-prediction models under distribution shift by adding inference computation inside the visual backbone, without changing model weights, task adapters, or prediction heads. The challenge is that repeated transformer-block computation must refine dense features without disrupting the pairwise patch relations that DINOv3 uses to preserve spatial structure. We introduce GramLoop, a training-free framework that replays a short transformer window and controls each replay through final-layer cosine-Gram consistency. Each proposal is propagated through the frozen suffix, measured against the standard DINOv3 trajectory, and accepted through a patchwise gate at the replay-window endpoint. Across object detection and semantic segmentation under corruptions, perturbations, and natural shifts, GramLoop improves all five shifted benchmarks over the paired DINOv3 baseline. On COCO-O, it improves mAP by +0.252 and Effective Robustness by +0.250, while preserving clean ADE20K performance. Code will be released at https://github.com/cheyan9/GramLoop.

0.37

Eco3R: Pinned-Key Inference for Efficient and Streamable Feed-Forward 3D Reconstruction from Long Videos

cs.CV cs.AI ·2026-11-13 ·Hongsen Cao, Shanxin Yuan, Mona Jaber, Ahmed Sayed
similarity 0.64
Abstract · 26 pages, 10 figures. v2: renamed the method from EcoVGGT to Eco3R, added Figure 1, and revised the text

Feed-forward multi-view transformers such as VGGT recover the cameras and geometry of a video in one pass, but the cost of their cross-frame self-attention grows quadratically with the frame count, and their accuracy drops on long videos. Most remedies approximate attention over all frames under a fixed computational budget or select a few keys by their content. We hypothesize that the keys of a long video can be selected independently of video content, and our experiments on VGGT and $π^3$ support it: a small number of uniformly sampled frames can serve as the keys of every frame. We present Eco3R, a training-free inference framework whose keys form a Pinned Key Set of 31 such frames. A pin pass stores their tokens in a Scene Cache, and the other frames read it in independent batches that reproduce joint inference with these keys exactly, at a constant cost per frame. An Online Stream updates the pinned frames during recording. Six middle global blocks align the frames and form a reusable VGGT module: the Alignment Score reads their attention to flag failures, and Alignment Refocusing reduces them by fine-tuning only these blocks (3.6 GPU hours). Eco3R ($π^3$) matches $π^3$'s success rate on 1,000-frame ScanNet videos while running 13.8 times faster. With refocusing, Eco3R (VGGT) raises VGGT's success rate from 70% to 82% while running 8.5 times faster on 500 frames. Eco3R can reconstruct sequences of up to 4,000 frames on a single A100 GPU, while Eco3R ($π^3$) streams 1,000 frames at 18.6 fps, achieving the lowest online absolute trajectory error.

0.13

FOCUS: Closed-Loop Attention Feedback for Efficient Vision-Language Understanding

cs.AI cs.CV ·2026-11-01 ·Akul Santhosh, Aastha Jhunjhunwala
similarity 0.64
Abstract

Processing every video frame through Vision-Language Models (VLMs) adds significant inference cost and limits their ability to comprehend real-time video. Existing approaches either sub-sample frames uniformly, ignoring content and missing important events, or prune visual tokens inside the VLM to cut per-call cost while still invoking the model on every sampled frame. We introduce FOCUS, a training-free three-layer framework that reduces the number of invocations and steers each one towards what matters. Temporal gating measures inter-frame dissimilarity in the patch-embedding space of a smaller VLM from the same family, triggering burst captures only when the scene changes in a semantically meaningful way. Patch-level dissimilarity is then accumulated across the burst and fused with a top-down attention prior into a spatial focus map that encodes where motion and semantic relevance coincide. That map calibrates the VLM's value cache during generation, progressively amplifying vision tokens in task-relevant regions and attenuating off-target ones, without removing any token from the context. The loop closes after each response: text-to-vision cross-attention is extracted from the decoder and fed back as the prior for the next burst, letting the system converge on semantically important regions over successive observations. We evaluate FOCUS on the 500-video EgoSchema subset in streaming mode, with a single Qwen3-VL-8B backbone shared by every method. It reaches 74.8% accuracy using 25.4 VLM calls per clip, against 74.6% with 55 calls for uniform sampling and 74.2% with 55 calls for BOLT. On egocentric video captioning across four domains, it matches the best baseline's quality while issuing 57% fewer VLM invocations than uniform sampling.

0.08