CS Paper Radar

Efficient Semantic Understanding from Digital Foveation

Caterina Caccavella, Vittorio Fra, Andreas Ziegler, Giulia D'Angelo, Yulia Sandamirskaya

cs.CV ·submitted 2026-09-03 16:56 UTC · v1
published

Abstract

Dense semantic segmentation allocates computational resources uniformly across the entire image, regardless of scene complexity or task relevance. Inspired by biological vision, we investigate whether semantic understanding can be achieved more efficiently through digital foveated perception. We introduce a lightweight active-vision pipeline that combines saliency-driven fixation selection, high-resolution foveal observations, low-resolution contextual information, semantic accumulation, and adaptive computation. Beyond conventional dense prediction metrics, we use object-level evaluation to measure semantic understanding under sparse observations. On ADE20K-Object, a single foveated observation achieves 95.9% of the baseline Top-1 accuracy and 96.9% of the baseline Top-3 accuracy while requiring only 4.7% of the computational cost. At the scene level, semantic accumulation recovers 90.6% of the baseline object recall while using 58.6% of the computation. These results suggest that substantial semantic understanding can emerge from sparse observations when computation is allocated selectively, highlighting active vision as an efficient alternative to uniform dense processing and motivating evaluation protocols beyond conventional pixel-wise segmentation metrics.

Comment
Accepted at the 3rd Human-inspired Computer Vision Workshop at ECCV 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation

cs.CV cs.LG ·2026-10-14 ·Utku Cicek, Mahip Singh, Ziyao Shang, Kimathi Kaai, C Thomas +3 more
top venue similarity 0.74
Abstract · Accepted at NeurIPS 2026. 19 pages, 6 figures

While foundation models and vision transformers (ViTs) have redefined visual representations, current active learning (AL) strategies for these backbones largely operate at the patch or region level, where annotation requirements remain high due to extensive pixel labeling. Extending ViT-based AL to the pixel-level, low-budget regime is uniquely challenging: with as little as one pixel query per image, a method must simultaneously identify the most informative locations and train a reliable decoder head from an extremely sparse signal. We introduce TEAL (Token-space Efficient Active Learning), the first pixel-level AL framework built around ViT representations. TEAL uses a frozen ViT backbone and performs diversity-based selection directly on its native token lattice, avoiding the artifacts introduced by interpolating token embeddings to dense pixel resolutions. Our framework applies a two-level MaxHerding strategy over multi-layer token descriptors to select representative candidates, followed by margin-based uncertainty refinement on a finer decoder grid. Across CamVid, Cityscapes, ADE20K, and Pascal Context, TEAL consistently outperforms previous baselines under extreme label scarcity, improving over the strongest baseline by up to +24.36 mIoU with a single labeled pixel (px) per image, and reaching up to 97% of fully supervised performance with just 10 px. To rigorously establish the source of the gains, we further show that the improvements persist when all baselines are equipped with the same DINOv3 backbone, demonstrating that coupling token-space selection with uncertainty refinement on the decoder grid makes ViT token geometry an effective foundation for extremely low-budget active segmentation.

0.34

Task-driven Processing with Coarse-to-Fine Glimpse-based Active Perception

cs.CV ·2026-09-08 ·Oleh Kolner, Thomas Ortner, Stanisław Woźniak, Angeliki Pantazi
top venue similarity 0.73
Abstract

State-of-the-art vision models process images in their entirety, lacking the ability to selectively zoom in on relevant regions. This limitation is particularly acute in scenarios where processing must be conditioned on a specific task - such as instance detection, which requires localizing a specific object in a high-resolution, cluttered scene. In such settings, critical details are easily lost as images are often resized to match the model dimensions and computational constraints. We introduce Coarse-to-Fine Glimpse-based Active Perception (CF-GAP), a task-driven front-end that enhances high-resolution processing of existing instance detectors. CF-GAP selectively directs a sequence of limited view glimpses across the scene, utilizing task information to iteratively refine focus on the most relevant regions. These localized regions are then processed at high resolution by a downstream instance detector. By avoiding full-image processing and eliminating irrelevant confounding information, CF-GAP improves Average Precision (AP) by up to 20% across various state-of-the-art instance detectors on the HR-InsDet and Robotools benchmarks, while further enabling lightweight detectors to outperform their larger counterparts.

0.54

Adaptive Observation Control for Active Visual Recognition

cs.CV ·2026-10-14 ·Dikai Li, Weijia Wang, Yuan-Gen Wang
publishedbenchmarkcode/data similarity 0.70
Abstract · This paper has been accepted for presentation in ACCV 2026

Visual recognition models usually process a fixed view under a fixed computation budget, although images often benefit from different scales and amounts of spatial evidence. Active vision can adapt by observing a coarse scene and then selecting informative fixations, but the public AdaptiveNN-DeiT-S pipeline still fixes both field of view and observation count. We extend this fixed-step recognizer with a controllable observation interface that adapts both the field of view and the number of fixations to the input. Specifically, the model predicts a per-fixation scale that sets its field of view, refines each selected crop with optional scale-gated foveation and a residual high-resolution detail-token branch, and uses label-grounded stop heads for input-dependent early termination. Experiments across fine-grained benchmarks, full ImageNet-1K, and ImageNet-A/R-derived classification splits show that the interface remains competitive with fixed-step references while exposing a controllable observation budget. The same checkpoint enables threshold-controlled accuracy-compute trade-offs without per-threshold retraining. On sequential ImageNet-1K, the learned stop rule outperforms fixed-step truncation of the same checkpoint, sustaining 81.9% top-1 using 2.45 of the four focus steps and exceeding three-step truncation at lower cost. On each benchmark, a variant combining scale control and detail-token fusion achieves the highest accuracy among the tested component-added non-exit variants. Our code is available at https://github.com/GZHU-DVL/LI-Dikai.

0.40

FG-DSA: Frequency-Guided Depthwise Separable Attention for Efficient Dense Prediction

cs.CV ·2026-11-07 ·Yiyao Yang
similarity 0.69
Abstract · 20 pages, 8 figures, 9 tables

Dense prediction and generative modeling in resource-constrained scenarios remain challenging, as repeated downsampling substantially reduces spatial detail and degrades boundaries. Although recent attention mechanisms have alleviated the problem of semantic ambiguity, they often fail to preserve high-frequency features or do so at the expense of a very high computational cost. In this paper, we propose the Frequency Guided Depthwise Separable Attention for Efficient Dense Prediction (FG-DSA), a novel framework for efficient dense visual understanding and generation. FG-DSA employs a 2D FFT high-pass filtering module to explicitly inject frequency domain priors into the spatial attention mechanism, allowing lightweight networks to recover sharp boundary transitions with linear complexity. A range of experiments show that FG-DSA is effective at recovering fine object boundaries and leads to a consistent performance gain in the task of semantic segmentation. Importantly, we show that FGA-DSA can be used as a module in diffusion-based generative U-Nets, leading to a significant improvement in the detail of the generated images with very little additional parameter and Floating Point Operation (FLOP) cost.

0.25

ElasticDino: Dense Semantic Features Using Elastic Deformations

cs.CV cs.AI ·2026-11-12 ·Ulysse Mizrahi, Sean Zaretzky, Dov Danon, Amit H. Bermano
similarity 0.69
Abstract

Vision transformers (ViTs) serve as the backbone for countless vision tasks, including dense prediction tasks, such as segmentation and depth estimation, even though the quadratic complexity of self-attention severely limits feature resolution and induces spatial artifacts. In this work, we introduce ElasticDino, a lightweight adapter which produces high-resolution feature maps, adhering to the fine details of the input image. As a key insight, we observe the consistency of artifacts across channels, and simultaneously resolve them to match the input image structure. Using a slim architecture based on spatial deformations, ElasticDino produces pixel-aligned, high-resolution feature maps efficiently. Leveraging deformations allows strictly preserving sharp semantics, and thereby avoiding the smoothness biases inherent to alternative approaches. Due to the absence of ground truth, we employ pixel-level semantic segmentation and depth estimation with linear heads to predict a set of downstream dense tasks, guiding the deformation process without leaking task-specific semantics into the model. We demonstrate superior accuracy over current methods on a range of popular dense task benchmarks, with better computational performance.

0.24

FoRIS: Progressive Foreground Refinement for Training-Free In-Context Segmentation

cs.CV ·2026-09-03 ·Ming Hu, Jianfu Yin, Mingyu Dou, Miaomiao Zhang, Yao Wang +3 more
code/data similarity 0.69
Abstract

In-Context Segmentation (ICS) aims to precisely segment arbitrary semantic concepts, such as objects or parts, given one or a few annotated visual exemplars. In this paper, we revisit ICS from a more classical segmentation perspective, viewing it as a coarse-to-fine progressive refinement process. Rather than directly predicting the final mask through reference-query matching, we progressively refine the segmentation from coarse and ambiguous foreground responses to precise and complete foreground structures. Building upon this perspective, we propose a training-free in-context segmentation framework, termed FoRIS. Specifically, FoRIS consists of three key stages: Foreground Purification, Foreground Localization, and Foreground Consolidation, which progressively suppress background distractions, localize discriminative target regions, and recover complete foreground structures through semantic aggregation. Experimental results demonstrate that FoRIS achieves SOTA performance across semantic and part segmentation tasks, with average improvements of 4.5 and 4.8 mIoU points over existing approaches in the 1-shot and 5-shot settings, respectively. Code: https://github.com/Xi-Mu-Yu/FoRIS.

0.32

CanopyKV: Remember beyond the Crop for Fast Active Visual Reasoning

cs.CV ·2026-11-18 ·Lei Yang, Dayan Wu, Yiliang Zhu, Xinze Liu, Hengjie Zhu +4 more
similarity 0.69
Abstract · 16 pages, 11 figures, including supplementary material

Active visual reasoning enables multimodal models to answer challenging visual questions by gathering fine-grained evidence through iterative image inspection and tool use. However, accumulating crops and reasoning tokens expands the key-value (KV) cache, making compression important for reducing memory and inference costs. Existing KV compression methods typically select historical tokens based on attention, recency, or redundancy. These criteria do not explicitly distinguish locally replaced evidence from complementary information needed for future exploration: each crop refreshes a local region, while earlier views may still support subsequent decisions. Discarding such evidence can impair accuracy or require additional search, while scoring and cache reorganization can offset the computational savings from compression. We introduce CanopyKV, a training-free KV compression framework that uses parent-child crop relationships to retain complementary evidence, keeps retained history stable, and protects the state needed for ongoing reasoning. Tool metadata provides retention signals without additional attention scoring, while compact KV storage reduces subsequent attention costs. Across VisualProbe, V*, HR-Bench, and MME-Lite, CanopyKV achieves state-of-the-art answer accuracy and end-to-end efficiency among the evaluated KV compression methods. On MME-Lite, it retains approximately one-quarter of Dense's total KV while improving accuracy by 10.5 percentage points and accelerating complete reasoning by $1.64\times$.

0.25

AffordAny: Open-World 3D Affordance Grounding from Monocular RGB Images via Vision-Language-Guided Geometric Reasoning

cs.CV ·2026-08-21 ·Junqi Wu, Kaihua Tang, Xuanwen Chen, Hongzhi Li, Jianqiang Huang +1 more
benchmarkcode/data similarity 0.68
Abstract · The code and dataset are publicly available. Code: https://github.com/lzlfwow/AffordAny. Dataset: https://modelscope.cn/datasets/lzlfwow/Aff…

Open-world 3D affordance grounding requires localizing functional object parts in 3D given free-form language queries. Existing methods typically assume pre-built object-centric 3D geometry and closed affordance ontologies, limiting deployment from raw RGB observations. We present AffordAny, an end-to-end framework that uses one monocular RGB image to construct large-scale text-conditioned 3D part supervision, ground affordances with a frozen vision-language model (VLM) guided decoder, and improve open-world generalization through pseudo-label self-training. Our automated pipeline produces a benchmark of 5,334 objects and 10,633 part-level samples spanning 473 categories, an order-of-magnitude increase in categorical diversity over prior work. The decoder progressively fuses frozen Cosmos-2B features with 3D geometry through spatial projection, instruction-conditioned semantic compression, and bidirectional geometry-semantics interaction. Minimal-perturbation pseudo-label self-training further adds new objects without human annotation. Under a systematic generalization protocol evaluating unseen objects, unseen categories, and unseen instruction paraphrases, our approach achieves 0.428 IoU on unseen objects and 0.315 IoU on unseen categories after self-training, with unseen-category mIoU improving by 6.3% relative (p<0.01) and an instruction sensitivity gap of only 0.105, demonstrating effectiveness and robustness of our method.

0.24