CS Paper Radar

Adaptive Visual Token Reduction for Accelerated Image Understanding

Seyoung Jeong, Jong Pil Yun, Sang Jun Lee

cs.CV ·submitted 2026-10-07 00:37 UTC · v1

Abstract

Large Vision-Language Models achieve strong VQA performance, but processing high-resolution, information-rich images requires substantial computation, motivating visual token reduction. However, existing methods often prune individual tokens or rely on fixed-size cropping, limiting their ability to preserve spatially structured information such as horizontally or vertically elongated text. To address this limitation, we propose ReFIT, an instruction-guided visual token reduction framework for efficient LVLM inference. ReFIT consists of Relevance-Guided Window Reshaping (RWR) and Instruction-Guided Token Refinement (ITR), where RWR captures instruction-relevant regions by adapting to their spatial characteristics, while ITR further removes unnecessary visual tokens. Experiments on four VQA benchmarks demonstrate that ReFIT improves answer accuracy while reducing computational cost, and qualitative results demonstrate its effectiveness in localizing relevant regions and removing unnecessary visual information.

Comment
5 pages, 2 figures. Under review

Closest papers

Nearest neighbours by meaning across the whole index.

When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model

cs.CV cs.LG ·2026-09-28 ·Minchan Kang, Kyeonghye Park, Seoyoung Cho, Daeshik Kim, Yucheol Cho
code/data similarity 0.79
Abstract

Visual token pruning has been widely studied as a practical approach to reducing the computational cost of large vision-language models. However, it struggles to preserve essential visual information, which can lead to substantial performance degradation. In particular, image-based token selection can overlook task-relevant details, while text-guided token selection may fail to capture the text--visual relationships needed for complex reasoning. We find that applying textual guidance too early can limit its ability to identify answer-relevant visual regions, whereas text-to-visual attention becomes more informative at intermediate decoder depths. This finding motivates our training-free method, which separates early vision-guided pruning from deferred text-guided reselection. We first prune visual tokens using vision-encoder attention, retain additional candidates until the decoder midpoint, and then use text-to-visual attention to determine the final visual-token set. Across eight benchmarks and three models, our method outperforms the best-performing baselines by an average of 11.10 and 16.84 percentage points in performance recovery at 80% and 90% pruning, respectively, with comparable or lower LLM-prefill latency than most baselines. The source code is publicly available at https://github.com/kmc3661/DeFT

0.30

Multi-Image Visual Token Pruning in Large Visual Language Models

cs.CV ·2026-08-27 ·Rongyang Zhang, Chengqiang Lu, Cong Li, Hongchao Gu, Tingjia Shen +7 more
publishedcode/data similarity 0.77
Abstract · 14 pages, 3 figures, EMNLP 2026 Findings

With the growing demand for processing multiple image sequences in real-world applications, various visual token pruning methods have emerged to mitigate the computational and context length constraints faced by Large Vision Language Models (LVLMs). However, most existing pruning approaches rely on static strategies that struggle to adapt across different architectural LVLMs and multi-image scenarios, and are additionally constrained by their dependence on attention computations that are incompatible with efficient techniques like FlashAttention. To address these limitations, we propose a training-free, Adaptive Visual Token Pruning (AVTP) framework, applicable to diverse LVLM architectures. We strategically determine pruning layers based on empirical analysis of visual attention distributions across various LVLMs, and implement adaptive pruning ratios in multi-image contexts where images of higher importance retain proportionally more tokens. We conduct extensive experiments across different LVLMs to demonstrate the effectiveness and robustness of AVTP. Specifically, Qwen3VL-8B achieves 2 times inference speedup while maintaining 96.1\% of its original accuracy on multiple multi-image benchmarks, InternVL3.5-8B retains 94.1\% accuracy, and LLaVA-OV-7B even exceeds its original baseline performance. Our code is available at \href{https://github.com/zry13/AVTP}{this link}.

0.67

Hierarchical and Query-Aware Visual Token Pruning for High-Resolution LVLMs

cs.CV ·2026-10-16 ·Luther Wei, Ling Zou, Kang-Yang Huang, Yu-Syuan Xu, Wen-Huang Cheng
similarity 0.77
Abstract · 6 pages, 2027 ICCE

High-resolution Large Vision-Language Models (LVLMs) preserve critical, fine-grained visual details by encoding a global thumbnail alongside multiple local crops. However, the resulting thousands of vision tokens introduce severe computational bottlenecks during Large Language Model (LLM) inference. Existing token pruning methods rely primarily on query-agnostic visual saliency, while current query-aware approaches either operate at a coarse crop-level resolution or compute token relevance too late, after vision tokens have already entered the heavy LLM layers. To address this, we propose a training-free, hierarchical token pruning framework that utilizes query guidance prior to executing any LLM blocks. Our top-down hierarchy first operates at the instance-level, dynamically scaling the vision token retention ratio of each sample based on its visual complexity. Next, at the crop-level, we distribute this sample budget across local crops using sentence-level cross-modal similarity. Finally, at the token-level, we exploit projected vision tokens alongside textual query token embeddings in the shared space to measure fine-grained alignment, applying a text-debiasing normalization to produce high-fidelity query scores. These query-guided budgets and scores are fused with local and global visual saliency metrics to select the most critical vision tokens. Extensive experiments on LLaVA-NeXT-7B demonstrate that our plug-and-play framework retains 98.0% and 94.8% of unpruned performance at 25% and 10% vision token retention, respectively. Notably, at 25% retention, our method delivers a 2.0x inference throughput speedup and even surpasses the unpruned baseline on POPE by 0.7 points.

0.17

PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference

cs.CV cs.AI ·2026-08-27 ·Junjie Liu, Shengyuan Ye, Xu Chen
top venuecode/data similarity 0.77
Abstract · 22 pages, 9 figures, 13 tables. Accepted to Findings of EMNLP 2026

Vision-Language Models (VLMs) demonstrate exceptional visual reasoning capabilities, yet their inference costs escalate rapidly with the proliferation of visual tokens. Existing visual token pruning methods exhibit two fundamental limitations. First, most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized. Second, under strict token budgets, these methods often fail to jointly preserve holistic visual contexts and fine-grained details, leading to performance degradation. To address these bottlenecks, we propose PACE (Pixel-Adaptive Condense and Extract), a training-free inference framework that accelerates both the vision encoder and the Large Language Model (LLM) via a unified Condense-and-Extract paradigm. During the Condense stage, an Adaptive Pixel Compressor (APC) evaluates visual information density prior to encoding, adaptively downsampling redundant inputs, curtailing encoder computation while preserving global context and essential visual cues. In the Extract stage, a Dynamic Dual-Attention Extractor (DDAE) selectively retains visual tokens via a fusion of internal visual signals from the encoder and semantic signals from the LLM, safeguarding task-critical details. By integrating PACE into Qwen2.5-VL-7B, the model retains 93.8% of its original performance while utilizing only 10% of the visual tokens, yielding a 3.1x speedup in time to first token (TTFT). Our code is available at https://github.com/jjL357/PACE.

0.68

VPRune: Efficient Training-free Pre-LLM Visual Token Pruning

cs.CV cs.AI ·2026-09-21 ·Guangchuan Lv, Dianxing Shi, Dingjie Fu
similarity 0.75
Abstract

Visual token pruning is a promising approach to reducing the inference cost of large vision-language models (LVLMs), yet aggressive token reduction often causes substantial performance degradation. We identify three key factors behind this degradation: text-guided selection bias, information loss from discarded tokens, and positional distortion caused by sequence compaction. Based on these observations, we propose \textbf{VPRune}, a training-free pre-LLM pruning framework consisting of visual-only diversity selection, similarity-guided token recycling, and position-preserving restoration. Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression. Furthermore, evaluations on edge-device show that VPRune effectively reduces end-to-end inference latency while maintaining superior task performance, demonstrating its practicality for resource-constrained LVLM deployment.

0.12

When Vision Becomes Text: Visual Token Pruning via Cross-Modal Residual Guidance in VLMs

cs.CV ·2026-08-11 ·Congyang Ou, Ruike Song, Yang Zhou, Libo Sun, Haokui Zhang +1 more
similarity 0.75
Abstract

Abundant visual information strengthens vision-language model (VLM) perception, yet massive visual tokens raise inference costs. Existing visual token pruning methods rely on similarity-based guidance, which exploits pairwise text-vision and vision-vision token correlations for compression. However, such methods only capture local layer-level signals and overlook the whole inference process in VLM. In this paper, we revisit VLM inference and present a new efficient guidance scheme that complements similarity-based guidance. In particular, we identify a key observation: as LLM layers deepen, text tokens continuously aggregate visual information via self-attention and progressively absorb partial visual content into textual representations. To quantify this phenomenon, we propose Cross Modal Absorption (CMA) from a geometric representation perspective to measure how much visual information is absorbed by text, revealing that more visual tokens in deeper layers can be approximately explained by the text subspace. We accordingly propose Cross Modal Residual (CMR). It projects visual tokens onto the text subspace via Tikhonov regularized least squares and exploits reconstruction residuals to quantify visual information that cannot be explained by text. Finally, based on CMR, we present SIEVE, a training-free visual token compression method that combines CMR, text-attention relevance, and residual-space diversity to retain task-relevant and complementary tokens. Experiments on diverse VLM architectures verify the effectiveness of SIEVE. For instance, on LLaVA-NeXT-7B, SIEVE keeps only $11.1\%$ of visual tokens while preserving $97.5\%$ of the original average performance, achieving $3.62\times$ prefill speedup, $2.49\times$ end-to-end speedup, and a $6.02\times$ KV-cache reduction.

0.34

STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models

cs.CV cs.AI ·2026-09-05 ·Yichen Guo, Tinghao Wang, Qizhe Zhang, Lingbei Meng, Yuan Zhang +13 more
code/data similarity 0.74
Abstract · 26 pages, 7 figures

Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose substantial computational overhead, motivating training-free visual token pruning. In this work, we conduct two complementary analyses of visual token pruning. First, we measure the feature-space coverage of tokens retained before cross-modal fusion and find that aggressive pruning discards substantial visual information. Second, we track text-to-visual attention across decoder layers and find that the visual tokens considered important change substantially with depth, making one-shot pruning decisions unreliable. Together, these findings show that effective pruning should preserve broad visual coverage before fusion and progressively refine the retained tokens as cross-modal evidence evolves during fusion. We therefore propose STAR-Pro (STage-Wise Adaptive Token Reduction with Progressive Refinement), a training-free two-stage framework. Its Adaptive Stage applies pivoted QR to construct an over-budget feature-coverage candidate pool, while its Progressive Stage uses evolving text-to-visual attention at selected decoder layers to prune a nested survivor set under a target layer-average token budget. Extensive experiments across seven LVLMs spanning multiple architectures and 18 image and video benchmarks demonstrate the effectiveness of STAR-Pro under aggressive pruning. On LLaVA-Video-7B, STAR-Pro reduces visual tokens by 90.5%, retains 92.7% of baseline performance, and achieves a $2.24\times$ measured inference speedup. Code is available at https://github.com/EasonAI-5589/starpro.

0.52

ACPruner: Visual Token Pruning as Biased Attention Coverage Maximization in LVLMs

cs.CV ·2026-09-28 ·Xu Li, Yuxuan Liang, Yi Zheng, Zhe Liu, Xiaolei Chen +4 more
similarity 0.74
Abstract

Large Vision-Language Models (LVLMs) face significant computational inefficiencies caused by the large number of visual tokens. Existing visual token pruning methods mainly focus on either retaining individually important tokens or selecting mutually diverse ones. In this work, we revisit visual token pruning from a coverage perspective and formulate it as a biased attention coverage maximization problem. The key idea is to select a compact token subset whose encoder-side outgoing attention can jointly cover the image while assigning higher coverage priority to more informative regions. From this perspective, we propose ACPruner, a training-free visual token pruning framework for efficient LVLM inference. ACPruner first estimates token importance by combining intra-modal saliency and inter-modal relevance, then derives token-wise coverage from attention patterns within the vision encoder, and finally performs greedy selection to maximize the proposed coverage objective. Extensive experiments across multiple LVLM backbones, including LLaVA-1.5-7B/13B, LLaVA-NeXT-7B/13B, Qwen2.5-VL-7B, and LLaVA-OneVision-7B, show that ACPruner consistently achieves strong performance retention while delivering substantial end-to-end inference speedups.

0.17