CS Paper Radar

SPECTRA-VLA: Base-Fine Hyperspectral Token Transfer at Frozen Vision-Language-Action Interfaces

Chih-Chung Hsu

cs.CV ·submitted 2026-10-14 06:17 UTC · v1

Abstract

Hyperspectral imaging distinguishes RGB-similar objects, but a pretrained vision-language-action model needs a state its frozen decoder can use. SPECTRA-VLA provides a native-width spectral upgrade without retraining the visual encoder or action decoder. A learned continuous residual enriches each target token through its own normalized false-color reference; projection, norm control and validation-calibrated injection regulate the change. Across 86 held-out acquisitions, category balanced accuracy rises from 0.812 to 0.883. With identical residuals and scale, own-token addition improves reuse of an external category-derived probe on frozen decoder logits: 0.701 versus 0.590 with pooled replacement, at similar image-disjoint recognition. Five grouped tests covering the same 86 acquisitions recover within-category spectral shape with $R^2=0.802/0.337$ from continuous and posterior-code states. Category-matched spectral swaps further show that own-token composition approximately doubles the target-specific recovery increment after frozen decoding. Spectral enrichment and receiving-token preservation thus work together at the pretrained interface.

Comment
34 pages: 11-page main paper and 23-page supplementary material

Closest papers

Nearest neighbours by meaning across the whole index.

Foveated Compression: Selective High-Resolution Preservation for Token-Efficient VLMs

cs.CV ·2026-10-06 ·Donghyun Han, Jangho Park, Yuseok Bae
published similarity 0.64
Abstract

Visual tokens are a major source of inference cost in vision-language models, yet simple image downsampling remains a surprisingly strong compression baseline. This raises a complementary question: under a fixed token budget, where should visual fidelity be preserved? We introduce Foveated Compression, which encodes a full-resolution image once and represents it with a mixture of native- and compressed-resolution visual tokens. A behaviorally self-distilled Foveated Merger compresses local visual tokens while preserving compatibility with their native counterparts, and a lightweight Foveated Selector chooses one of nine spatial cells to retain at native resolution using exhaustive budget-matched intervention supervision. At 11.11% visual tokens, uniform Foveated Compression shows no significant paired difference from iso-token downsampling. At 20.99%, the learned selector significantly outperforms random and fixed allocation, but remains below strong whole-image resizing, showing that localized fidelity is not universally preferable. A budget-matched region-choice oracle reaches 82.73 macro accuracy versus 69.61 for the learned selector, revealing substantial headroom within the same spatial action space. Matched probing further shows that signals predicting when compression breaks the answer are substantially more accessible after language-model computation than to the lightweight prefill-free selector. These results expose complementary bottlenecks in region selection and compressed-region fidelity.

0.32

Spectral-spatial vision test-time training for hyperspectral image classification

eess.IV cs.CV ·2026-11-02 ·Xiang Zhao
similarity 0.64
Abstract · 25 pages,9 figures,10 tables

Hyperspectral image (HSI) classification requires the representation of contiguous spectral signatures and spatial context under limited labeled supervision. Vision test-time training (TTT) offers an alternative to fixed-weight token mixing by performing self-supervised inner-model updates on the input sequence, but its application to center-pixel HSI classification remains insufficiently characterized. We present ViTTT-HSI, a spectral-spatial adaptation of the vision test-time training (ViT3) architecture. A spectral-aware embedding stem extracts ordered-band information using spectral convolution and forms spatial tokens without reducing their spatial resolution. Two ViT3-based TTT encoder blocks perform patch-conditioned token mixing, followed by the fusion of center-pixel and neighborhood-average features for classification. We evaluate the framework on Indian Pines, Pavia University, Salinas, and Houston 2013 using ten random seeds and dataset-specific training fractions of 1% or 5%. The mean overall accuracies are 97.6390%, 98.8753%, 99.3292%, and 98.4283%, respectively. Ablations reveal a substantial contribution from the TTT inner update on Indian Pines and a smaller positive contribution on Pavia University, while the relative benefits of the spectral stem and feature fusion depend on the dataset. The results demonstrate the feasibility of using patch-conditioned vision TTT for spectral--spatial HSI classification and highlight the role of the inner adaptation mechanism.

0.29

TIVS: Mapping the Sensitivity Landscape of Visual Token Transformations in MLLMs

cs.CV ·2026-11-15 ·Yikang Qiao, Jianye Zeng, Duan Huang, Ziying Song
similarity 0.61
Abstract · 9 pages, 12 figures, 3 tables

Recent efforts to improve the efficiency of multimodal large language models (MLLMs) have explored visual token pruning, quantization, and feature compression. However, the properties that determine whether transformed visual tokens remain usable by a frozen decoder are not well characterized. We introduce \textbf{TIVS (Token Identity Versus Statistics)}, a training-free intervention framework that separately studies sequence permutation, token retention, numerical precision, and statistical replacement under controlled conditions. On 192 POPE questions, three token permutations change TinyLLaVA accuracy by only $-0.17$ percentage points on average, while changing 4.69\% of individual predictions. Diagonal and rank-64 Gaussian replacements decrease accuracy by 25.0 and 28.1 points on the same subset. Further experiments with exact moment constraints, quantization error, token retention, and additional visual tasks reveal how strongly the effects depend on the original feature values and the evaluation setting. These findings provide an empirical basis for assessing visual token transformations in frozen MLLMs and designing more reliable multimodal interfaces.

0.16

ChangeT$^3$: Evaluating Vision Test-Time Training for Remote Sensing Change Detection

cs.CV ·2026-10-11 ·Sharukshan Niranjan, Oshadha Samarakoon, Nichula Wasalathilaka, Roshan Godaliyadda, Parakrama Ekanayake +1 more
similarity 0.61
Abstract · 8 pages, 2 figures, 6 tables

Bitemporal remote sensing change detection requires representations that preserve spatial detail while capturing contextual information relevant to changes between acquisitions. Visual test-time training offers an alternative to explicit token-to-token attention by adapting a compact inner model to each image. This letter describes ChangeT$^3$, a vanilla Siamese configuration combining the existing hierarchical ViT$^3$-Tiny encoder with the multiscale ChangeFormer decoder. The two image streams share learned encoder parameters, while their fast weights are computed independently from their respective spatial features. Each encoder block retains the inherited global simplified SwiGLU and local adaptive depthwise-convolution branches. Temporal comparison is performed by the decoder through learned fusion of paired features at four resolutions, followed by coarse-to-fine aggregation and binary prediction. This separation provides a direct setting for examining image-conditioned inner updates without introducing an additional temporal interaction module. Trained from scratch on LEVIR-CD+ with the same decoder, recipe, and budget as a ChangeFormer control, ChangeT$^3$ reaches an F1 of 80.61$\pm$0.90 against 78.84$\pm$0.21 for ChangeFormer (three seeds each); as in the standard protocol for this dataset, checkpoints are selected on the test split. Trained with both inner updates disabled, the same encoder reaches 78.63$\pm$0.20. Neither model transfers zero-shot to SYSU-CD, DSIFN-CD, or WHU-CD, and ChangeT$^3$ is the slower of the two at small inputs despite similar parameter and operation counts.

0.21

CatchUp: Depth-Aware Re-Admission of Visual Tokens for Efficient Large Vision-Language Models

cs.CV ·2026-10-16 ·Eunju Park, Uiwon Hwang
publishedcode/data similarity 0.60
Abstract · WACV 2027. Code: https://github.com/pej0918/CatchUp

Large Vision-Language Models (LVLMs) repeatedly process hundreds of visual tokens across decoder layers, incurring substantial computation and memory costs. Existing visual-token reduction methods often irreversibly discard tokens despite depth-varying query relevance. Although recoverable routing allows excluded tokens to return, re-admitted tokens retain stale hidden states after bypassing intermediate decoder transformations. Thus, token recoverability does not guarantee representation readiness at the current decoder depth. Under a fixed routing trajectory at a 64-token budget, replacing stale re-admitted states with current-depth oracle representations improves average RefCOCO Acc@0.5 from 18.96 to 23.18, revealing the impact of representation staleness. We propose CatchUp, a training-free framework combining reversible visual-token routing with Local Transition Correction (LTC). CatchUp repeatedly re-evaluates query-conditioned relevance over the complete visual-token set, preserving deferred tokens for later re-admission. LTC approximates their skipped representation evolution by transferring reference-to-current hidden-state transitions from semantically and spatially related active anchors. This correction improves alignment with current-depth oracle representations without recomputing the skipped decoder layers. Across three referring-expression grounding datasets and six multimodal benchmarks, CatchUp achieves the best grounding performance across all eight splits and the highest average multimodal retention among evaluated efficient methods at 64-, 128-, and 192-token budgets. At a 64-token budget, CatchUp processes 88.9% fewer active visual tokens per post-routing decoder interval, retains 92.17% of dense-model multimodal performance, and reduces end-to-end latency by 8.0%. Code is available at https://github.com/pej0918/CatchUp

0.34

Decoding Register Tokens Preserves Global Scene Details

cs.CV cs.LG ·2026-10-09 ·David Jacob Drexlin, Julius Hense, Jonas Dippel
similarity 0.60
Abstract · 31 pages, 7 figures

Pretrained vision transformers learn structured image representations spanning spatial patch tokens and non-spatial tokens. Recent analyses suggest that non-spatial register tokens encode global scene properties such as illumination, blur, and style. Yet patch-based image decoders discard these outputs, leaving the decoder to recover image appearance from patches alone. We investigate whether retaining register features improves decoding when the complete spatial patch representation is already available. We study this question using representation autoencoders (RAEs) with frozen register-equipped encoders. Our decoder processes four register tokens together with all patch tokens through joint self-attention. Appearance probes, paired reconstruction-error analysis, and fixed-decoder register substitutions examine which global image properties are accessible from registers and how their values affect reconstructed pixels. For generation, we train a multi-head attention pooling (MHAP) predictor to recover register features from sampled patches, keeping the pretrained diffusion model and its samples unchanged. On ImageNet-1K, our ViT-B decoders achieve mean reconstruction FIDs of 0.472 with DINOv2 and 0.442 with DINOv3, reductions of 9.5% and 5.2% relative to their respective patch-only baselines. The corresponding PSNRs are 19.73 and 20.54 dB. For generation, our DINOv2 models achieve a mean FID of 1.641, 9.8% below the patch-only baseline on identical sampled patches.

0.22

Beyond Reconstruction Error: Analytical and Data-Driven Action Tokenization for Autoregressive Vision-Language-Action Models

cs.RO cs.LG ·2026-09-22 ·Yuxin Yang, Gaohan He, Changxue Guan, Hangming Liu
similarity 0.60
Abstract · 5 pages, 1 figure, 4 tables

Discrete action tokenization is central to autoregressive vision-language-action (VLA) models, yet action representations are often evaluated primarily through reconstruction fidelity. We ask which representation properties actually matter for closed-loop control by comparing fixed analytical, data-driven linear, and nonlinear neural representations under a unified tokenization interface. Across rate-distortion analysis, sequence-modeling diagnostics, and 3,500 LIBERO rollouts, representation rankings change with the evaluation criterion. PCA achieves lower nominal reconstruction error than Temporal-DCT, but produces less predictable token sequences and 3.0 percentage points lower mean seen-task success across three policy-training seeds, with the policy ordering reversing in one seed. In a matched seed-42 ablation, an autoencoder further reduces reconstruction error yet does not yield the strongest policy and exhibits greater sensitivity to discrete token perturbations. These findings show that reconstruction fidelity alone cannot reliably select action representations for autoregressive control, motivating joint evaluation of geometric fidelity, sequence predictability, decoder stability, and closed-loop performance.

0.19

The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence

cs.CV cs.CLcs.LG ·2026-09-01 ·Genpei Zhang
similarity 0.60
Abstract · 17 pages (7-page main text plus technical appendix), 10 figures, 6 tables

Vision-language models are evaluated by aggregate accuracy on multimodal benchmarks, a practice that implicitly assumes the model uses its visual input. We show this assumption fails on 40%--97% of samples across six VLMs and three perceptual benchmarks: blurring the question-relevant visual region leaves the next-token distribution nearly unchanged. We name this phenomenon the Visual Insensitivity Gap and quantify it with a per-sample Visual Sensitivity Index (VSI). The gap is a property of samples, not of models: VSI ranks correlate across models (grand-mean Spearman rho=+0.40, permutation p<10^-3), so the same samples are flagged insensitive by VLMs sharing no architectural detail beyond a contrastively pretrained vision tower. The mechanism is concrete: on the insensitive samples, a linear probe on each model's own vision tower distinguishes perturbed from clean images at 0.72--0.79 accuracy, yet the model's argmax token changes on only 2%--11% of the same samples, an encoder--LLM gap above 0.65 on every model. Mapping VSI's diagnostic utility cell by cell surfaces a strong regime (multi-choice reasoning on capable VLMs: AUROC=0.85--0.87) and a weak regime (well-calibrated factuality, where softmax confidence already leads). VSI is not a universal best abstention signal; it is a sample-intrinsic indicator of vision-ignoring failure, best used as a conditional ensemble component.

0.52