CS Paper Radar

DEFTS: Data-Efficient, Training-Free Few-Shot Learning for CLIP in the Text Subspace

Yijin Zeng

cs.CV stat.ML ·submitted 2026-11-16 23:23 UTC · v1

Abstract

Contrastive Language-Image Pre-training (CLIP) achieves impressive zero-shot image classification performance by comparing image embeddings against class-specific text embeddings. While recent few-shot methods leverage labeled training samples to improve upon this zero-shot baseline, they often predominantly rely on image embeddings and underutilize the semantic structure of the text embeddings. In this paper, we introduce DEFTS, a data-efficient, training-free few-shot classification method strictly in the text subspace. We first demonstrate empirically that while raw text embeddings can be suboptimal as direct classifiers, the linear subspace they span remains highly discriminative. DEFTS therefore proposes to reduce the image-text modality gap by projecting estimated image prototypes directly into this subspace. Importantly, by expressing these prototypes as linear combinations of text embeddings, DEFTS reduces a high-dimensional estimation problem to the derivation of just a few coefficients. These coefficients are then computed via a Bayesian posterior mean, using a prior naturally centered on the original text embedding coordinates. Extensive experiments across 12 image classification benchmarks demonstrate that DEFTS significantly improves upon existing training-free methods.

Closest papers

Nearest neighbours by meaning across the whole index.

When Modality Gap Reduction Fails: Prediction-Level Hubness in CLIP

cs.CL cs.CV ·2026-09-01 ·Shota Sato, Hajime Kiyama, Tosho Hirasawa, Mamoru Komachi
top venue similarity 0.69
Abstract · Accpeted to EMNLP 2026 Main Conference

Reducing the modality gap between image and text representations in CLIP is widely expected to improve cross-modal alignment and downstream performance. However, a smaller average image-text gap does not necessarily lead to consistent accuracy gains. We analyze this mismatch from the perspective of the decision structure in zero-shot classification, i.e. selecting the most similar class-text prototype for an input image. Zero-shot accuracy depends not only on average image--text alignment, but also on class-wise decision margins. Using Linear correction as an analytically tractable case, we show that modality gap correction can alter the relative decision structure among classes and cause predictions to concentrate on a small subset of classes. We refer to this output-space failure mode as prediction-level hubness. Furthermore, experiments across multiple datasets show that accuracy degradation under gap correction is consistently associated with increased prediction concentration, both for Linear correction and for learning-based correction methods. This provides a systematic explanation of why modality gap reduction does not consistently improve CLIP zero-shot accuracy from the perspective of downstream decision structure. Our results suggest that gap correction should be evaluated not only by average alignment, but also by its impact on downstream prediction structure.

0.58

Domain Recentering and Confidence-Weighted Prior Calibration for Vision-Language Models

cs.CV cs.AI ·2026-09-24 ·Youngeun Seol, Jimin Shin, Heeseo Yoon, Uiwon Hwang
similarity 0.68
Abstract

Vision-language models such as CLIP achieve strong zero-shot classification, yet under distribution shift, visual embeddings drift from fixed text embeddings. Training-free calibration avoids the per-sample optimization of prompt learning, but prior feature calibration gives each image the full bias of one hard cluster. We propose Domain Recentering with Confidence Calibration (DRC), a training-free method adapting CLIP from a set of unlabeled target images. DRC fits a Gaussian mixture once and subtracts from each embedding a posterior-weighted average of component means. It then removes residual class preference with a log-prior correction, estimating the prior from confidence-weighted predictions. Among compared methods, DRC achieves the highest average accuracy on cross-domain datasets, exceeding zero-shot CLIP by 4.13 and 5.07 points with ViT-B/16 and ResNet-50, with gains over CLIP also holding under ImageNet distribution shifts.

0.15

DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations

cs.CV ·2026-08-20 ·Hangyu Tian, Zhenqi He, Yanghao Wang, Long Chen
similarity 0.68
Abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions by leveraging knowledge of primitive concepts learned from seen compositions. Although recent works achieve impressive performance in CZSL by leveraging large vision-language models, they primarily rely on discriminative representations that may not explicitly preserve the structured relationships between primitive concepts and their compositions. Motivated by the recent success of diffusion-based classifiers and their competitive performance relative to discriminative models, we investigate whether intermediate diffusion representations can provide complementary cues for CZSL. To this end, we propose DIFFCZSL, a diffusion-augmented framework that injects generative priors from pre-trained diffusion models into CLIP-based CZSL pipelines. We extract intermediate diffusion representations and project them into the CLIP embedding space to provide auxiliary supervision on both image and text modalities. Through contrastive alignment between CLIP embeddings and diffusion features during training, our method encourages the embedding geometry toward richer composition-aware semantics, while introducing no additional cost at inference time. Extensive experiments on three public CZSL benchmarks demonstrate consistent improvements over strong CLIP-based baselines under both closed-world and open-world settings. Our results highlight the complementary strengths of generative diffusion representations and discriminative vision-language models for compositional generalization.

0.26

Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal

cs.CL ·2026-10-16 ·Ailiang Lin, Zhuoyun Li, Yusong Wang, Keyu Mao, Kotaro Funakoshi +1 more
similarity 0.68
Abstract

While deriving text embeddings directly from frozen LLMs has emerged as a promising direction, existing methods focus almost exclusively on English, leaving multilingual settings largely unexplored. In this work, we present the first systematic study of multilingual training-free text embeddings in LLMs. Our analysis reveals a language-conditioned embedding geometry in which semantically equivalent sentences remain misaligned across languages. We hypothesize that language-specific variation can be disentangled from semantic content to improve embedding quality. To this end, we propose Paired-Residual Subspace Removal (PRSR), a training-free inference-time intervention that steers LLM representations toward a more language-invariant semantic space. PRSR first constructs paired residuals from multilingual parallel sentences to capture language-specific deviations. It then estimates layer-wise low-rank subspaces from these residuals and suppresses the corresponding components in the last-token hidden states. We conduct more than 200 monolingual evaluations across 18 languages using MTEB and MMTEB, along with cross-lingual bitext mining between English and over 200 other languages. Empirical results demonstrate that PRSR consistently improves the performance of existing training-free baselines in both monolingual and cross-lingual settings across different LLMs.

0.20

Exact Degeneracy Under Balanced k-Shot Sampling:Consequences for Small-Sample Discriminant Analysis on LLM Embeddings

cs.LG ·2026-09-09 ·Lingxiao Qu
similarity 0.67
Abstract

Balanced k-shot sampling, drawing exactly k labeled examples per class, is the standard protocol in few-shot evaluation. We show that it induces an exact degeneracy in Kernelized Linear Principal Component Discriminant Analysis (KLPCDA), a family of seven supervised dimensionality reduction methods. The within-class operator becomes exactly a scaled orthogonal projector, so two variants have all signal eigenvalues equal and a third has a zero operator, and the truncated pseudo-inverse used to implement them removes exactly the subspace that contains all between-class information. We verify this to float64 precision on sentence embeddings and on residual-stream activations of a decoder-only model, and find that the accuracies of the truncated variants change with the number of BLAS threads, by up to 12 percentage points (57 for the variant with a zero operator). Replacing the truncation with a ridge selected within the support set, we evaluate few-shot text classification on frozen embeddings (d up to 4096) over four datasets, three embedding models, and three trained baselines. The regularized formulation outperforms the best original variant in 45 of 48 settings, but in general not a tuned linear probe or nearest centroid on bidirectional encoders. Its gains are largest on E5-Mistral, the only decoder-based embedding model, and come from regularized within-class whitening rather than from KLPCDA: shrinkage LDA outperforms both baselines in all 16 E5-Mistral settings, and pre-specified tests show that the effect depends on the embedding model rather than on its dimension. E5-Mistral's remaining gap is largely an estimation-efficiency effect: increasing k to 30-50 reduces the probe's gap to a bidirectional encoder by more than 80%. We give five guidelines limited to the conditions tested and release the code and per-seed results.

0.40

G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification

cs.CV ·2026-08-27 ·Zehua Hao, Fang Liu, Qinliang Wang, Yaoyang Du, Xinyan Huang +1 more
top venuecode/data similarity 0.67
Abstract · Accepted at ACM MM 2026. 10 pages, 5 figures

Zero-shot classification needs efficient label retrieval and fine-grained visual reasoning, yet discriminative and generative vision-language models fail in complementary ways.When CLIP's top-1 prediction is wrong, the correct label often remains in its top-$K$ shortlist, making disambiguation rather than recall the key challenge.Standalone generative models, however, are hindered by large label spaces and unconstrained outputs.This complementarity motivates separating broad candidate retrieval from fine-grained, image-grounded verification.We propose G2D, a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image.Candidate names and CLIP probabilities provide a structured prior for resolving visually similar classes.Fixed confidence routing, entropy-adaptive candidate sizing, and trie-constrained decoding focus generative reasoning on uncertain samples and ensure one valid output for each input at test time.Across eight benchmarks, G2D achieves 68.85% average accuracy, versus 59.35% for CLIP and 63.11% for the standalone VLM.Across seven generator configurations, candidate-set verification improves average accuracy by 1.08--27.42 percentage points.G2D also transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning. Code: https://github.com/Harzva/G2D

0.50

Visual Memory and Textual Alignment for Cross-Modal Few-Shot Segmentation

cs.CV ·2026-10-19 ·Tianqi Wei, Zhi Chen, Xin Yu, Scott Chapman, Zi Huang
similarity 0.67
Abstract

Few-shot segmentation (FSS) aims to segment target objects with limited annotated examples. Recent studies have begun to explore large foundation models to improve generalization for this task. In particular, SAM 3 provides strong visual representations and supports text-guided segmentation. However, its performance remains limited in specialized domains, where semantic prompts and visual representations are often poorly aligned. To address this challenge, we propose ViTA-SAM, a framework built upon SAM 3 that integrates visual memory and textual features through cross-modal alignment. Specifically, we construct a visual memory module that encodes support features together with their corresponding masks, providing support-aware visual guidance for query prediction. In parallel, we leverage the text encoder of SAM 3 to extract semantic features as complementary priors. To bridge the gap between semantic and visual representations, we introduce a cross-modal alignment module that conditions textual features on memory-enhanced visual representations, yielding more consistent and visually grounded predictions. In addition, we propose a pseudo-label refinement strategy that updates the memory with predicted query masks, enabling query-adaptive representation learning. Extensive experiments on multiple benchmarks show that ViTA-SAM achieves the best overall performance among the compared methods, outperforming the strongest competing method by over 5% in average mIoU.

0.17

Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models

cs.CV cs.AIcs.LGcs.MM ·2026-10-29 ·Yubo Cui, Xianchao Guan, Zheng Zhang
similarity 0.66
Abstract

Vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial attacks. Existing adversarial fine-tuning methods often use pre-trained model guidance to preserve image--text alignment while improving robustness. Our analysis of visual representation drift reveals complementary limitations of two representative forms of guidance. Matching visual features constrains all directions equally, providing broad coverage without semantic emphasis. Matching prediction distributions prioritizes semantically relevant directions, but remains limited by the candidate texts and prediction probabilities. These limitations can hinder preservation of pre-trained image--text relationships and degrade zero-shot performance. We therefore propose Textual Response-Aware Guidance (TRAG), which constrains visual representation drift using the pre-trained text encoder's responses to local variations in input embeddings. TRAG constructs anchor-level constraints from the response directions and magnitudes around semantic anchors, then aggregates them using pre-trained image--anchor relevance, yielding an image-adaptive constraint. Extensive experiments across architectures and tasks demonstrate state-of-the-art clean and robust performance. Further analysis shows that TRAG helps preserve pre-trained image--text relationships.

0.11