CS Paper Radar

Can a Vision-Language Model Tune Clustering In Context?

Javith Ranatunga, Paimon Goulart, Evangelos E. Papalexakis

cs.LG ·submitted 2026-11-12 02:28 UTC · v1

Abstract

Clustering is a widely used unsupervised learning method, but selecting an appropriate algorithm and set of hyperparameters for a specific dataset can require many possible configurations and evaluations. In this work, we investigate whether a Vision-Language Model can instead visually evaluate clustering results and use those decisions to then tune an explicit clustering algorithm. We study this through three progressive experiments using Qwen3.5-27B and five controlled synthetic clustering scenarios. First, we test whether the VLM can distinguish the best clustering result from a pair of plots. Second, we test whether it can assign numerical scores to those plots where a higher score represents a better result. Finally, we use these judgments within an agentic loop in which the VLM scores a k-means clustering and proposes updated hyperparameters that may result in better clustering. Our exploratory results show that the VLM performs well when clear cluster structure is present, while struggling on more ambiguous and noisy settings. In the full agentic experiment, the VLM produces clustering configurations with silhouette scores relatively close to those of grid search and Bayesian optimization, while terminating in much fewer iterations. Overall, these results provide an initial proof of concept that VLM-based visual feedback can be used to autonomously tune an explicit clustering algorithm.

Closest papers

Nearest neighbours by meaning across the whole index.

VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training

cs.CV ·2026-09-29 ·Yuanwei Hu, Bo Peng, Yuheng Jia, Xinting Hu, Yadan Luo +1 more
benchmarkcode/data similarity 0.70
Abstract

Vision-language pre-training has reshaped image clustering, giving rise to language-assisted image clustering (LaIC), which leverages textual semantics to complement visual representations. Despite the rapid proliferation of LaIC methods, it remains unclear how much LaIC has actually advanced image clustering, as existing studies generally suffer from major limitations, including inconsistent experimental settings, inadequate dataset selection, and limited evaluation dimensions. To address this gap, we introduce VLM4Cluster, a comprehensive benchmark for image clustering in the era of pre-trained vision-language models (VLMs). VLM4Cluster implements 17 representative methods spanning classical, deep, and language-assisted image clustering, and evaluates them on 20 datasets covering classical, challenging, fine-grained, large-scale, and out-of-distribution settings. Beyond effectiveness, VLM4Cluster systematically investigates image clustering along three complementary dimensions: robustness to adversarial perturbations, generalization under distribution shifts, and computational efficiency. Our study shows that LaIC substantially advances the clustering performance frontier on many semantically demanding benchmarks, generally exhibits stronger generalization under distribution shifts, and achieves a more favorable effectiveness-efficiency trade-off. However, its gains become less consistent on large-scale and fine-grained datasets, while language assistance does not systematically reduce sensitivity to adversarial perturbations. VLM4Cluster is released at https://github.com/YuanweiHuu/VLM4Cluster.

0.32

CAESAR: Clustering via Autonomous Embedding-Space Agglomerative Reorganization

cs.LG ·2026-09-26 ·Ilan Bacry, Rémi Devaux, Antoine Jardin
benchmark similarity 0.65
Abstract · 14 pages, 3 figures

Clustering algorithms that operate on nearest-neighbor graphs, such as FINCH (First Integer Neighbor Clustering Hierarchy), depend heavily on the quality of the embedding space they are given. However, pretrained vision and language model embeddings are not optimized for this purpose. We propose CAESAR, a method that reorganizes a pretrained embedding space: a reorganization network is trained to pull mutual nearest neighbors together and push non-neighbors apart, yielding a reorganized embedding space substantially better suited to clustering. CAESAR offers a second major advantage: it never requires the number of clusters $K$. This matters because in realistic unsupervised settings, $K$ is typically unknown and discovering it is often part of the problem, yet most strong clustering methods take it as an input. We therefore design the entire CAESAR pipeline to infer $K$ rather than assume it is known. Empirically, reorganizing the embeddings consistently improves clustering over the raw space on both text and image datasets. Since the few deep clustering methods that also infer $K$ do not release their code, we complement controlled comparisons with methods that infer $K$ on the same embedding space by comparisons with strong deep clustering methods that are given the true $K$, giving them a substantial oracle advantage. Even so, CAESAR outperforms all of them on text, achieves the best results on the most challenging image benchmark and remains competitive on the others. Reorganizing pretrained embeddings thus emerges as a simple and powerful route to clustering realistic data, where classes overlap and the number of clusters is unknown.

0.18

Cluster Validation Indices as Self-Supervised Objectives for Text Representation Learning

cs.CL cs.AI ·2026-10-04 ·Kishor Kumar Bhaumik, Nicolas Roque dos Santos, Neil Shah, Jia Chen, Evangelos E. Papalexakis
similarity 0.64
Abstract

Self-supervised fine-tuning refines the embedding space of a pretrained language encoder without labels. However, the commonly used approaches are computationally expensive. Specifically, contrastive learning-based methods need multiview data and in-batch negative examples, while negative-free approaches require auxiliary graphs/networks. An interesting question arises: can self-supervised fine-tuning be done without relying on either additional negatives or graph data? To answer this question, we introduce SilK (Silhouette-guided K-means), which trains on a Cluster Validation Index, an internal measure of cluster quality without using labels. SilK clusters the corpus and then regresses a simplified silhouette toward a target value. Each document is compared only against the k cluster centroids, never against other documents, so the method needs no augmentation, no negative pairs and one view per document. On BERT-base, SilK trains 1.46x faster per epoch than the fastest baseline we evaluate and uses 45.4% less peak GPU memory than the leanest one. Under frozen-encoder linear probing, SilK stays competitive with the best baselines on three downstream tasks.

0.34

Image Clustering with Vision-Language Model Embeddings

cs.CV cs.AIcs.LG ·2026-10-23 ·João P. Matos-Carvalho, Nuno Fachada
similarity 0.63
Abstract

Pretrained vision-language models provide image representations whose suitability for clustering may vary with feature extraction, image domain, and partitioning algorithm. We compare nine representation pipelines across seven image benchmarks and four clustering configurations, yielding 252 baseline combinations. Eight pipelines extract visual features, including two vision-only references; the ninth generates descriptions and embeds the resulting text. With agglomerative clustering (Ward linkage) fixed, Qwen3-VL achieves the highest adjusted Rand index (ARI) on five datasets, including 0.925 on MNIST, while BLIP-2 leads on CIFAR-10 and CIFAR-100 with ARIs of 0.925 and 0.591, respectively. A separate sensitivity analysis compares raw and L2-normalized embeddings at their original dimensionality across all 63 dataset-representation pairs. Normalization preserves the highest-ARI representation in every dataset, increases ARI in 35 pairs, decreases it in 26, and leaves two numerically unchanged. On the SVHN dataset, the highest ARI remains low, rising from 0.097 to 0.098 after normalization. The evaluated caption-mediated pipeline obtains relatively high ARI on isolated digits but does not consistently exceed direct visual features. These findings support selecting and evaluating complete representation-clustering pipelines for the intended domain, with explicit attention to extraction and normalization choices.

0.07

ClusterSplat: Semantic Cluster Selection for 3D Visual Grounding in Gaussian Splatting

cs.CV ·2026-10-27 ·Jaesung Lee, Hyeontaek Hwang, Gabriel Manalu, Daeyoung Kim
top venuecode/data similarity 0.61
Abstract · Accepted at NeurIPS 2026 as a Spotlight paper. Project page: https://jaesung8.github.io/clustersplat/

We study 3D visual grounding in 3D Gaussian Splatting (3DGS), where a referring expression should identify a target object as both a rendered 2D mask and a set of explicit 3D Gaussians. Prior methods typically extract 3D targets by applying heuristic thresholding or top-ratio filtering to dense primitive-level relevance scores. However, these scores often vary substantially across scenes and prompts, making such heuristics sensitive and leading to unstable targeting. ClusterSplat addresses this by reframing 3D target selection as cluster-level selection over scene-adaptive candidates. The method first learns instance-aware Gaussian features, forms feature-aware seed voxels, and merges neighboring seed voxels when a description-length criterion decreases, producing a scene-specific set of cluster candidates. Given a referring expression, a query-conditioned cluster scorer ranks the cluster candidates with a lightweight MLP, and the selected cluster directly defines the explicit 3D target while its cluster score is rendered for the 2D mask. On ScanRefer, ClusterSplat achieves the best rendered 2D segmentation and explicit 3D target metrics among the compared 3DGS baselines. It also obtains the best average Ref-LERF scores, supporting the same trend under fine-grained referring descriptions. On ScanNet v2 class name queries, ClusterSplat achieves the strongest rendered 2D results and the highest 3D matching accuracy, while maintaining competitive 3D IoU. Our project page is available at https://jaesung8.github.io/clustersplat/ .

0.49

Hyperbolic Hierarchical Clustering for Visual Representation Learning

cs.CV cs.AI ·2026-08-24 ·Jianan Wei, Guikun Chen, Zhiyuan Weng, Chunchao Guo, Yujia Wang +1 more
top venue similarity 0.60
Abstract

We investigate the token mixer in vision backbones by revisiting clustering, one of the most classic approaches in machine learning. An effective token mixer is a fundamental component of modern vision backbones like vision Transformers, facilitating information exchange between image patches. Mainstream token mixers, which rely on convolution, attention, MLP, or their hybrids, primarily focus on navigating the trade-off between accuracy and computational cost. However, a significant drawback of these methods is their black-box nature; their encoding process is opaque and lacks interpretability. Diverging from these opaque designs, we introduce ClusterMixer, a transparent token mixer that is grounded in a clustering paradigm and interpretable by design. ClusterMixer explicitly formulates the token mixing process through a hierarchical clustering mechanism. To model the natural, tree-like relationships inherent in visual data, the clustering is performed in hyperbolic space, which is well-suited for embedding hierarchies with low distortion. Building on this innovation, we present HCFormer, a new backbone architecture that integrates ClusterMixer with a series of meticulously designed clustering strategies to ensure robust performance across tasks. Extensive experiments demonstrate that HCFormer consistently outperforms its counterparts across diverse tasks, including image classification, object detection, instance segmentation, and semantic segmentation. Considering its transparency and efficacy, we hope HCFormer can facilitate a paradigm shift toward interpretable backbones.

0.45

More Accurate, Less Human: Gestalt Grouping in Vision Models

cs.CV cs.LG ·2026-08-10 ·Sudhanva Manjunath Athreya, Sai Phani Kumar Malladi
benchmark similarity 0.60
Abstract · Accepted to VISxVision 2026, a workshop at IEEE VIS 2026. Includes appendix with per-task stimuli, metric derivations, and full per-model re…

Human vision organizes what it sees into wholes: same-colored points group into series, similar marks cohere into categories, and shapes complete into recognizable objects. These are the Gestalt operations that visualization design builds on. Whether vision models organize visual content this way has not been systematically tested. We introduce a behavioral battery that scores models against human data from prior perception studies on four grouping tasks: mark-color odd-one-out, color-series counting, silhouette recognition, and object odd-one-out. We apply it to 45 models across five training families: supervised, self-supervised, and contrastive vision-language encoders, open-weight VLMs, and closed foundation models. The battery reveals that agreement with human responses captures aspects of perceptual organization that conventional performance metrics fail to distinguish, with several closed models exhibiting substantially lower alignment than their benchmark accuracy would suggest. Scoring against published perception data therefore gives visualization research a reusable yardstick, requiring no new user study, for auditing whether the models now entering visualization pipelines organize what they see the way their human audience does.

0.13

CLIVO: Differentiable Cluster-Based Instruction Mixing with Learned Validation Objectives

cs.AI ·2026-10-12 ·Yuhan Wang, Rui Hu, Batian Di, Tan Jiang, Dongsheng Ma +3 more
similarity 0.59
Abstract

Data selection turns large multimodal instruction pools into compact training subsets tailored to a vision-language model's target capabilities. Effective selection requires both useful examples and an appropriate balance of supervision. Sample-level rankings can overrepresent similar content without optimizing this balance, while group-level methods are constrained by coarse allocation units, predefined validation sets, and costly or inaccurate mixture search. We propose CLIVO, a target-aware framework that connects fine-grained semantic allocation, automatically learned validation objectives, and differentiable mixture search. Specifically, semantic clusters define independently adjustable training shares, and merged cluster experts provide reusable proxies for evaluating mixtures. We select validation examples by correlating their proxy losses with real probe performance, then learn nonnegative weights to form a target-specific objective. With experts and validation weights fixed, a single backward pass supplies gradients for all shares, avoiding an additional mixture-response regression model. Selecting 100K instructions from an 8M pool, CLIVO attains 97.2% of the full-data four-target average score using only 1.25% of the data. At this budget, it outperforms the strongest evaluated selection baseline by 1.16 points with random within-cluster sampling; combining learned quotas with sample ranking increases this advantage to 1.55 points. CLIVO further supports interpretable, flexible, and reusable data curation through semantic visualization, modular selectors, and shared experts.

0.20