CS Paper Radar

Compositional Objectives: Learning Structure in Structure

Pranavchandra Vivekananda, Sumukh Bettadapura, Ajan Subramanian

cs.LG ·submitted 2026-09-26 12:40 UTC · v1

Abstract

Intelligence is defined in many ways. One of these definitions defines intelligence as the pursuit of learnable novelty. However, learnable novelty can be meaningless without the ability to compose the learned structures to take action and achieve goals. Learnable novelty builds on epiplexity, which is a way to measure learnable structure in data through a bounded observer. In this paper, we investigate a closed-form spectral approximation to compute epiplexity. We use a fixed-trace constraint and find that the epiplexity objective prefers a more uniform distribution of spectral mass rather than concentrating it in a small number of directions. However, a representation may spread information across many directions without organizing that information into features useful for a particular task. To address this gap, we propose a compositional objective whose observer measures the relationships between the parts and interactions of an image. We compare it with the original spectral objective given only the masked parts. In our ImageNet training runs, the spectral objective with masked parts produces an almost maximally spread representation while achieving the strongest frozen-feature classification performance on most evaluations, more than doubling the linear-probe accuracy of the whole-image baseline. Across multiple image benchmarks, changing what the observer sees matters more than adding relation and interaction tokens. At the same time, our prediction-oriented compositional objective produces substantially better held-out observer prediction but relatively weaker classification, revealing that spectral diversity, predictability, and downstream utility are distinct properties. These results suggest that the usefulness of spectral spreading depends not only on how much structure is preserved, but on which relationships the observer makes available to the objective.

Closest papers

Nearest neighbours by meaning across the whole index.

BISECT: Bilevel Spectral Decomposition for Low-Rank Continual Learning

cs.LG ·2026-11-02 ·Tom Saliencro, Wen-Hsuan Liu, Rohan Desai, Amara Okonkwo, Maya Lindqvist
similarity 0.62
Abstract

Exemplar-free continual learning on pre-trained Vision Transformers increasingly relies on Low-Rank Adaptation (LoRA), yet prevailing methods govern the low-rank update space through strict subspace isolation. By projecting each incoming task into the approximate null space of historical token activations, isolation-based learners equate any cross-task feature overlap with catastrophic interference. We show that this orthogonality premise induces a structural bottleneck we term \emph{spectral starvation}: when down-projection bases are locked before task optimization, the gradient norm reaching the trainable up-projection is upper-bounded by the energy of current-task features captured by the chosen basis. Because natural visual tasks share low- and mid-level covariance structure, historical null tails contain barely $5.5\%$ of incoming task variance, starving new-class plasticity while blocking positive consolidation along shared directions. We propose BISECT (Bilevel Spectral Decomposition), a continual LoRA framework that partitions a fixed rank-$r$ budget into two orthogonal functional blocks: an invariant carrier spanned by the dominant modes of the joint historical-and-current Gram matrix, and a novelty complement extracted via generalized Rayleigh whitening followed by orthogonal complement projection. After single-task training, BISECT folds both blocks back into the frozen backbone using a closed-form precision gate that scales each rank-1 invariant atom by its current-to-joint Rayleigh ratio while preserving novelty atoms at unit gain. Across eight incremental settings on ImageNet-R, ImageNet-A, CIFAR-100, CUB-200, and DomainNet, BISECT outperforms prompt, adapter, and LoRA baselines with zero extra inference parameters or FLOPs, and remains complementary to sparse analytic expansion.

0.18

$λ$-JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning

cs.LG cs.CV ·2026-09-28 ·Berker Demirel, Clémentine Dominé, Valentino Maiorca, Marco Fumero, Marco Mondelli +1 more
code/data similarity 0.62
Abstract

Joint-embedding self-supervised learning typically combines an invariance objective across augmented views with additional mechanisms to prevent representational collapse. These objectives are often applied after a projection head, while downstream tasks use the backbone representation before the projector. We find that this mismatch does not necessarily prevent dimensional collapse in the backbone, which can retain low effective rank and potentially limit downstream transfer. To address this, we introduce SACReg, a spectral anti-collapse regularizer motivated by an analysis of $λ$-balance, which captures the relative scale of weight matrices across layers. In a two-layer linear network, we show that (i) $λ$-balance prevents collapse, and (ii) our regularizer applied to the backbone induces $λ$-balance. In the nonlinear case, this regularizer leads to anti-collapse as well and, in realistic architectures on ImageNet100, it empirically increases the representations' ranks. We apply SACReg to JEPA and propose $λ$-JEPA, which improves over LeJEPA and VISReg on ImageNet-1k classification and in average linear-probe transfer performance across eight downstream image datasets. On video self-supervised learning, $λ$-JEPA improves over LeVJEPA and V-JEPA 2 on the Something-Something-v2 and Kinetics-400 benchmarks. Code is available at https://github.com/berkerdemirel/lambda-jepa.

0.18

InfMasking: A Unified Variational Framework for Synergistic Multimodal Learning

cs.CV cs.AIcs.LG ·2026-11-05 ·Liangjian Wen, Qun Dai, Jiang Duan, Yong Dai, Jianzhuang Liu +5 more
top venuecode/data similarity 0.61
Abstract · Accepted for publication in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 16 pages; supplementary material provide…

Multimodal models are expected to exploit synergistic information, that is, information available only when multiple modalities are observed together. In practice, however, a fusion module can fit the training objective by relying on a dominant modality or on information that modalities share, and the synergy between modalities is often left unused. We propose InfMasking, a unified multimodal learning framework based on Infinite Masking. Before fusion, InfMasking randomly masks the features of each modality; each sampled mask produces a different partial combination of cross-modal evidence and a fused representation of the corresponding modality interactions. All masked fused representations are required to remain consistent with a reference target: the unmasked fused view when training is unsupervised, and the class proxy of the task label when labels are available. Maintaining this consistency across diverse partial combinations discourages reliance on a dominant modality or on shared information alone, and drives the fusion module to learn cross-modal interaction patterns that remain meaningful under partial observations. Since the objective is an expectation over infinitely many masks, we model the masked fused representations as Gaussian and optimize closed-form variational bounds, whose population-moment gaps are controlled by the masking-induced covariance under this model. On the evaluated benchmarks, InfMasking improves predictive performance and yields higher mean PID estimates of task-label synergy in learned modality-specific representations, alongside higher InterSHAP interaction attribution in the trained predictors. Code is available at https://github.com/brightest66/InfMasking.

0.46

Sharp Regret Bounds and a Task-Covariance Correction for Spectral Representation Learning

cs.LG cs.ITstat.ML ·2026-09-14 ·Dier Tang, Jing Yee Tan, Guangyue Han
similarity 0.61
Abstract · 45 pages, 4 figures, 15 tables. Feedback and comments are welcome!

Spectral contrastive learning keeps the $k$ directions along which two views of the data depend most strongly, while downstream tasks may need other directions. Existing guarantees for spectral features assume that the distribution of tasks has no preferred direction. We show that this isotropy condition is the wrong criterion. With linear probing, the task distribution affects the average probing gain only through the matrix $BΛB^\top$, where $B$ records the dependence between views and $Λ$ is the task covariance: optimal features span its top-$k$ eigenspace, whereas spectral features span that of $BB^\top$. Transfer therefore depends on alignment, and strongly uneven task preferences cost nothing when they favour the spectral directions. Otherwise the regret of spectral features is at most linear in the anisotropy of $Λ$. No representation chosen without task information improves this rate when the leading spectrum is flat, and a spectral gap makes it quadratic. With labelled tasks, the same matrix gives a correction: Fisher discriminant directions for a frozen encoder, or a one-term change to the spectral contrastive loss. We bound the regret of the correction by the error in estimating $BΛB^\top$, and compare this error with the misalignment of spectral features to decide when to correct. The analysis extends to finite samples, continuous variables and noisy views. Correction reduces synthetic held-out regret from $0.858$ to $0.003$ with $2000$ labelled tasks, and the decision rule gives the lowest regret among the tested rules at every bank size. On CIFAR-100, spectral features of frozen encoders keep most of the task signal on unseen classes, and where a bank of other classes would lower accuracy, the rule keeps them in every case.

0.61

A General Concept-based Decomposition for Vision-Language Embeddings

cs.CV ·2026-11-11 ·Simone Alberto Peirone, Ortal Senouf, Francesca Pistilli, Giuseppe Averta, Pascal Frossard
top venuecode/data similarity 0.59
Abstract · Accepted at NeurIPS 2026. Project page at https://sapeirone.github.io/grace/

Vision-Language Models are designed to capture the compositional structure of the world through multi-modal alignment; however, it remains unclear whether their representations truly align with the way a human would naturally describe the same visual input. Prior works attempt to decompose visual embeddings into their concept-level contributions, but their instance-level optimization is inherently local and does not capture the model's global semantic structure, which is essential for compositional generalization across different inputs, tasks, and domains. We introduce GRACE, a method that learns this concept-based structure at the feature space level, producing semantic and general concept decompositions. To achieve this, we propose a novel training objective that preserves the geometric relations of different samples in the concept space, enabling a better understanding of how concepts compose and relate to each other. We validate GRACE on several image and video benchmarks, showing that its concept decomposition is aligned with human perception and achieves faithful decompositions across different tasks and domains. With minimal overhead, GRACE makes existing Vision-Language Models more interpretable, while preserving most of their original performance.

0.38

AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models

cs.LG cs.AIcs.CV ·2026-09-28 ·Konstantinos D. Polyzos, Eleni Oikonomou, Tara Javidi
similarity 0.59
Abstract

Large foundation models have been introduced with the promise of efficient adaptation to downstream tasks. Yet, under limited supervision, MLLMs, an important class of large foundation models, remain challenging to adapt to various downstream tasks. Adaptation typically relies either on MLLM parameter fine-tuning or on training neural-based decoders. Both approaches struggle under limited supervision, while fine-tuning additionally requires access to model parameters, which is often unavailable for closed-source models. We introduce AdaKerNet, a novel learnable task-adaptive neural kernel decoder. AdaKerNet is fully agnostic to the parameters of the underlying MLLM and operates solely on its (frozen) rich representations obtained from the diverse available modalities. AdaKerNet relies on (i) a set of learnable, Lipschitz-controlled multimodal features derived from these MLLM representations; (ii) a reference kernel that provides a soft structural prior on those features; and (iii) a lightweight nonlinear neural predictor that adaptively deforms that structure. Learning the kernel representation and the neural predictor jointly within a unified optimization framework allows AdaKerNet to capture features and geometric relationships relevant to the downstream task. Numerical tests across four MLLMs: BLIP-2, LLaVA-1.5, Qwen2.5-VL, and Gemini Embedding 2, and multimodal inputs spanning text, audio, images, and tabular measurements demonstrate significant and consistent improvements over direct MLP, attention-, autoencoder- and kernel-based decoders, across a range of scarce-label budgets, with average error reduction of up to 41% across baselines. These results establish AdaKerNet as an effective approach for prediction from frozen multimodal representations in the scarce label regime. Additional structural ablations highlight the complementary contributions of AdaKerNet's components.

0.23

MLLM-Routed Heterogeneous Ensembles for Robust Cross-Dataset Image Classification

cs.CV cs.AIcs.CLcs.LG ·2026-08-13 ·Daniel Perkins, John Squires, Janou Milligan, Chandra Raskoti, Linda Ungerboeck
similarity 0.59
Abstract · 15 pages (single column), 4 figures, 7 tables

Modern image classification models excel when trained on single task-specific datasets but often struggle to generalize across domains and difficulty levels. We propose ARMDIL, an Adaptive Router for Multi-Domain Image Classification with LLMs. ARMDIL is an ensemble that uses a multimodal large language model (MLLM) agent to dynamically route each image to the most suitable vision backbone. Our diverse ensemble employs convolutional neural networks (ResNets), self-supervised representation learners (SSL), and vision language models (VLMs), each trained on a unified label space constructed from multiple image datasets with differing distributions and characteristics. Empirical evaluations illuminate the distinct capabilities and vulnerabilities of each architecture across disparate visual domains. Crucially, we show that ARMDIL effectively navigates these tradeoffs, performing competitively with specialized training-based routers. Furthermore, it drastically improves adaptability by allowing new information to be integrated via simple prompt modifications, while enhancing interpretability through natural language reasoning traces. These advances in cross-dataset image classification pave the way for more reliable general-purpose vision systems such as AI assistants and autonomous robots.

0.10

Breaking Visual Conventions: Spectrally Coupled Weight Perturbation for Creative Image Generation

cs.CV ·2026-11-17 ·Fu Feng, Ruoheng Zhou, Yucheng Xie, Xu Yang, Jing Wang +1 more
similarity 0.59
Abstract

Text-to-image (T2I) models often produce high-quality images that follow familiar visual conventions, even in response to open-ended prompts such as ``a creative [CLS]''. Existing methods encourage novelty by exploring semantic conditions or modifying features and guidance during denoising. Pretrained weights offer a complementary source of creative variation, but unstructured perturbations can have little visual effect or degrade image quality. We introduce Coupled Spectral Perturbation (CosPert), a training-free method that targets attention projections in layers identified by a one-time sensitivity analysis. For each projection, CosPert uses a shared random transformation and its transpose to couple updates to the singular-vector factors while perturbing the spectral gains. A common relative budget calibrates the magnitude of weight changes across projections, and a sigmoid schedule concentrates their influence in early denoising and gradually attenuates them during refinement. CosPert provides plug-and-play creativity enhancement at inference time. Once the perturbations are prepared, sampling requires no auxiliary networks or additional model evaluations. On both FLUX.1 and Stable Diffusion 3.5, CosPert enhances visual creativity, producing more varied object structures while preserving category identity. Across 50 object categories, CosPert achieves the highest mean CREward among the evaluated methods while maintaining strong prompt alignment.

0.23