CS Paper Radar

Multi-Marginal Inverse Optimal Transport for Contrastive Learning Via Explicit Anchor-Positive-Negative Coupling

Ngoc-Hai Nguyen, Thuan Nguyen, Prakash Ishwar, Shuchin Aeron

cs.LG stat.ML ·submitted 2026-09-27 16:41 UTC · v3
theory

Abstract

Inverse Optimal Transport (OT) based methods for representation learning learn representations such that the global OT coupling between a pair of data marginals in the representation space, concentrates on the positive pairs. This is in contrast to previous methods that primarily focused on pairwise matching. However, these methods $\textit{DO NOT}$ utilize negative pairs and hence are not truly contrastive in their approach. We show that this leads to issues of dimensional collapse and hence degraded downstream performance. To alleviate this, we develop a novel multi-marginal (MM) inverse OT (IOT) contrastive learning (CL) approach called Neg-MMIOT-CL, which learns representations such that the global multi-marginal OT (MMOT) coupling between a triple of data marginals, with respect to a carefully designed ground-cost between triplets of data points in the representation space, concentrates on the anchor-positive-negative $\textit{triplets}$. For a latent class model, we empirically show that Neg-MMIOT-CL alleviates dimensional collapse. Furthermore, for a specific choice of ground cost for all triplets in representation space, we prove that the optimal representation configuration for Neg-MMIOT-CL exhibits equiangular property for within-class and across-class representations, which translates to Neural-Collapse when the representation dimension is larger than the number of classes minus one -- a result that is $\textit{previously established only}$ for pairwise contrastive learning methods. Finally, we propose Neg-IOT-CL-PushPull, that is a computationally efficient alternative to Neg-MMIOT-CL, alleviating the high cost of computing MMOT plans needed during implementation. We apply these methods on both synthetic and real-world datasets and show significant improvements over existing OT-based contrastive learning methods.

Closest papers

Nearest neighbours by meaning across the whole index.

What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies

cs.CV ·2026-10-10 ·Duo Li, Haojun Pan, Zuhao Yang, Xintong Wang, Chengfu Huo +2 more
similarity 0.63
Abstract · 16 pages, 3 figures

Contrastive learning has become the standard paradigm for multimodal representation learning, aligning semantically matched pairs while separating mismatched ones in a shared embedding space. Although various contrastive training strategies have been proposed across negative selection, batch construction, and gradient modulation, their actual effectiveness under controlled, reproducible conditions remains poorly understood. This motivates a fundamental question: Do sophisticated contrastive strategies consistently bring meaningful downstream gains in multimodal embedding learning? To address this, we systematically evaluate representative strategies under a unified benchmarking framework on MMEB. Surprisingly, direct evaluations reveal that most existing strategies offer marginal gains over a standard aligned baseline, with several even degrading overall performance. To uncover the underlying failure modes, we introduce an MLLM-as-a-Judge diagnostic framework that dissects top-1 retrieval behaviors into labeled positives, false negatives (unlabeled valid matches), and hard/easy negative errors. Our systematic analysis yields three key insights: (1) Gradient distortion: Upweighting gradients from high-similarity negatives consistently hurts downstream generalization; (2) Distribution illusion: Wider separation between positive and negative similarity distributions does not necessarily translate to better retrieval performance; and (3) Synergistic training: While false-negative mitigation or hard-batch construction yields limited gains in isolation, integrating hard-negative mining, false-negative mitigation, and hard-batch construction achieves optimal performance, delivering an average improvement of 6.0% on MMEB with Qwen2-VL-2B. Overall, our findings demystify popular contrastive strategies and provide actionable recipes for training universal multimodal embedding models.

0.31

Positive Pair Geometry Matters: Optimal Transport for Contrastive Learning of Visual Representations

cs.CV cs.CG ·2026-09-21 ·Akshit Nanda, Shahzad Ahmad, Ram Prasad Padhy
benchmark similarity 0.62
Abstract · 12 pages, 5 figures

Contrastive self-supervised learning has achieved strong performance by learning representations from multiple augmented views of the same image. However, most existing methods construct positive pairs using independently sampled stochastic augmentations, which may alter semantic content and ignore the intrinsic geometry of the data distribution. In this work, we propose OTCLR, an optimal transport-aware framework for contrastive learning representations that generates geometry-consistent positive samples. Instead of directly contrasting two randomly augmented views, we construct intermediate views between the original image and its augmented variants through entropic optimal-transport displacement interpolation. These transport-interpolated samples serve as positive views that better preserve image structure while explicitly modeling spatial distributional geometry. To further promote smooth representation learning, we evaluate auxiliary Sinkhorn regularization terms that encourage transport-interpolated views to remain consistent with their endpoint images. The proposed method can be incorporated into standard contrastive learning pipelines without modifying the encoder architecture. Experiments on multiple benchmark datasets show that our approach improves representation quality and transfer learning performance compared with conventional augmentation-based contrastive learning baselines.

0.28

Contrastive latent representation learning with denoising score matching

cs.LG stat.ML ·2026-11-13 ·Christopher Marouani, Grigory Bartosh, Christian A. Naesseth
code/datahot direction similarity 0.62
Abstract · 20 pages, 2 figures, 12 tables. Code: https://github.com/chris-mrn/ScoreCLR_code

Contrastive representation learning methods operate by pulling different views of the same data sample together while pushing them away from views of different samples. A fundamental challenge is the estimation of the repulsive force, which typically requires large batch sizes, can exhibit high variance, and is susceptible to the curse of dimensionality. We introduce ScoreCLR, a contrastive representation learning method that offers a novel way to estimate this repulsion term. We consider the framework of representation learning by maximizing the mutual information between two latent views of a sample. By deriving the gradient of the Shannon entropy with respect to the encoder parameters, we obtain an equivalent objective that involves the score of the latent representation. We then use denoising score matching techniques to estimate this score with a lightweight network that is trained jointly with the encoder. We further propose regularization techniques using different noise levels and a Gaussian prior on the latent representation. ScoreCLR replaces the batch of explicit negative samples in contrastive methods with a learned repulsion whose cost is linear in the batch size rather than quadratic. On CIFAR-10 and ImageNet-100 it does so with minimal loss of linear-probe accuracy compared to SimCLR under a shared recipe, while producing projector outputs of higher effective rank, closer to the Gaussian prior.

0.57

Internal Discrepancy for Contrastive Learning: Generalization Bounds and Downstream Behaviors

cs.LG cs.AI ·2026-10-23 ·Jialiang Wang, Qing Ma, Xiong Zhou, Gangfeng Hu, Haoliang Li +1 more
similarity 0.61
Abstract

Contrastive learning has emerged as a popular paradigm for training representation models. While prior studies pursue more practical and less restrictive assumptions to enhance theoretical understanding of contrastive learning, they essentially describe the interconnections among representations within the data or induced by the function class. To encapsulate these assumptions, we propose the measure of internal discrepancy that quantifies the difference between the mean representation of positive samples and latent cluster centers. We then prove an internal discrepancy-aware generalization bound for downstream supervised tasks following contrastive learning, without requiring additional assumptions. This generalization bound not only serves as a unifying framework connecting various assumptions but also motivates us to further reduce the internal discrepancy, thereby introducing the relaxed k-means loss as an approximation. Moreover, we investigate the downstream behaviors of contrastive learning from the perspective of contrastive learning-incurred supervised losses. In the presence of class imbalance, we shed light on why contrastive learning can facilitate imbalanced learning. Finally, we investigate the optimality of these incurred supervised losses, proving several intriguing properties.

0.20

Approximation in Contrastive Representation Learning

cs.LG ·2026-10-11 ·Yuanfan Li, Zihan Zhang, Yiming Ying, Ding-Xuan Zhou
top venue similarity 0.58
Abstract · 54 pages, accepted by NeurIPS 2026

Contrastive representation learning (CRL) is a fundamental paradigm for learning transferable representations. Recent work identifies its population target as the pointwise mutual information (PMI) $\log\frac{p(x,y)}{p_X(x)p_Y(y)}$ between two modalities $X,Y$, where $p$ is the joint density function and $p_X,p_Y$ are the marginals. However, it remains unclear why standard inner-product scores $f(x)^\top g(y)$ can effectively approximate a generally nonseparable population target PMI. In this paper, we address this question from the aspect of approximation for CRL. By linking PMI to a compact operator, we show that $D$-dimensional inner-product models perform rank-$D$ spectral approximation, with error controlled by the spectral tail. We then study neural network realizations: in general settings, we show that deep neural network scoring functions are universally consistent, and the approximation error rates depend on the smoothness of the PMI function; under structured low-rank data models we obtain improved rates attained by moderate-width encoders. Finally, we establish a fast calibration rate from contrastive excess risk to downstream retrieval error under a density-ratio Tsybakov noise condition.

0.44

Data-Efficient Networks for Multi-Contrast MRI Reconstruction based on a Generalized Content/Style Prior

eess.IV cs.CV ·2026-09-02 ·Chinmay Rao, Efe Ilıcak, Matthias J. P. van Osch, Mariya Doneva, Laurens Beljaards +3 more
similarity 0.58
Abstract

Multi-contrast MR scans contain redundant structural information that can be leveraged during reconstruction and potentially accelerate acquisition times. This idea has inspired end-to-end guided reconstruction models, leveraging one or more contrasts to guide the reconstruction of a different contrast. However, these models require large paired multi-contrast raw datasets for training, limiting their application in low-data regimes. In this work, we propose a modular framework, namely CoSMo-RecNet, for learning guided reconstruction models in the low-data regime. At its core is a reusable multi-contrast representation based on a content/style model, which can be learned from large-scale, publicly accessible, unpaired multi-contrast image datasets, without available k-space data. Using this frozen model as a multi-contrast prior and using a set of reference contrasts, the reconstruction problem reduces to a much simpler refinement problem that can be solved by a lightweight unrolled network and thus learned from small, task-specific reconstruction datasets. We demonstrate the efficacy of CoSMo-RecNet by evaluating it on the low-field 0.3 T M4Raw dataset, showing stable reconstruction quality on decreasing the raw training data budget. CoSMo-RecNet achieved higher reconstruction quality with 5 training subjects or lower compared to a parameter-count-matched MoDL trained on 100 subjects. On a data-limited and severely out-of-distribution ultra-low-field 47 mT Halbach scanner dataset, CoSMo-RecNet was superior to other viable strategies, including classical reconstruction, transfer learning, and zero-shot reconstruction.

0.28

Unlocking the Potential of Contrastive Regularization for Diffusion Transformer Training

cs.CV cs.LG ·2026-11-02 ·Yi Tan, Beier Zhu, Sirui Chen, Yunzhe Lin, Yuan Zhou +5 more
similarity 0.58
Abstract · 29 pages

Representation regularization has emerged as an effective way to improve diffusion transformer training. Yet one of the most canonical regularizers--InfoNCE-style contrastive learning--remains largely underexplored in this setting. We naturally ask a question: can standard contrastive learning serve as an effective regularizer for diffusion transformers? Surprisingly, we find that naive InfoNCE regularization brings only marginal gains. Our analysis reveals that the objective is easily dominated by a shortcut: near-orthogonal noise components in diffusion inputs provide a bypass for enlarging negative-pair distances while weakening positive-pair alignment, especially at high noise levels. To unlock the benefit of contrastive regularization, we propose Clean Representation-ANchored Contrastive (CRAN) regularization. Specifically, CRAN anchors contrastive dispersion with clean-sample representation distances, transferring clean relational semantic structure to noisy diffusion representations to suppress the noise shortcut. Extensive experiments on ImageNet show that CRAN consistently improves diffusion transformer training across model scales and training budgets, outperforming internal feature-alignment and dispersive regularization baselines while matching external DINO-guided REPA. These results re-establish InfoNCE-based contrastive learning as an effective regularizer for diffusion training, once its noise-driven over-dispersion is properly constrained.

0.25

MM-LeJEPA: Correlated Gaussian Regularization for Multimodal Joint-Embedding Predictive Architectures

cs.AI cs.LG ·2026-10-17 ·Haowen Bai, Rui Gong$, Meichen Liu, Ming-Xuan Weng, Bihan Wen +1 more
similarity 0.57
Abstract

LeJEPA derives an isotropic Gaussian embedding target by minimizing, under its task assumptions, a Fisher information criterion that controls the task-averaged squared prediction bias. This enables unimodal self-supervised learning without reconstruction decoders, exponential-moving-average teachers, or contrastive negatives. However, its isotropic task prior does not account for the cross-modal structure of multimodal tasks that read shared content. We introduce MM-LeJEPA by extending LeJEPA's task assumptions to multimodal learning. Specifically, we model multimodal tasks as strength-weighted combinations of components that read independent information from the two modalities and components that read the same content from both. Under explicit gradient-moment, correspondence, and symmetry assumptions, we derive a correlated Gaussian target that minimizes a weighted Fisher information functional under a fixed total variance constraint. An analytic transform adapts Sketched Isotropic Gaussian Regularization (SIGReg) to this target, yielding a two-term objective that combines distributional regularization with local-to-global prediction. The extension adds a single scalar hyperparameter, the target cross-modal correlation, and retains the simplicity of LeJEPA. On public audio-visual classification benchmarks, our results are competitive with representative open-source methods. Code will be released.

0.18