CS Paper Radar

RaGCD: Reliability-Aware 2D-3D Multimodal Generalized Category Discovery

Junyong Liu

eess.SP ·submitted 2026-11-10 09:53 UTC · v1

Abstract

Generalized Category Discovery (GCD) aims to recognize known categories while discovering novel ones from partially labeled data. Existing GCD studies primarily focus on unimodal observations or image-text interactions, leaving the complementarity between visual appearance and explicit 3D geometry underexplored. To exploit complementary visual appearance and spatial geometry, we investigate GCD with paired images and point clouds. However, the relative reliability of these modalities varies across samples, so unreliable evidence can compromise both multimodal fusion and cross-modal knowledge transfer. This challenge is particularly pronounced for novel categories, where labeled supervision is unavailable. To address it, we propose RaGCD, a reliability-aware framework for 2D-3D multimodal GCD. Specifically, Reliability-aware Modality Allocation (RMA) combines prediction-derived reliability cues with modality-specific features to adaptively allocate image and point-cloud contributions at the channel level. Reliability-conditioned Alignment (RCA) further reweights feature-level and prediction-level alignment and selects reliable observations for prototype-level alignment, regulating cross-modal interaction according to sample reliability. Extensive experiments on ModelNet40, Washington RGB-D, and ShapeNet-ViPC demonstrate that RaGCD outperforms the compared methods in overall and novel-category clustering accuracy. Ablation studies support the complementary roles of RMA and RCA, while controlled corruption experiments show that modality allocation shifts toward the less corrupted modality. These results support the effectiveness of reliability-aware interaction for category discovery from paired appearance and geometry.

Closest papers

Nearest neighbours by meaning across the whole index.

Refine Then Fusion: Training-Free 3D Point Cloud Adaptation with Priority Refinement and Multi-Modal Knowledge Fusion

cs.CV ·2026-09-18 ·Hang Cheng, Yan Chen, Mingyu Fan, Long Zeng
similarity 0.65
Abstract

Recent pre-trained foundation models provide rich multi-modal priors for downstream 3D vision tasks. However, the effectiveness of these representations in few-shot scenarios is limited by two fundamental challenges: High-dimensional features often contain substantial channel redundancy and task-irrelevant noise, while the reliability of different modalities varies across samples. Consequently, direct aggregation of heterogeneous representations overlooks sample-dependent modality reliability and may obscure the discriminative cues essential. To address these limitations, we propose Refine Then Fusion(RTF), a training-free framework for few-shot 3D recognition. RTF first identifies discriminative feature channels by jointly modeling inter-class similarity and intra-class stability, thereby decoupling domain-specific knowledge refinement from the cached representations of pre-trained models. It then introduces a reliability-aware fusion mechanism that estimates sample-wise modality reliability from the distribution shifts induced by feature refinement, enabling adaptive aggregation of multi-modal representations. Furthermore, RTF constructs a memory cache that integrates instance-level support features with class-level prototypes to infer query labels. Extensive experiments on five benchmarks demonstrate that RTF consistently outperforms single-modal baselines, partial-fusion variants, and existing lightweight adaptation methods, achieving state-of-the-art few-shot 3D recognition performance without gradient optimization, additional training data, auxiliary training, or parameter updates.

0.17

AggCLIP: Aggregating CLIP for Zero-Shot Multimodal Anomaly Detection

cs.CV ·2026-11-18 ·Zhi Li, Hanyu Wang, Fei Shen, Zhengtao Zhang
top venue similarity 0.65
Abstract · 13 pages, 4 figures, 6 tables. Author version of the ACM Multimedia 2026 paper

Zero-Shot Multimodal Anomaly Detection (ZSMAD) aims to identify both appearance and geometric defects in novel categories without relying on target-specific training data, making it especially valuable in real-world scenarios constrained by data scarcity, privacy concerns, or high annotation costs. While recent approaches successfully adapt vision-language models like CLIP to achieve this open-vocabulary generalization, they still face significant challenges: (i) severe geometric information loss during 3D-to-2D point cloud rendering; (ii) spatially noisy 2D feature maps produced by raw CLIP patch embeddings, compounded by shallow late-stage modality fusion; and (iii) severe cross-modal semantic misalignment due to decoupled processing streams. To address these limitations, we propose AggCLIP, a novel framework that adapts frozen CLIP representations for ZSMAD by explicitly incorporating local geometric structure and visual spatial context. Specifically, to accurately localize fine-grained defects, we design a Geometry-Aware Spatial Aggregation (GSA) module that refines back-projected 3D point features via density-adaptive neighborhood weighting, alongside a Context-Aware Spatial Aggregation (CSA) module that smooths multi-scale 2D patch embeddings using a lightweight trainable adapter. Furthermore, to prevent semantic drift between modalities, we introduce a Dual-Stream Prompt Contrastive (DSPC) loss, which aligns the abstract concepts of normality and abnormality across the point cloud and RGB streams. Extensive experiments demonstrate that the proposed AggCLIP framework achieves state-of-the-art performance on the MVTec 3D-AD and Eyecandies benchmarks under both one-vs-rest and cross-dataset zero-shot settings, notably surpassing the strongest competitive methods by up to 2.9% in pixel-level AUPRO.

0.41

Bidirectional Adaptive Transformers for Multimodal Anomaly Detection

cs.CV ·2026-10-10 ·Yuxin Jiang, Yunkang Cao, Weiming Shen
published similarity 0.65
Abstract

The effective integration of features derived from RGB images and 3D point clouds presents a substantial challenge in the field of multimodal anomaly detection (MAD). Existing MAD methods typically rely on static fusion paradigms built upon frozen pre-trained encoders. While these frameworks facilitate spatial alignment, they lack the mechanism to establish latent semantic correspondence, thereby inducing cross-modal interference that degrades anomaly discriminability. To mitigate these limitations, this study proposes a Bidirectional Adaptive Transformers (BAT) framework, which constructs comprehensive and semantically aligned multimodal representations by leveraging complementary attributes across modalities. The BAT framework introduces a Bidirectional Feature Alignment (BFA) module, which employs a mutual predictive mechanism to bridge the gap between 2D textural patterns and 3D geometric structures. This bidirectional interaction explicitly enhances the textural granularity of point cloud descriptors while augmenting the spatial contextual awareness of RGB features. To ensure that the latent representations possess sufficient informational density for alignment, BAT incorporates a parameter-efficient adaptation architecture. By utilizing lightweight image and point cloud adapters, the framework facilitates specialized fine-tuning of pre-trained models, mitigating the information loss inherent in frozen encoders. Extensive experiments conducted on MVTec 3D-AD and Eyecandies confirm that BAT achieves state-of-the-art anomaly detection performance, significantly outperforming existing MAD methods.

0.37

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

cs.AI cs.CV ·2026-10-03 ·Songyuan Sui, Zhen Tan, Mohan Zhang, Rana Muhammad Shahroz Khan, Xia Hu +1 more
publishedtheory similarity 0.64
Abstract · NeurIPS 2026 Main Conference

Missing modality remains a longstanding challenge in multimodal learning. Existing methods typically address this issue through modality recovery or adaptive strategies. However, they overlook models' internal cross-modal dependencies formed during multimodal training, which later impair robustness. We systematically characterize a counterintuitive deployment-time failure mode: models trained on full modalities can underperform unimodal models when one modality is missing at inference time. This pattern appears across diverse architectures, such as fusion models, CLIP-style two-tower models, and vision-language models. We show that such degradation is closely associated with learned cross-modal dependencies in the principal parameter subspaces. Multimodal training induces structured rotations of these subspaces, particularly in cross-modal interaction layers. These rotations are associated with reduced task-aligned margins and larger task-aware representation harm under missing-modality inputs. We propose Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness. It rotates the principal input subspace toward a unimodal reference along a geodesic path. We prove that this correction minimizes the distance to the reference within a fixed subspace-distance budget. Experiments across architectures and datasets show that GU improves performance under missing-modality inference while preserving full-modality accuracy, outperforming strong missing-modality robustness baselines. These findings support a geometric view of deployment-time missing-modality degradation and suggest localized subspace editing as a practical route for robustness correction.

0.28

Calibrating Multimodal Test-Time Adaptation under Reliability and Availability Modality Shifts

cs.CV ·2026-11-02 ·Jiangmeng Li, Jinglin Xu, Yi Li, Yumeng Liu, Jiahuan Zhou +3 more
code/data similarity 0.64
Abstract

Multimodal test-time adaptation (TTA) enables pretrained multimodal models to accommodate distribution shifts during inference using only unlabeled test data, without recourse to the original training data. Compared with unimodal adaptation, multimodal adaptation is further complicated by the coexistence and interaction of multiple representation perspectives. This complexity is particularly pronounced under modality reliability shifts, which preserve modality availability but degrade information quality, and modality availability shifts, which render the corresponding modality entirely inaccessible. Category-conditional Gaussian modeling offers a natural starting point for handling such distributional variations, having proven effective in unimodal test-time adaptation for characterizing representation distributions, yet its extension to multimodal settings requires a more elaborate formulation. To this end, we propose Adaptive Probabilistic Gaussian Calibration (AdaPGC), which extends category-conditional Gaussian modeling across modality-specific and multimodal fusion representations for probabilistic calibration under multimodal distribution shifts. AdaPGC continuously updates category-conditional Gaussian models over modality-specific and fusion representations, using the resulting probabilistic predictions to calibrate source model predictions and guide reliability based rectification, while exploiting their joint distributions to recover unavailable fusion representations. Extensive experiments under diverse modality corruption and missingness conditions demonstrate that AdaPGC effectively improves robustness under both modality reliability and availability shifts. Our source code will be released at https://github.com/XuJinglinn/AdaPGC-plus.

0.39

LEGAU: Learning Semantic Gaussian Priors for Scalable Category-level Pose Estimation

cs.CV ·2026-09-28 ·Hongli Xu, Zhaowei Lu, Junwen Huang, Jiaqi Hu, Peter KT Yu +3 more
similarity 0.64
Abstract

Category-level 6D pose estimation from a single RGB-D observation is inherently under-constrained, since partial visible geometry must be interpreted together with a canonical object structure before a stable pose can be determined. We present LEGAU, a unified framework that jointly predicts NOCS correspondence, object pose and size, and a canonical Semantic Gaussian Field. Rather than treating reconstruction as a detached auxiliary task, LEGAU uses the Gaussian field as a category-conditioned structural prior that participates in multimodal feature fusion and provides global guidance for local pose reasoning. Conditioned on a categorical text embedding, LEGAU processes RGB-D observations through a transformer-based fusion module that integrates visual, geometric, and category-level cues, decoding the NOCS map, pose and size information and the Gaussian-based object representation. Extensive experiments on synthetic and real-world benchmarks show that this coupled pose-shape formulation achieves strong performance in a single-model multi-category setting, with up to 22\% on SOPE and competitive transfer to real-world data. These results highlight the benefit of jointly learning canonical correspondence, object shape, and pose alignment within a unified representation.

0.13

Reliability-Consistent Appearance Modeling for RGB-Infrared Multi-Object Tracking

cs.CV ·2026-10-16 ·Bingyang Wang, Jiazuo Yu, Lu Zhang, Lihe Zhang, Huchuan Lu +1 more
similarity 0.64
Abstract · 10 pages, 6 figures. Submitted to IEEE Transactions on Multimedia

Recent RGB-infrared multi-object tracking (MOT) methods have substantially improved cross-modal representation learning and multimodal information integration. However, most existing methods compress RGB and infrared evidence into a unified descriptor. When one modality is temporarily degraded, the tracker can no longer regulate its contribution separately; unreliable evidence may distort association and contaminate appearance memory. We therefore propose RCA-MOT, a reliability-consistent appearance modeling framework that preserves cross-modal complementarity while retaining post-representation evidence controllability. RGB-specific, infrared-specific, and fused identity representations are maintained as jointly accessible appearance states. For each detection, a simple training-free target-scene estimator produces branch-wise appearance reliability used across the two temporal roles of appearance: relative reliability regulates different identity branches, absolute reliability calibrates borderline candidates near the ReID acceptance boundary, and the same branch-wise reliability controls how strongly matched descriptors update historical appearance memory. The individual components are intentionally simple, focusing evaluation on reliability-controllable appearance organization rather than elaborate fusion or learned reliability-estimation modules. RCA-MOT sets new SOTA results on VT-MOT and UniRTL, improving the previous best HOTA by 5.402/6.075 points and IDF1 by 11.549/12.789 points, respectively, while post-hoc analyses further validate reliability as an informative control signal. Code will be released upon acceptance.

0.23

Attribute-Evidence Learning with Multimodal Prompt for Generalized Category Discovery

cs.CV cs.LG ·2026-11-08 ·Wei Zhang, Zhu Teng, Zhixiang Chi, Junnan Zou, Wenxin Luo +3 more
top venue similarity 0.64
Abstract · IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

Generalized category discovery (GCD) aims to discover novel categories from unlabeled data by leveraging limited labeled samples from old classes. Existing methods that rely solely on visual cues often neglect the multimodal nature of human perception, prompting recent approaches to incorporate textual information such as category tags and attributes. However, these approaches typically entangle global tags and local attributes within a shared global image representation, rendering attributes unreliable and potentially providing misleading cues that can drive the model toward incorrect predictions. To this end, we propose AptM-GCD, a novel multimodal framework that exploits apt textual guidance through attribute evidence learning and multimodal prompt reasoning. Specifically, we design an Attribute-Evidence Learning (AEL) mechanism that treats attributes as localized visual evidence rather than global category descriptors. By interacting attribute embeddings with patch-level visual tokens, AEL identifies reliable attribute-related regions and extracts discriminative local evidence, thereby reducing the impact of noisy or irrelevant attribute cues. Building upon the extracted evidence, we further develop a Multimodal Prompt Learning (MPL) mechanism to bridge local attribute semantics and global visual understanding. MPL converts fine-grained textual evidence into visual prompts and injects them into a multimodal prompt transformer, enabling attribute information to enhance category discrimination through contextual reasoning without enforcing direct global alignment between modalities. Extensive experiments on six datasets demonstrate that AptM-GCD consistently achieves SoTA performance across both fine-grained and generic benchmarks. Moreover, the proposed framework exhibits strong generalization ability, delivering consistent gains under unimodal and class-imbalanced GCD settings.

0.34