CS Paper Radar

Information-Guided Selective Modality-Interest Alignment for Multimodal Recommendation

Wenze Ma, Chenyu Sun, Yanmin Zhu, Qiwen Gu, Xuhao Zhao

cs.IR ·submitted 2026-08-28 05:46 UTC · v1
publishedcode/datahot direction

Abstract

Multimodal recommendation (MMRec) aims to enhance recommendation performance by leveraging rich item content from multiple modalities. However, directly incorporating all modality information does not necessarily lead to better preference modeling, since user interests are often more related to a subset of modality signals, while other signals may be weakly aligned with user preferences or even introduce noise. Although recent MMRec methods improve modality utilization through invariant learning, attention mechanisms, graph refinement, or contrastive learning, their alignment processes are often implicit or heuristic and lack a clear objective for selecting modality signals that better match user interests. In this paper, we propose AMUR, an information-guided selective modality-interest alignment framework for multimodal recommendation. Inspired by an information-theoretic view, AMUR aims to enhance modality information that is more related to user interests while reducing the influence of less aligned signals. Specifically, AMUR first refines modality graph structures towards user behavior, and then selectively aligns shared interest-related semantics across modalities. This enables AMUR to improve modality-interest alignment while preserving useful modality-specific complementary information. Extensive experiments on three real-world datasets demonstrate the effectiveness of AMUR over competitive baselines. The code is available at https://github.com/Wenze1/AMUR.

Comment
Accepted at the 35th ACM International Conference on Information and Knowledge Management (CIKM 2026)

Closest papers

Nearest neighbours by meaning across the whole index.

Multimodal Graph Retrieval-Augmented Sequential Recommendation via Collaborative Filtering Paths

cs.LG ·2026-10-08 ·Jason Marcell Setiadi, Xin Cao, Lina Yao
published similarity 0.74
Abstract · Accepted to AJCAI 2026

Multimodal Large Language Models (MLLMs) have demonstrated strong potential for sequential recommendation through their ability to reason over complex multimodal data. However, existing approaches either rely solely on the target user's own interaction history, neglecting collaborative signals from neighboring users, or incur substantial computational overhead through repeated MLLM inference over long interaction histories. To address these challenges, we propose MGRASRec, a multimodal graph retrieval-augmented framework for sequential recommendation. MGRASRec injects collaborative filtering signals conditioned on the candidate item directly into the MLLM prompt by retrieving structured paths from a user-item interaction graph, extended via multimodal similarity to increase coverage beyond exact co-interaction overlap. This retrieval also surfaces the history items most relevant to the candidate at no additional cost, removing the need for recurrent summarization and keeping inference to a single forward pass per candidate. All components are unified into an augmented prompt for parameter-efficient fine-tuning of an MLLM. Extensive evaluations across three publicly available datasets validate the effectiveness of MGRASRec, achieving the best performance on all metrics with particularly strong gains in ranking quality.

0.25

Take a Step Back: A Dual-Granularity Raw Feature Empowered Intermediate Fusion Graph Convolutional Network for Multimodal Recommendation

cs.IR ·2026-11-15 ·Jinfeng Xu, Zheyu Chen, Shuo Yang, Jinze Li, Wenhao Yuan +6 more
publishedposition similarity 0.74
Abstract · Accepted by WSDM 2027

Although multimodal recommendations have shown the ability to learn user preferences and item properties by leveraging explicit multimodal information, they remain constrained by two critical limitations: low-quality representation learning and modality fusion. For the former, existing solutions primarily focus on multi-granularity representation learning; however, we argue that training such representations is difficult due to the inherent data sparsity problem in recommendation scenarios. For the latter, existing solutions often rely on contrastive learning to align representations of different modalities to facilitate modality fusion, yet alignment-only objectives may over-compress valuable modality-specific information. To this end, we suggest \underline{``Take a Step Back"} and propose a \textbf{D}ual-granularity raw feature empowered \textbf{I}ntermediate fusion \textbf{G}raph \textbf{C}onvolutional \textbf{N}etwork (\textbf{DIGCN}) for multimodal recommendation. DIGCN leverages the multimodal understanding capabilities of Large Language Models (LLMs) to construct dual-granularity raw features directly. Furthermore, DIGCN introduces an intermediate fusion graph convolutional network equipped with dynamic cross-modality weight correction, designed to capture cross-modality signals and achieve effective modality fusion. Comprehensive experiments on four real-world datasets validate that DIGCN consistently outperforms a variety of strong baselines. Additional analyses confirm its superiority under varying levels of data sparsity and cold-start scenarios. Notably, DIGCN demonstrates high extensibility, capable of further benefiting from additional contrastive learning-based alignment tasks.

0.21

LUMIR: Decoupled Multimodal Representation Learning from Heterogeneous Data for Next-POI Recommendation

cs.IR cs.LGcs.MM ·2026-11-10 ·Zehui Wang, Wolfram Höpken, Dietmar Jannach
publishedcode/data similarity 0.72
Abstract · 16 pages. Accepted at the DaQuaMRec Workshop, co-located with ACM RecSys 2026

As multimodal tourism-related information collected from heterogeneous data sources has become increasingly available, learning effective representations has attracted attention for improving recommendation performance. Existing approaches often tightly couple multimodal fusion with the architecture and optimization objective of a specific downstream model, which limits the generality and reusability of the learned representations. To address these limitations, we propose LUMIR, a decoupled multimodal representation learning framework based on contrastive learning. Multimodal information collected from heterogeneous data sources is first semantically unified into a shared textual representation space using large language models (LLMs). An interaction-aware fusion mechanism is then designed to model cross-modal interactions, while sequential signals are incorporated as a structural constraint to guide representation learning. To evaluate the learned representations, experiments are conducted on a multimodal point-of-interest (POI) dataset under the next-POI recommendation setting across multiple sequential recommendation backbones. The results show that LUMIR consistently outperforms existing multimodal fusion baselines in both accuracy-oriented and beyond-accuracy metrics, demonstrating that decoupling representation learning from downstream optimization provides an effective paradigm for learning reusable multimodal representations. The code is publicly available at https://github.com/wzehui/LUMIR.

0.36

Agents as Knowledge Integrator and Utilizer in Multimodal Recommendation

cs.IR ·2026-08-29 ·Jinfeng Xu, Zheyu Chen, Shuo Yang, Jinze Li, Puzhen Wu +7 more
similarity 0.70
Abstract

Online platforms increasingly rely on multimodal recommender systems to rank products, media, and other Web content. Existing methods usually inject visual and textual features into item representations or build homogeneous graphs from modality-level similarity, but the resulting signals can remain misaligned with the recommendation objective. We study this semantic gap from a knowledge-integration perspective: multimodal content should be interpreted together with user behavior before it is used to construct recommendation graphs or adjust rankings. We propose AgentMMRec, an agent-based multimodal recommendation framework with two coordinated roles. The Integrator Agent infers behavior- and multimodal-aware user preferences and item properties from training interactions and item content, then stores them in a reusable knowledge memory. The Utilizer Agent consumes this memory to refine modality-specific item-item graphs, construct behavior-aware homogeneous graphs, and rerank candidate lists under a frozen evaluation-time memory. This design differs from direct LLM feature augmentation and pure LLM reranking because the generated knowledge is first converted into graph structure and model representations before recommendation. Experiments on three Amazon multimodal recommendation datasets show that AgentMMRec consistently improves Recall and NDCG over recent multimodal baselines, remains effective under sparsity and item cold-start settings, and can transfer its constructed knowledge to existing backbones.

0.34

AdaM-Rec: Adaptive Modality Routing for Multimodal Recommendation

cs.IR ·2026-09-29 ·Honghao Fu, Jiacheng Chen, Manxi Lin, Junjun Zheng, Xiangheng Kong +5 more
similarity 0.70
Abstract

While recent multimodal recommender systems have demonstrated the effectiveness of incorporating visual and textual information to improve downstream performance, most existing methods rely on static modality fusion, assuming that the relative importance of textual and visual signals remains stable across recommendation scenarios. This design may not fully account for an important variation across recommendation requests: some queries require fine-grained visual cues, whereas others are better served by textual or functional semantics, in which case indiscriminate modality fusion brings in uninformative cues and impairs recommendation quality. To address this, we propose AdaM-Rec, an LLM-based framework for adaptive modality routing in multimodal recommendation, which enables dynamic calibration of reliance on textual and multimodal evidence for user-specific queries. Built on structured natural-language representations of items and user preferences, it estimates modality reliability using proxy recall tasks. Specifically, it generates pseudo-queries that match the granularity of the actual query while pointing to the user's positively interacted items as verifiable proxy targets, evaluating which modality yields better recall performance in analogous scenarios and optimizing the routing strategy in an agentic manner. It then performs routed recall with optimized strategy, enriches results with collaborative items, and ranks candidates by their relevance to both the query and user preferences. Experiments demonstrate that AdaM-Rec delivers strong performance against state-of-the-art baselines, highlighting the effectiveness and broader potential of adaptive control over modality reliance in multimodal recommendation.

0.15

InstMoE: Adaptive Multimodal Routing with Specialized Experts

cs.CL ·2026-10-04 ·Guimin Hu, Xiang He, Yingjian Li, Zheng Lian, Boyan Xu +1 more
similarity 0.69
Abstract

Multimodal inputs are inherently heterogeneous, not only across modalities but also in the information pathways required for effective prediction. To address this limitation, we propose InstMoE, an adaptive expert routing framework for multimodal learning. InstMoE dynamically routes each input to specialized unimodal and cross-modal experts, allowing the model to adapt its information pathways to the characteristics of the input. However, routing can be misled when modality-specific variations obscure task-relevant semantics. Such irrelevant variations may distort routing decisions, causing inputs to be assigned to inappropriate experts. We therefore introduce a Contrastive Semantic Alignment module, which encourages semantically similar inputs to share task-relevant representations while suppressing irrelevant modality-specific variations. Experiments on multimodal sentiment analysis benchmarks demonstrate that InstMoE achieves state-of-the-art performance on CMU-MOSEI and CH-SIMS v2 while using substantially fewer parameters than competitive baselines. Further analysis shows that different inputs exhibit distinct expert preferences, demonstrating that InstMoE moves beyond fixed fusion toward adaptive multimodal computation.

0.14

Toward Explainability in Multimodal Recommendation

cs.IR ·2026-10-11 ·Chen Liang, Jinghui Li, Xiuze Zhou, Zongyue Wang, Yuanguo Lin +1 more
similarity 0.69
Abstract · 19 pages, 2 figures, 4 tables

Multimodal recommender systems effectively alleviate data sparsity and cold-start problems by integrating heterogeneous information, yet complex cross-modal representations and model decision-making processes also introduce new interpretability challenges. Despite the rapid growth of research on the interpretability of multimodal recommendation in recent years, relevant methods and evaluation criteria remain relatively fragmented, lacking a unified conceptual framework. To address this gap, this paper categorizes existing methods into modality explanation and post-hoc explanation based on the timing and manner of explanation generation, and further constructs a progressive system for each category. Building on this foundation, this paper synthesizes four categories of evaluation metrics from both objective quantitative and subjective perceptual perspectives, and reviews relevant datasets along the dimensions of application domain and data source. Through a comprehensive analysis of existing research, we further propose future research directions. This paper aims to establish a unified analytical perspective across multimodal representation, recommendation decision-making, and user preferences, providing a research foundation for building transparent and trustworthy multimodal recommender systems.

0.25

Deciding When to Rely on Visual Information: Gated Multimodal Fusion in Sequential Recommendation

cs.IR ·2026-08-11 ·Natalija Glisovic, Danica Kragic, Martin Tegner
publishedposition similarity 0.68
Abstract · 8 pages, 6 figures, Accepted at CARS @ RecSys 2026

Multimodal sequential recommender systems commonly fuse visual and collaborative signals uniformly, treating visual features as generically informative regardless of item or user context. We argue that visual utility, defined as the contribution of visual signals to recommendation quality, is a latent contextual variable that depends on both the item and the user's interaction history rather than a fixed item property. To model this variability, we introduce VisGate, a framework that makes adaptive item-level fusion decisions conditioned on item embeddings and the user's current sequence context. Visual representations are learned through a contrastive objective over sequential co-occurrence patterns, preserving complementarity with collaborative embeddings rather than aligning them into a shared space. Beyond achieving competitive recommendation performance, VisGate's learned gate serves as a measurement tool for understanding when and why visual information is beneficial. Our analyses show that visual utility varies across items, increases under interaction sparsity when collaborative signals are weak, and correlates with visual distinctiveness in semantically meaningful ways. Together, these findings highlight the importance of both fine-grained fusion and modality complementarity, while demonstrating that item-level visual utility can be estimated and interpreted through learned gating behaviour.

0.33