CS Paper Radar

Cross-Dimension and Cross-Modality Relational Tensor Representation Learning for Multimodal Health Data Modeling

Jiawei Guo, Meng Zhao

cs.LG ·submitted 2026-10-17 03:31 UTC · v1

Abstract

Multimodal health data integrate information from heterogeneous clinical sources, creating substantial opportunities for data-driven decision support. Realizing these opportunities, however, requires effectively modeling complex dependencies that arise both within individual modalities and across modalities. This paper proposes Relational Tensor Representation (RTR), a unified framework that jointly learns dimension-specific tensor representations and cross-dimensional and cross-modality relationships. RTR introduces learnable relational bridge matrices that maintain relational information while preserving the structural identities of tensor dimensions and modalities. A structured fusion mechanism further integrates factor-level and relationship-level information into compact representations for downstream prediction. Experiments on simulated data and two multimodal neurophysiological datasets, covering emotion recognition and brain activity classification, demonstrate that RTR consistently outperforms representative tensor learning, multimodal fusion, and Transformer-based baselines. These results highlight the value of explicitly modeling cross-dimensional and cross-modality relationships for learning informative representations from heterogeneous multimodal tensor data. The source code is planned for public release to support reproducibility and future research.

Comment
30 pages

Closest papers

Nearest neighbours by meaning across the whole index.

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

cs.AI cs.LG ·2026-10-08 ·Qun Dai, Liangjian Wen, Jiang Duan, Yong Dai, Dongkai Wang +5 more
top venuecode/data similarity 0.71
Abstract · Accepted at NeurIPS 2026

Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL.

0.69

Tensor-based Fusion of Multi-modal Data for Predicting Cancer Risk Levels

cs.LG q-bio.QM ·2026-10-09 ·Ali Anaissi, Yuanzhe Jia, Qishun Peng, Weidong Huang, Ali Braytee +1 more
similarity 0.70
Abstract

Cancer diagnosis and treatment increasingly rely on multi-modal diagnostic technologies that generate high-dimensional datasets. Although these datasets are rich in pathological information, they pose considerable challenges in interpretation, dimensionality reduction, and clinical applicability. Conventional machine learning approaches often fail to extract meaningful cross-modal features, thereby limiting predictive accuracy in patient risk stratification. To address these limitations, this paper proposes an unsupervised machine learning framework that integrates feature extraction, tensor decomposition, and clustering to improve cancer risk prediction. The framework employs tensor decomposition to fuse heterogeneous genomic and clinical data, enabling the extraction of latent, biologically meaningful features that outperform unimodal feature sets. Multiple feature extraction strategies are investigated, and the resulting features are subsequently clustered using a range of clustering algorithms. Comparative evaluations based on internal and external metrics are then used to identify the most effective configurations. The framework is validated on LinkedOmics data comprising 11,158 patients across 32 cancer types. For patient risk stratification, clinically relevant attributes such as tumor purity, pathological stage, and survival outcomes serve as ground-truth indicators. Experimental results demonstrate the superiority of tSVD+GMM, which consistently outperforms other configurations across clinical benchmarks.

0.34

DIGOM: Disentangled Interaction Graphs for Outcome prediction in Multimodal healthcare

cs.LG cs.AI ·2026-10-23 ·Hakima Laribi, Olivier Lefebvre, Dan Poenaru, Martin Vallières
similarity 0.68
Abstract

Multimodal healthcare data contain relational information that extends beyond individual patient observations. Effective clinical outcome prediction therefore requires not only integrating information from multiple modalities, but also modeling the relationships between them. Standard fusion approaches do not explicitly capture these interactions, while existing graph-based approaches typically model only a subset of them. We propose DIGOM (Disentangled Interaction Graphs for Outcome prediction in Multimodal healthcare), a graph-based framework that jointly models three complementary interaction types: Intra-patient cross-modal, Cross-patient same-modal, and Cross-patient cross-modal interactions. DIGOM learns a sparse multiplex graph with disentangled views, each encoding interactions between pairs of modalities across patients. Discrete adjacency matrices with patient-adaptive connectivity are sampled via a differentiable mechanism, enabling end-to-end learning and allowing the model to select informative relationships without imposing a fixed number of neighbors. A regularization objective maintains overall graph sparsity, while message passing integrates this relational context into patient representations for outcome prediction. We evaluate DIGOM on two multimodal healthcare datasets, MIMIC-IV and NIH-TB, across five clinical prediction tasks. DIGOM achieves the highest AUROC on all five tasks while using fewer connections than existing multi-interaction approaches. Ablation studies reveal that each interaction type's contribution varies across tasks, highlighting the need to adapt relational structures per task. DIGOM also demonstrates greater robustness to missing modalities, suggesting learned relationships provide complementary information when modalities are unavailable. Post-hoc analyses further reveal task- and outcome-specific connectivity patterns in the learned graphs.

0.19

Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning

cs.IT cs.AIcs.CVcs.LG ·2026-09-13 ·Liangjian Wen, Linjie Li, Jiang Duan, Yong Dai, Jianzhuang Liu +1 more
survey similarity 0.65
Abstract

Recent advances in multimodal foundation models have intensified the need to understand how different modalities share, preserve, and complement information. Mutual Information (MI), the Information Bottleneck (IB), and Partial Information Decomposition (PID) provide complementary perspectives, yet existing studies often treat them as isolated tools. This survey presents an information-theoretic perspective connecting these principles as progressively refined views of multimodal information processing: MI characterizes inter-modal dependency, IB explains task-oriented information preservation under compression, and PID decomposes preserved information into redundancy, uniqueness, and synergy. We review 170 recent studies (2018--2026) and 12 foundational works, organizing multimodal learning around four challenges: cross-modal alignment, information-efficient fusion, interaction-type characterization, and scaling to multimodal foundation models. Rather than using application domains as primary taxonomy axes, we interpret healthcare, robotics, recommendation systems, affective computing, and wireless communications as empirical validations of these information principles. Beyond taxonomy, we organize existing multimodal paradigms within a single information-theoretic coordinate system -- the Generalized Multimodal Information Lagrangian -- in which they occupy exact or approximate parameter corners, and whose unoccupied regions name candidate method families the literature has not yet built. We further discuss how emerging multimodal foundation models instantiate these principles at scale and identify open challenges including scalable information estimation in high-dimensional settings, standardized evaluation across information-theoretic methods, combinatorial complexity of multimodal PID, and the transition from post-hoc information analysis toward information-aware multimodal learning.

0.33

SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosis

cs.CV cs.LG ·2026-09-04 ·Yuqing Yang, Alexander Schmatz, Zhaozhao Ma, Changkyu Choi, Robert Jenssen +1 more
similarity 0.65
Abstract · 30 pages, 10 figures

Explainability is increasingly seen as a crucial requirement in AI-based medical diagnosis, particularly in safety-critical clinical decision-making. Most existing explainability methods in healthcare operate in a post-hoc manner and are predominantly designed for unimodal data, which limits their applicability in increasingly prevalent multimodal diagnostic settings. This paper addresses the problem of self-explainable multimodal diagnosis by formulating it within the information bottleneck (IB) framework. We propose a unified learning paradigm that jointly optimizes predictive performance and modality-specific explainability by identifying the most informative elements inside each modality that contribute to diagnostic decisions. To enable tractable and stable optimization, we employ a matrix-based Renyi's $α$-order entropy functional under the assumption of sufficiently expressive encoders. Extensive experiments on representative medical datasets spanning heterogeneous modalities demonstrate that the proposed method consistently achieves strong diagnostic performance, including an absolute accuracy improvement of 9.1 percentage points on the iCTCF dataset. Moreover, the learned explanations provide transparent and modality-aware insights into feature relevance, thereby improving both the explainability and generalization.

0.18

M2G-LLM: Enhancing Clinical Prediction via Multimodal Graph Reasoning and LLM Context Injection

cs.LG ·2026-09-18 ·Inyoung Choi, Sukwon Yun, Jiayi Xin, Jie Peng, Tianlong Chen +1 more
similarity 0.64
Abstract

Integrating diverse data modalities --- such as clinical notes, laboratory results, and medical imaging --- is essential for advancing clinical decision-making. While Large Language Models (LLMs) have shown remarkable performance in processing unstructured clinical text, their limited capacity to incorporate non-text modalities hinders their broader utility in healthcare applications. Here, we introduce M2G-LLM (Multimodal MedGraph-LLM), a novel framework that enhances LLMs with multimodal integration and alignment via Graph Neural Networks (GNNs). Our approach models temporal relationships between patient visits, propagates information across clinically similar patients, and aligns heterogeneous data sources to construct enriched multimodal context vectors. These vectors are injected into the intermediate layers of the LLM, enabling joint reasoning over textual and non-textual modalities. We evaluate M2G-LLM on the MIMIC-IV and MIMIC-CXR datasets, demonstrating improvements in clinical prediction tasks over strong baseline models. Our results highlight the promise of combining the language understanding of LLMs with the relational reasoning capabilities of GNNs for comprehensive, multimodal healthcare analysis.

0.11

CNsEMD: An Expert-Annotated Multi-Field-Strength MRI Dataset and a Hyperspherical Manifold Network for Multimodal Cranial Nerve Parcellation

cs.CV ·2026-09-07 ·Lei Xie, Junxiong Huang, Guoqiang Xie, Jiawei Zhang, Jiahao Huang +7 more
benchmarkdataset similarity 0.64
Abstract

Cranial nerves (CNs) play essential roles in sensory, motor, and autonomic functions. Accurate CN parcellation from multimodal magnetic resonance imaging (MRI) is crucial for neuroanatomical analysis and neurosurgical planning. However, accurate CN parcellation remains extremely challenging because CNs are very small, exhibit low image contrast, and have slender tubular morphologies and complex anatomical trajectories. Moreover, the lack of publicly available, expert-annotated datasets has impeded the development and fair benchmarking of learning-based CN analysis methods. In this work, we introduce CNsEMD, an expert-annotated multimodal dataset for CN parcellation. It comprises data from 202 subjects acquired on 3T, 5T, and 7T MRI scanners. We further propose the projective hyperspherical manifold network (PHM-Net), which learns cross-modal representations by capturing angular relationships in a shared hyperspherical embedding space. Rather than performing multimodal fusion in Euclidean space, the proposed Hyperspherical cross-modal interaction (HCI) module enables bidirectional feature exchange between T1-weighted (T1w) and direction-encoded color (DEC) representations on a unit hypersphere. The Magnitude-preserving projective hyperspherical orientation representation (PHOR) captures the axial nature of DEC orientations while preserving diffusion magnitude. The hyperspherical prototype segmentation head (HPSH) further extends angular similarity to voxel-wise classification using normalized voxel embeddings and learnable class prototypes. Extensive experimental results on the CNsEMD dataset demonstrate the effectiveness of our PHM-Net against state-of-the-art methods. CNsEMD establishes a reproducible benchmark for multimodal CN imaging, while PHM-Net provides a geometry-consistent solution for CN parcellation across diverse MRI acquisitions.

0.16

Design Choices for Multimodal Transformer Learning with Missing and Longitudinal Healthcare Data

cs.LG cs.CV ·2026-10-10 ·Rita Cordeiro Mendes, Maria Manuel Pereira, Carlos Santiago, Catarina Barata
similarity 0.64
Abstract

Multimodal learning offers the potential to improve predictions by integrating complementary information from heterogeneous data sources. However, multimodal datasets are frequently incomplete, raising important questions about how models should handle missing information. Although recent approaches increasingly rely on Transformer-based architectures for multimodal fusion, incomplete multimodal data introduces challenges such as missing modalities and intra-modal missingness, which motivate key architectural design choices, such as tokenization granularity. In addition, many datasets are longitudinal, with observations collected across multiple time points. The sequential nature of the Transformer architectures provides a natural framework for modeling such data, but introduces additional considerations, such as how temporal information should be represented. In this study, we systematically investigate these design choices using three different cancer datasets. We compare modality-level and feature-level tokenization and evaluate attention masking and imputation as strategies for handling missing information, both naturally occurring in the datasets and synthetically induced. We further investigate approaches for incorporating temporal information in our training process and how to model it. Experiments are conducted with unimodal and multimodal configurations, as well as different levels of missingness for binary survival classification and multiple time encoding strategies for discrete-time survival prediction. Our results show that while multimodal outperforms unimodal results, missing data heavily impacts predictions and thus effective design choices such as tokenization granularity, and intra-missing handling is essential for reliable prediction.

0.17