CS Paper Radar

Toward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach

Xunkai Li, Chenxi Wan, Yinlin Zhu, Wang Luo, Hongchao Qin, Rong-Hua Li, Guoren Wang

cs.LG ·submitted 2026-10-02 06:18 UTC · v1

Abstract

Multimodal graph foundation models (MGFMs) seek to learn generalizable representations from large-scale graphs with heterogeneous node modalities. However, real-world Multimodal-Attributed Graphs (MAGs) often contain incomplete node attributes, limiting the scale and diversity of available pretraining corpora. Besides, existing MGFMs primarily incorporate graph topology as structural context, overlooking its role in guiding multimodal binding and shaping a unified representation space. To address these challenges, we propose GraphBind, a topology-driven approach that uses graph topology to bind rich modality information into a unified shared space. GraphBind is motivated by the stability of graph topology, which provides structural references and complementary semantic information for multimodal binding. Concretely, GraphBind uses topology to organize self semantics and reliable neighborhood semantics into a global shared space that integrates structure and semantics, and adapts this space to discriminative and generative tasks through lightweight interfaces. Extensive experiments against 11 representative baselines demonstrate that GraphBind achieves leading performance on both discriminative and generative tasks, with relative improvements of up to 28.1% over the strongest baseline.

Closest papers

Nearest neighbours by meaning across the whole index.

TEAM: Topology-Aware, Semantic-Sensitive, General Transformer for Multimodal-Attributed Graphs

cs.LG ·2026-11-08 ·Yushuo Li, Xunkai Li, Zhengyu Wu, Ronghua Li, Guoren Wang
similarity 0.79
Abstract · 10 pages, 6 figures, 5 tables

Multimodal-Attributed Graphs (MAGs) organize heterogeneous data in graphs whose nodes are characterized by modalities such as text and images rather than traditional unimodal attributes. However, many representative MAG methods organize propagation around node-level states and topology-defined neighborhoods, which can obscure fine-grained modality contributions: a bounded neighborhood may contain semantically weak evidence, while compatible context need not be confined to it. To address both limitations, we propose TEAM, a modality-centric framework that constructs and encodes graph context with node-modality tokens rather than treating each node as a single interaction unit. Concretely, TEAM unfolds the local topology into a Modality Encoding Tree and performs tree-anchored semantic evidence acquisition, organizing local structural context while expanding the accessible evidence space outside the constructed local Tree. We further introduce the Modality-Source Context Transformer (MSCT), which encodes the selected node-modality tokens and coordinates interactions among modality-source groups. By retaining textual and visual attributes as separate interaction units until readout, it supports fine-grained cross-node cross-modal exchange without first collapsing them into whole-node features. Across the evaluated baselines, TEAM attains the highest reported mean on all graph-learning metrics, all three G2Text metrics, and both G2Image metrics. On Toys, it surpasses the strongest baseline by 3.37 and 2.93 percentage points in Accuracy and Macro-F1, respectively.

0.21

Towards Unified Multimodal Graph Foundation Model: A Bridge-Router-Adapter Based Approach

cs.LG ·2026-09-06 ·Sirui Zhang, Yubing Zhou, Xunkai Li, Zekai Chen, Shumeng Li +4 more
similarity 0.77
Abstract

Multimodal graphs couple node attributes in different modalities, such as text and images, with relational structure, enabling topological structure and cross-modality attributes to be modeled jointly. Multimodal graph foundation models seek unified representations from such data that transfer across different graph domains and downstream tasks. However, existing methods exhibit two fundamental limitations. (1) Cross-Scope Context Entanglement. They merge scope-specific graph contexts into a unified representation, obscuring their distinctions during multimodal construction. (2) Scope-Ignorant Modality Routing. They route modalities within a fixed graph scope, overlooking how modality relevance varies across neighborhood ranges. To address these challenges, we propose BRAIN, a unified model that focuses on graph context that combines neighborhood scope with modality composition. BRAIN comprises a scope-conditioned Bridge that combines structural information spanning local-to-global neighborhood scopes with different modality compositions; a hierarchical Router that estimates the relevance between the scope and the task, and selects compositions separately within each scope, allowing modality utility to vary with graph range; and a lightweight residual Adapter that further specializes the routed embedding for downstream prediction. BRAIN is trained through multi-graph pretraining followed by task-specific adaptation. Experiments across nine datasets and four task families demonstrate its broad effectiveness, improving node-classification and link-prediction performance by up to 4.73% relative to the strongest baseline, while achieving an average relative improvement of 14.72% across four graph-to-text and two graph-to-image metrics.

0.21

ReCoG: Reciprocal Co-Evolution for Multimodal Graph Learning

cs.LG ·2026-08-24 ·Rui Xue, Tianfu Wu
hot direction similarity 0.75
Abstract

Multimodal graph learning requires jointly training over graph structure and heterogeneous node attributes, yet existing methods largely decouple these processes: prior multimodal graph neural networks (GNNs) focus on aligning modalities in a shared embedding space while operating on fixed or weakly adapted graph structures, and graph structure learning approaches infer topology from unimodal node representations without accounting for multimodal interactions. This separation fundamentally limits the ability of GNNs to capture semantically meaningful relationships in multimodal settings, where observed edges are often noisy, incomplete, or misaligned with underlying semantics. We propose ReCoG (Reciprocal Co-Evolution for Multimodal Graph Learning), a new learning paradigm that tightly couples graph structure learning and multimodal representation learning through end-to-end reciprocal interaction. Concretely, ReCoG integrates (i) a multimodal graph refiner that infers and corrects edges using cross-modal semantic evidence, and (ii) a coupled cross-modal message passing mechanism that performs joint intra- and inter-modality propagation over the refined graph. This unified design yields greater expressiveness than decoupled or two-stage formulations and allows dynamic interaction between topology and representation learning. Across diverse benchmarks for node classification and link prediction, ReCoG consistently outperforms strong multimodal graph structure learning baselines, including graph foundation models. Our results demonstrate that reciprocal co-evolution of structure and semantics is important for effective multimodal graph learning, challenging the prevailing separation between topology and representation learning.

0.45

Multimodal Graph Learning: A Survey of Models, Benchmarks, and Frontiers

cs.LG ·2026-11-08 ·Rui Xue, Tianfu Wu
survey similarity 0.74
Abstract

Multimodal attributed graphs (MAGs), in which nodes carry text, images, and other modalities while edges encode relations among entities, underlie e-commerce, social media, content platforms, and creative applications. Learning on MAGs has developed along two complementary lines: graph-centric methods redesign how modalities are fused, propagated, and connected within message passing, while foundation-model-centric methods pretrain transferable multimodal graph encoders or bring graph context into multimodal large language models for prediction and generation. In parallel, several benchmarks have appeared with different datasets and evaluation protocols. This survey takes an algorithm-centric and evaluation-grounded view of the field and makes three contributions. Modeling: we introduce a unified message-passing formulation and organize methods into five integration levels (feature, propagation, topology, learning, and foundation model), covering discriminative and graph-conditioned generative models; within this formulation we derive representative methods as special cases, compare their assumptions, expressiveness, and cost, and identify cross-modal sensitivity as a property that separates otherwise similar architectures. Datasets and evaluation: we review recent benchmarks, summarize their datasets and metrics, define model-independent descriptors (label homophily, average degree, clustering, and edge provenance), and map each evaluation protocol to the baselines it uses. Frontiers: we connect multimodal graph learning to AI agents, covering the multimodal graphs that agents perceive and remember and graph engineering, the practice of organizing agent workflows as explicit graphs; we then discuss open problems, concrete efforts for the community, and research directions, including when graph structure helps in the presence of strong multimodal foundation models.

0.19

MAGE: Meta-path Adaptive Graph Evidence for Transferable Multimodal Graph Learning

cs.IR ·2026-11-16 ·Haiyang Jiang, Tong Chen, Xinyi Gao, Hao Wang, Hongzhi Yin
similarity 0.71
Abstract

Multimodal graphs integrate textual and visual attributes with relational structures, providing a rich representation of complex real-world data. To enable the broad applicability and reuse of graph models, it is crucial for models trained on source graphs to generalize to unseen graph domains. However, this is challenging because cross-modal interactions among neighbors and the neighboring information needed for prediction can vary across target nodes and graph domains. This gives rise to two key challenges: adaptive cross-modal neighborhood fusion, which captures and integrates cross-modal interactions among neighbors to accommodate variations across nodes and domains; and target-specific evidence selection, which identifies the neighboring textual and visual information needed for each target node to support LLM-based prediction. In light of these challenges, we propose MAGE (Meta-path Adaptive Graph Evidence Prompting), a framework for transferable multimodal graph learning. MAGE represents textual and visual node views separately in a heterogeneous graph and introduces two complementary components. Meta-path Adaptive Multimodal Propagation (MAMP) adaptively aggregates neighborhood information along heterogeneous meta-paths and compresses it into graph-aware soft tokens. Evidence-Guided Instruction Tuning (EGIT) selects relevant neighboring evidence for each query and incorporates its graph-aware representations into instructions for a frozen multimodal LLM. Together, these components support transfer from multiple source graphs to unseen target graphs without target-domain parameter updates. Experiments with three source graphs and four unseen target graphs demonstrate strong cross-domain transfer performance on node classification and link prediction, without requiring parameter updates on the target domains.

0.04

No-Free-Graph: Learning When Multimodal Data Should Be Graphified

cs.LG ·2026-10-02 ·Zekai Chen, Kai Hu, YuXin Zeng, Xunkai Li, Xun Wu +4 more
similarity 0.71
Abstract

Multimodal graph learning has recently emerged as an effective paradigm for in corporating inter-entity relationships into multimodal representations. Existing studies have made substantial progress on how to construct and optimize graphs, but rarely consider a more fundamental question: whether additional relational structures should be introduced for a given dataset and task. Through empirical studies across diverse datasets, tasks, and graph constructors, we reveal that graphification is not consistently beneficial: introducing relational structures can provide substantial improvements in some cases, while offering limited or even negative gains. This observation motivates a new perspective that graph construction should be treated as a selective decision based on its expected utility rather than a default preprocessing step. To address this issue, we propose MAG-SCOUT, a pre-construction graph assessment framework that estimates whether introducing graph structures is beneficial before generating the complete topology. MAG-SCOUT collects limited relational evidence, analyzes its potential taskspecific contribution, and estimates the expected utility of graphification together with construction cost to make a build-or-skip decision. Extensive experiments across six multimodal datasets, three downstream tasks, and diverse graph constructors demonstrate that MAG-SCOUT effectively identifies when graph structures should be introduced, saving 33.1% of task-macro graph work while retaining 96.7% of held-out positive-gain mass under the pre-registered floor.

0.16

Coupling Perception and Reasoning in Federated Multimodal Graph Foundation Models

cs.LG ·2026-09-24 ·Zekai Chen, Xun Wu, Hailin Zhang, Xunkai Li, Yu Liu +5 more
similarity 0.71
Abstract

Federated multimodal graph foundation models (GFMs) aim to adapt pretrained multimodal models to decentralized graph data, where each client owns a private multimodal graph and cannot share raw information. These models typically combine a multimodal Encoder that extracts semantic evidence from heterogeneous modalities and a graph neural network (GNN) that performs relational reasoning over graph structures. However, existing federated GFM adaptation methods mainly update graph-side modules while keeping the multimodal Encoder frozen, limiting adaptation to \emph{how information is propagated} while fixing \emph{what information is extracted}. Through empirical studies, we reveal that Encoder and GNN adaptations are not independent: Encoder adaptation is affected by graph relations, while cross-client module swapping reveals substantial pairing sensitivity between separately parameterized Encoder and GNN updates. Motivated by this observation, we propose \textbf{FedCORE}, a federated adaptation framework that represents Encoder and GNN updates through a shared low-dimensional latent state. FedCORE jointly optimizes this core from multimodal and structural signals and performs federated evolution directly in the shared state space, preserving compatibility between perception and reasoning adaptations. Extensive experiments demonstrate that FedCORE reduces the Encoder--GNN pairing gap from $30.6$ to $5.9$, corresponding to an $80.7\%$ reduction over independent joint adaptation.

0.09

TopoAgent: A Structure-Aware Perception-to-Reasoning Framework for Diagram-to-Graph Topology Extraction with Large Vision-Language Models

cs.CV ·2026-08-27 ·Bangwei Guo, Xujiang Zhao, Yanchi Liu, Wei Cheng, Shengyu Chen +5 more
benchmarkcode/data similarity 0.69
Abstract

Diagram-to-graph topology extraction aims to extract a graph of entities and their connections from a structural diagram. This task remains challenging for current vision-language models because it requires both fine-grained perceptual grounding and topology-aware reasoning with global consistency. We present TopoBench-180, a human-verified benchmark for diagram-to-graph topology extraction, and TopoAgent, a structure-aware perception-to-reasoning framework for reliable topology extraction using large vision-language models. TopoBench-180 contains 180 structural diagrams spanning Web-style and Network-style categories, paired with canonical graph annotations. TopoAgent progressively extracts the target graph by combining grounded perception, global structural priors, canonical node inventory construction, node-centric local-to-global relation reasoning, and topological consistency enforcement. Experiments on TopoBench-180 show that TopoAgent outperforms strong vision-language model baselines and recent visual reasoning frameworks, especially on edge extraction. More broadly, this work fills an important gap in multimodal structured understanding by establishing a benchmark and framework for diagram-to-graph topology extraction. The benchmark and associated resources will be publicly released at https://huggingface.co/datasets/WayneGuo0011/TopoBench-180.

0.53