CS Paper Radar

Gaussian Primitive Fusion for Any-Resolution Multimodal Images

Baoshuo Cai, Xinrui Jiang, Bing Cao

cs.CV cs.AI ·submitted 2026-11-18 10:06 UTC · v1

Abstract

Multimodal image fusion integrates complementary observations from different modalities into a unified representation. Existing methods typically estimate modality contributions on a common discrete grid. When source modalities have different native resolutions, constructing such a grid generally involves resampling, which may introduce artifacts and affect subsequent fusion decisions. To address this problem, we propose Gaussian Primitive Fusion (GPF), which reformulates heterogeneous-resolution fusion in a shared continuous Gaussian space. Instead of first converting source images to a common resolution, GPF directly fits their native-resolution observations with shared Gaussian geometry and modality-specific content representations, and learns modality contributions within the resulting continuous representation. Once optimized, the fused representation is rendered at the finest native resolution without resampling the sources. With heterogeneous-resolution inputs, GPF produces clearer target contours and finer textures, with less blurring and blocking artifacts than resample-then-fuse pipelines. Quantitatively, it achieves the best results on multiple infrared-visible benchmarks against state-of-the-art fusion methods equipped with interpolation or super-resolution frontends.

Comment
15 pages, 8 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond Resemblance: Image Fusion Reimagined

cs.CV ·2026-11-19 ·Hao Zhang, Yumin Huang, Linfeng Tang, Junjun Jiang, Jiayi Ma
similarity 0.72
Abstract

Image fusion often lacks ground-truth fused images, forcing existing deep methods to indirectly define fusion objectives by requiring the fused image to simultaneously resemble multiple source images. However, the distinct imaging characteristics of different modalities can impose competing constraints on the fused result, potentially limiting the effective integration of their complementary information. To address this long-standing challenge facing the image fusion community, we propose ApexFuse, a novel generative image fusion paradigm that moves beyond source-resemblance constraints toward learning a general distribution of high-quality scene representations. Specifically, we develop a human-LMM-in-the-loop pipeline that leverages large multimodal generative models and iterative human feedback to translate abstract fusion objectives into reliable visual references, thereby defining the target distribution of ideal fusion outcomes. Through target-distribution-guided adaptation, we transfer the rich generative priors of pretrained FLUX to image fusion, enabling high-quality scene reconstruction conditioned on multimodal observations. Extensive experiments demonstrate that ApexFuse harmoniously integrates salient thermal targets, natural colors, and fine textures, achieving superior visual quality as well as semantic segmentation and object detection performance over previous source-resemblance methods. Moreover, ApexFuse exhibits remarkable cross-task generalization, extending from infrared-visible fusion to multi-exposure fusion, multi-focus fusion, and visible-depth fusion without additional fine-tuning.

0.19

FoundDSR: A Generalizable Foundation Model with Guided 2D Gaussian Splatting for Depth Super-Resolution

cs.CV ·2026-09-26 ·Zhengxue Wang, Zhiqiang Yan, Yuan Wu, Guangwei Gao, Xiang Li +1 more
similarity 0.70
Abstract

We introduce FoundDSR, a generalizable foundation model for robust depth reconstruction across unseen data distributions using RGB-D pairs. FoundDSR begins with a guided 2D Gaussian Splatting strategy to model depth representations with Gaussian primitives. This strategy employs high-resolution RGB as prompts to optimize the Gaussian parameters, thereby encouraging each Gaussian primitive to anisotropically deform along high-frequency structural directions. The resulting Gaussian-upsampled representations are then mapped to high-resolution depth through an effective depth reconstruction branch. Furthermore, to mitigate training instability and bias toward dominant sources caused by distribution gaps in large-scale heterogeneous data, we introduce heterogeneous federated learning that allocates each data source to an independent client for local optimization and global aggregation. This design effectively endows FoundDSR with stable scalability to diverse and large-scale training data. Extensive zero-shot evaluations on synthetic, real-world, arbitrary-scale, and noisy conditions demonstrate that FoundDSR consistently outperforms existing state-of-the-art approaches, confirming its strong robustness and generalization to unknown scenes.

0.10

GenRF: Generative Registration-free Image Fusion

cs.CV ·2026-11-18 ·Linfeng Tang, Zhijun Fan, Yang Chen, Zizhuo Li, Hao Zhang +1 more
similarity 0.69
Abstract

Multi-modal image fusion combines complementary observations to improve scene visibility and perception. However, in real-world acquisition, differences in sensing principles and configurations introduce geometric discrepancies, while degradations further compromise source quality. Existing methods typically rely on explicit registration, yet large parallax complicates correspondence estimation, while mismatched fields of view leave some regions without cross-modal counterparts. Meanwhile, independent restoration underuses cross-modal information, whereas restoration-fusion methods assuming alignment remain vulnerable to misregistration. This study revisits misaligned and degraded image fusion in the era of generative foundation models, in which cross-attention enables position-agnostic complementary information aggregation, while generative priors support degradation recovery and inference in missing regions. Building on this perspective, we propose GenRF, which adapts pretrained FLUX through LoRA to jointly enhance degraded observations, infer missing cross-modal context, and fuse complementary information without explicitly estimating deformation fields. Extensive experiments on real-world infrared and visible captures show that GenRF delivers high-quality fusion directly from unregistered, degraded images without preprocessing, while improving downstream task performance under misalignment. Additional experiments demonstrate that GenRF can generalize to misaligned multi-exposure and multi-focus image fusion without task-specific fine-tuning.

0.26

Spackle: Completing Large View Single Image NVS with Adaptive Gaussians

cs.CV cs.AI ·2026-09-25 ·Xuanzhi Liu, Yuhe Zhou, Xinyi Wu, Zhenyao Wu, Jinghao Chen +2 more
similarity 0.69
Abstract

Single-image novel view synthesis (NVS) enables photorealistic rendering of un- observed viewpoints from a single input. Practical NVS systems require two key capabilities: robust reconstruction of occluded regions and high inference effi- ciency. While hybrid decoupled frameworks combining feedforward 3D Gaussian Splatting (3DGS) and diffusion models show promise for large-view-deviation NVS, they suffer from capacity competition: a fixed number of Gaussians forces resource shifts from visible to newly disoccluded areas, degrading original scene fidelity when the target view deviates significantly from the input. To address this, we propose Spackle, a lightweight residual learning framework that mit- igates capacity competition without sacrificing efficiency. Spackle operates in three stages: predicting base 3DGS attributes from given views, automatically identifying poorly reconstructed regions, and learning a residual 3DGS optimized exclusively for these areas. At inference, we combine the baseline and aug- mented Gaussians for NVS. We conduct comprehensive experiments and show that Spackle achieves state-of-the-art performance on large-view-deviation cases.

0.10

Single Image Super-Resolution via Semantic Prompt and Cross-Modal Training on Content Area

cs.CV ·2026-11-09 ·Xiao Tan
similarity 0.69
Abstract · 21 pages, 7 figures, 5 tables

Enhancing image quality through super-resolution is crucial in image and video processing. While contemporary deep learning models, such as CNNs and Transformers, excel in pixel-level enhancements, they often overlook the rich semantic context within images. To address this, we introduce a novel approach that synergizes pixel intensity with semantic information for superior detail recovery. Our method involves extracting semantic data, including textual descriptors and spatial positioning, and encoding this information. We also present a Cross-Modal Feature Fusion module for generating semantic feature maps via multimodal training, and a Prompt Traction Head to augment the backbone network with semantic insights, thereby refining image details. Our empirical results across various datasets validate the superiority of our approach.

0.28

StereoSplat: Metric-Scale Novel View Synthesis via Stereo-Grounded Gaussian Splatting

cs.GR eess.IV ·2026-09-29 ·Vladimir Yugay, Denis Rozumny, Theo Gevers, Elias Vansteenkiste, Martin R. Oswald +1 more
dataset similarity 0.68
Abstract

We present StereoSplat, a feed-forward 3D Gaussian Splatting architecture designed explicitly for stereo videos to achieve reliable, metric-scale novel-view synthesis. Unlike monocular approaches that neglect the nature of binocular pairs, StereoSplat adopts a native stereo architecture that extracts dense multi-scale latent features and disparity from a foundation stereo backbone and feeds them into a dedicated Gaussian decoder. Our decoder utilizes cross-view feature fusion and gated depth refinement to directly translate these rich representations into per-pixel 3D Gaussians. To resolve the high redundancy of per-pixel predictions across overlapping views, we propose a learnable depth-adaptive aggregation mechanism that clusters and aggregates primitives in inverse-depth space while preserving fine-grained details. To support training and generalization, we introduce XRStereo, a large-scale synthetic dataset of stereo video featuring various rig configurations tailored to match the physical hardware of real-world platforms. Trained exclusively on synthetic data and evaluated across challenging benchmarks, StereoSplat yields substantial improvements in the depth accuracy of rendered novel views over established baselines while maintaining high-fidelity novel view synthesis. The model also transfers zero-shot to real-world stereo video, with applications in mixed reality, autonomous driving, and robotics.

0.16

DimFlow: Bidirectional Grid-Expert Learning for Dynamic Gaussian Reconstruction

cs.CV ·2026-10-26 ·Gabriel Adineera, Jinoh Kim
similarity 0.67
Abstract

Dynamic Gaussian reconstruction requires compact scene features to support coordinated changes in geometry and appearance. A factorized grid provides efficient local queries, but accurate interpolation alone does not ensure that the resulting dynamic attributes remain compatible across space and time. We introduce DimFlow, a framework that learns these features through bidirectional interaction between a multi-resolution grid and a coordinate-based expert branch. Feature-level adversarial alignment connects their distributions, adaptive fusion transfers complementary information, and activation-masked residual feedback refines the grid during training. Four attribute experts organize deformation, rotation, opacity, and temporal information before Gaussian rendering. In the aggregate comparison across D-NeRF, HyperNeRF, and DAVIS, DimFlow achieves 39.73 dB PSNR, improving on 4D-GS by 1.92 dB while matching its reported 85 FPS rendering speed. The model occupies 18.9 MB, compared with 18.3 MB for 4D-GS. Component and attribute-expert ablations reveal complementary contributions from feature alignment, adaptive exchange, and temporal processing.

0.22

InfMasking: A Unified Variational Framework for Synergistic Multimodal Learning

cs.CV cs.AIcs.LG ·2026-11-05 ·Liangjian Wen, Qun Dai, Jiang Duan, Yong Dai, Jianzhuang Liu +5 more
top venuecode/data similarity 0.67
Abstract · Accepted for publication in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 16 pages; supplementary material provide…

Multimodal models are expected to exploit synergistic information, that is, information available only when multiple modalities are observed together. In practice, however, a fusion module can fit the training objective by relying on a dominant modality or on information that modalities share, and the synergy between modalities is often left unused. We propose InfMasking, a unified multimodal learning framework based on Infinite Masking. Before fusion, InfMasking randomly masks the features of each modality; each sampled mask produces a different partial combination of cross-modal evidence and a fused representation of the corresponding modality interactions. All masked fused representations are required to remain consistent with a reference target: the unmasked fused view when training is unsupervised, and the class proxy of the task label when labels are available. Maintaining this consistency across diverse partial combinations discourages reliance on a dominant modality or on shared information alone, and drives the fusion module to learn cross-modal interaction patterns that remain meaningful under partial observations. Since the objective is an expectation over infinitely many masks, we model the masked fused representations as Gaussian and optimize closed-form variational bounds, whose population-moment gaps are controlled by the masking-induced covariance under this model. On the evaluated benchmarks, InfMasking improves predictive performance and yields higher mean PID estimates of task-label synergy in learned modality-specific representations, alongside higher InterSHAP interaction attribution in the trained predictors. Code is available at https://github.com/brightest66/InfMasking.

0.46