CS Paper Radar

3D Diffusion Models for Medical Imaging

Frank-Peter Schilling, Daniel Barco, Philipp Denzel, Marc Stadelmann

cs.CV ·submitted 2026-10-19 14:22 UTC · v1
publishedsurvey

Abstract

Generative deep learning methods such as Variational Autoencoders, Generative Adversarial Networks or Diffusion Models have achieved remarkable results in image synthesis. In particular, Diffusion Models have demonstrated superior image fidelity and sample diversity compared to other approaches on standard benchmarks, while also offering more flexible training dynamics as evidenced both in natural image domains and in medical applications. In medical imaging including Computed Tomography, Magnetic Resonance Imaging or Positron Emission Tomography, such models have various important use-cases including data augmentation, domain transfer, image reconstruction, inpainting, image augmentation and artifact correction. Further, generated images can be conditioned on patient characteristics, medical reports, segmentation masks or other images. Specific challenges arise due to the 3D volumetric nature of medical images, which translate to substantial demands in terms of memory, compute, and training data. This article provides a comparative overview of 3D Diffusion Models which have been developed for medical images, with a focus on current performance, limitations, and an outlook for future work.

Comment
27 pages, accepted for publication in Informatics in Medicine Unlocked
Journal ref
Informatics in Medicine Unlocked, Vol. 66, p. 101832 (2026)
DOI
10.1016/j.imu.2026.101832

Closest papers

Nearest neighbours by meaning across the whole index.

3D CT-to-PET Translation via Latent Brownian Bridge Diffusion

cs.CV cs.AI ·2026-09-11 ·Sarita Mourya, Francesco Di Feola, Pierangelo Veltri, Paolo Soda
similarity 0.74
Abstract

Computed tomography (CT) and positron emission tomography (PET) provide complementary anatomical and functional information for cancer diagnosis and treatment planning. However, the widespread use of PET is limited by high radiation exposure, elevated costs, and restricted availability. To address these limitations, deep learning-based CT-to-PET translation has emerged as a promising approach for synthesizing PET-like information directly from CT images, although accurately modeling the large cross-modal gap remains challenging. In this work, we propose a 3D CT-to-PET translation framework based on latent Brownian Bridge Diffusion (BBDM). The method consists of two stages. First, a Variational Autoencoder (VAE) is trained on paired CT-PET patches, integrating contrastive learning to improve latent alignment between anatomical and metabolic representations. Second, a BBDM is trained in the latent space to translate CT latent representations into their corresponding PET counterparts. The translated PET latents are then decoded and stitched to reconstruct the final 3D PET volume. We evaluate the proposed approach on two publicly available datasets. Quantitative results based on image fidelity and lesion-level PET-specific metrics demonstrate improved performance compared with competing methods. In particular, the proposed approach improves PET signal fidelity, better preserves clinically relevant uptake patterns, and shows improved performance in preserving small-lesion metabolic activation, paving the way for virtual imaging applications.

0.22

Diffusion-Based Tumor Inpainting for Renal Segmentation under Clinical Data Scarcity

cs.CV ·2026-09-18 ·Ekaterina Sedykh, Salme Ussanov, Dmytro Fedorenko, Dmytro Fishman
similarity 0.73
Abstract

Deep learning segmentation of renal tumors requires large annotated datasets, yet clinical deployments typically offer only a handful of tumor-positive cases from the target site. We propose a diffusion-based inpainting framework that synthesizes anatomically plausible renal tumors within healthy CT scans, requiring no additional annotation, and provide the first systematic comparison of 2D, 2.5D, and full 3D (MAISI) synthesis strategies for this task. Training the diffusion model on public data (KiTS23, KIRC) and evaluating nnU-Net segmentation on a internal cohort across three low-data regimes, we find that 2.5D and 3D augmentation substantially reduce false positives (from $\sim$18--20\% to $\sim$3--6\%) while maintaining Dice, whereas 2D provides no consistent benefit. Crucially, the proposed 2.5D method matches full 3D synthesis on every metric at substantially lower computational cost, indicating that local volumetric consistency alone is sufficient for effective augmentation in data- and resource-scarce clinical settings.

0.12

Unified Multi-plane Autoregressive Diffusion for 3D Multi-contrast MRI Synthesis

cs.CV ·2026-10-07 ·Yejee Shin, Geonhui Son, Jinglu Wang, Minwoo Jung, Yan Lu +1 more
top venue similarity 0.73
Abstract · Accepted to ECCV 2026

Acquiring a complete set of magnetic resonance imaging (MRI) contrasts is time-intensive and uncomfortable for patients, despite the diagnostic value of multi-contrast imaging. This motivates synthesizing missing contrasts from those already acquired, which is an inherently 3D problem requiring anatomical coherence across axial, sagittal, and coronal planes. However, fully 3D generative models are often impracti- cal under computational resources that scale cubically with volume size. We propose a unified Multi-Plane Autoregressive Diffusion (MPAD), a latent diffusion framework that achieves full-volume 3D synthesis using efficient plane-wise 2D operations while preserving volumetric coherence. A 3D autoencoder first compresses MRI scans into an isotropic 3D la- tent representation. A 2D diffusion model is then trained to reconstruct masked latent slices of the target contrast, conditioned on both source- contrast slices and unmasked target-contrast slices. During inference, we introduce plane-wise autoregressive synthesis with inter-plane priors. Slices are generated autoregressively in random order within one plane orientation to maintain intra-plane continuity, then propagated as con- ditioning priors to orthogonal plane orientations to enforce inter-plane consistency. Compared to 3D latent diffusion baselines, MPAD reduces training and inference FLOPs by 7x and 3x, respectively, while also lowering inference time and peak memory consumption. Experiments on multiple datasets demonstrate that MPAD achieves superior perfor- mance, generating high-fidelity 3D volumes and supporting one-to-many translation within a single unified model.

0.33

VascuSR: Iterative Anatomy-Aware Diffusion for CT Super-Resolution

cs.CV cs.AIcs.LG ·2026-10-14 ·Hairong Ni, Jiahe Huang, Minghao Guo, Wojciech Matusik, Rose Yu
similarity 0.73
Abstract

Generative super-resolution (SR) can recover sharp detail in computed tomography (CT) images from severely downsampled observations, but the missing high-frequency content is not uniquely determined by the low-resolution input. Consequently, visually plausible reconstructions may contain anatomically unsupported vascular detail, such as missing, spurious, or incorrectly connected vessels. To overcome this challenge, we introduce VascuSR, an iterative anatomy-aware diffusion framework that combines image reconstruction with explicit 3D vascular guidance. First, to provide richer anatomical guidance, we use an explicit 3D vascular representation that captures structural information beyond individual slices. This representation also simplifies vascular refinement by allowing a lightweight 3D generative refiner to focus on vascular structure rather than complex CT appearance. Second, to guide slice-wise SR, a lightweight anatomy controller injects the refined structure into the diffusion decoder, while iterative feedback updates this guidance from each new SR volume. On re-annotated MSD Task08 and HVM datasets, VascuSR outperforms interpolation and learning-based SR baselines in segmentation accuracy (Dice and clDice) and 3D vascular structure (connectivity and mean centerline distance), while maintaining image quality comparable to its SR backbone. Compared with learning-based baselines, VascuSR reduces class-averaged connected-component counts by over 20% on both datasets and mean centerline distance by over 10% on MSD.

0.23

NV-Generate-Body: 3D Latent Diffusion Model forControllable Anatomy, Tumor, and Demographic Body Mask Synthesis

cs.CV ·2026-10-16 ·Can Zhao, Pengfei Guo, Dong Yang, Yufan He, Yucheng Tang +6 more
code/data similarity 0.72
Abstract · Code and model weights open-sourced

Curating large labeled CT datasets is expensive and constrained by patient privacy. We present NV-Generate-BodyMask, a 3D latent diffusion model that synthesizes whole-body CT segmentation masks with precise, joint control over organ anatomy, tumor burden, and patient demographics. The model operates in the compressed latent space of a frozen mask autoencoder, generating $256^3$ volumetric label maps conditioned on a 19-dimensional vector via Adaptive Group Normalization and cross-attention. A size-adaptive ROI loss with spatial halo weighting and a paired tumor injection pipeline enable accurate tumor size conditioning despite the scarcity of annotated tumor cases. Trained on 8,571 masks across 26 cohorts spanning 132 labels, the model achieves Spearman $ρ= 0.70$--$0.92$ for organ size, 85--100\% tumor recall for lung, pancreatic, colon, and bone lesions, and strong demographic alignment (BMI $ρ= 0.78$, sex-conditioned perfect prostate control).

0.34

Physics-Guided Flow Matching for CT Image Reconstruction

cs.AI cs.CV ·2026-08-28 ·Davide Evangelista
similarity 0.72
Abstract · 16 pages, 6 figures, 2 tables

Deep generative models have recently emerged as powerful priors for solving ill-posed inverse problems in CT, with diffusion-based approaches achieving state-of-the-art reconstruction performance. However, diffusion models typically rely on stochastic sampling procedures, long inference trajectories, and carefully tuned noise schedules, which can limit computational efficiency and numerical stability, especially at high spatial resolutions. In this work, we investigate Flow Matching as an alternative generative prior for CT reconstruction. We train a high-resolution Rectified Flow Matching model on 256x256 chest images from the Mayo Clinic Low-Dose CT dataset. To mitigate overfitting and limited anatomical variability, we employ a two-stage training strategy consisting of an initial phase with strong, anatomically informed data augmentation, followed by a fine-tuning phase with reduced or no augmentation to refine structural fidelity. The resulting model is capable of generating high-quality and anatomically coherent CT-like images, serving as a strong learned prior. We then evaluate multiple reconstruction methods specifically designed for Flow Matching models, including Plug-and-Play Flow, FlowDPS, Flower, and Flow-Priors (ICTM), and compare them against state-of-the-art diffusion-based reconstruction algorithms such as DDRM, DPS, and DiffPIR. Experimental results across several CT inverse problem settings show that Flow Matching-based approaches consistently outperform diffusion-based methods in terms of PSNR, SSIM, and perceptual quality, while requiring fewer sampling steps. Finally, we publicly release the trained Flow Matching model and accompanying code to facilitate reproducibility and future research. Overall, this work demonstrates that Flow Matching provides a stable, efficient, and effective alternative to diffusion models for high-resolution CT image reconstruction.

0.42

Large-Scale Pretraining for Improving Deep Learning-Based Geometric Distortion Correction of Diffusion-Weighted Imaging

cs.CV cs.LG ·2026-09-06 ·Saroj Khanal, Yashawant Kumar Yadav, Kritam Bhattarai, Jeevan Neupane, Shristi Subedi +10 more
similarity 0.72
Abstract

Diffusion-weighted imaging (DWI) is widely used in clinical settings but remains vulnerable to geometric distortion. Conventional correction methods often require additional acquisitions or vendor-specific solutions, limiting their feasibility in high-throughput, resource-constrained settings. This study investigates whether large-scale pretraining strategies can improve deep learning-based distortion correction for single-phase-encoding DWI. We formulate the task as image reconstruction, and compare a non-pretrained baseline against a self-supervised and a generative pretrained model, evaluated using both quantitative image-similarity metrics and qualitative expert assessment. The best-performing model was further tested for transferability on data collected in an LMIC setting with acquisition shift. Pretrained models outperformed the non-pretrained baseline, with cWDM achieving the strongest results across both quantitative and qualitative evaluation. However, application to LMIC data revealed transferability challenges, including contrast alteration and over-reliance on T1-weighted anatomical structure. Registering images to a common standard space improved predictions, suggesting that harmonized preprocessing may enhance cross-domain deployment.

0.22

Beyond Representation Learning: A Systematic Study of Joint-Embedding Predictive Generation for 3D Brain MRI

cs.CV ·2026-08-28 ·Meng Zhou, Wenhao You, Yuxing Chen, Yueying Tian
similarity 0.71
Abstract · Preprint, code will be released after review

Joint-embedding predictive architectures (JEPAs) have primarily been developed for self-supervised representation learning. Denoising JEPA (D-JEPA) recently demonstrated strong generative capabilities on natural images, yet the applicability to 3D medical imaging remains unexplored. Building on the D-JEPA framework, we present Med-D-JEPA, a systematic adaptation and evaluation of joint-embedding predictive generation for 3D brain MRI. Med-D-JEPA operates on continuous latent tokens produced by a 3D KL-regularized adversarial variational autoencoder, and combines masked context prediction, representation-level alignment, per-token diffusion, and iterative next-set-of-token sampling. We evaluate unconditional and class-conditional generation quality on BraTS2019 and OASIS-1 datasets; downstream classification utility; and preliminary whole-tumor segmentation on BraTS2020. Across different generation settings, Med-D-JEPA achieves superior or competitive performance compared to several strong baselines on fidelity and diversity metrics. Compared to training with real samples, Med-D-JEPA-based synthetic pretraining improves classification AUC from 0.63 to 0.85 on BraTS2019 and from 0.78 to 0.87 on OASIS-1. In the segmentation study, pretraining on Med-D-JEPA samples improves Dice from 0.74 to 0.80 and reduces HD95 from 13.40 to 9.56 mm. These findings establish joint-embedding predictive generation as a promising direction for 3D medical image synthesis and encourage further research in this direction.

0.47