CS Paper Radar

Uncertainty-Aware Multimodal Deep Learning for Prediction of Radiation-Induced Xerostomia in Head and Neck Cancer

Jin Yang, Tian Liu, Jose Penagaricano, Jing Wang, Erin L. Moshier, Jerry Liu, Richard Bakst, Ming Chao

eess.IV ·submitted 2026-11-19 04:13 UTC · v1

Abstract

To develop and evaluate an uncertainty-aware multimodal learning framework for predicting radiation-induced xerostomia in patients receiving head-and-neck cancer radiotherapy. We developed a Bidirectional Attention-enhanced Diffusion Transformer (BA-DiT) that combines self-supervised multimodal representation learning with supervised Bayesian fine-tuning. The model integrates pretreatment CT images and three-dimensional dose distributions, focusing on bilateral parotid glands while incorporating surrounding anatomical and dosimetric information. During pretraining, bidirectional cross-attention integrates anatomical and dosimetric features, while a latent Diffusion Transformer learns joint representations through diffusion-based learning, feature reconstruction, and contrastive alignment. Supervised fine-tuning employs a Bayesian classifier to predict individualized xerostomia risk and quantify predictive uncertainty, with a learnable label-transition model accounting for potential toxicity label uncertainty. BA-DiT was developed using four independent cohorts (Moffitt, Emory, MSHS, and TCIA) and evaluated against baseline and ablation models. Evaluation included predictive performance, calibration, uncertainty quantification, decision curve analysis, and patient-level case studies. In independent MSHS and Emory cohorts, BA-DiT achieved AUCs of 0.83 and 0.80, respectively, outperforming baseline models. Predictive uncertainty peaked near the decision boundary, with aleatoric uncertainty being the predominant component. Uncertainty-based selective prediction generally improved net benefit among retained patients, with greater gains at lower prediction coverage. BA-DiT enables individualized xerostomia risk prediction while providing complementary uncertainty estimates, potentially supporting informed clinical decision-making and personalized toxicity management.

Comment
24 pages, 4 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Multimodal Deep Learning for Uncertainty-Aware Radiation Pneumonitis Risk Prediction

eess.IV ·2026-08-28 ·Jin Yang, Tian Liu, Jing Wang, Sooyun Caroline Tavolacci, Benjamin Tran +5 more
similarity 0.79
Abstract · 28 pages, 9 figures, 5 tables

Radiation pneumonitis (RP) is a potentially serious complication of thoracic radiotherapy, making reliable risk prediction important for identifying high-risk patients and guiding effective toxicity management. Although deep learning models can predict RP risk, uncertainty quantification remains underdeveloped in radiation toxicity assessment. We developed a Multimodal Bayesian Diffusion Transformer (MM-DiT) that integrates pretreatment computed tomography (CT) and three-dimensional planning dose distributions for RP risk prediction and uncertainty quantification. MM-DiT combines self-supervised multimodal representation learning with Bayesian classification to estimate individual risk and characterize aleatoric, epistemic, and label uncertainty. In independent Mount Sinai Health System (MSHS) and RTOG cohorts, MM-DiT achieved areas under the receiver operating characteristic curve (AUCs) of $0.89$ and $0.86$, respectively, with expected calibration errors of $0.02$ and $0.03$. Incorporating post-treatment CT increased AUC to $0.95$ in MSHS and reduced aleatoric and epistemic uncertainty by approximately $40-45\%$ and $40\%$, respectively. These findings establish a framework for uncertainty-aware radiation toxicity prediction that complements risk estimates with information about predictive reliability. By integrating multimodal learning and uncertainty quantification, MM-DiT may advance more interpretable and trustworthy AI-assisted toxicity assessment and support individualized radiotherapy planning.

0.19

Learning the Dose Manifold: A Latent Diffusion Framework for Volumetric Dose Prediction in Automated Radiotherapy Treatment Planning

cs.CV ·2026-10-19 ·J. Boyer-Chammard, R. Vauclin, M. C. Biston, P. Maury, M. Gau +5 more
similarity 0.71
Abstract · 27 pages, 7 figures plus graphical abstract

Automated treatment planning (ATP) aims to reduce the unwarranted inter-planner variability of manual planning, while improving treatment quality and efficiency. Most ATP pipelines rely on the prediction of the achievable three-dimensional dose distribution for a given anatomy and the subsequent optimization of machine parameters to reproduce it. Diffusion-based methods are largely slice-wise due to the high dimensionality of radiotherapy data, and assume a one-to-one mapping between anatomy and optimal dose despite the multi-objective nature of treatment planning. They are evaluated on their ability to match reference doses and rarely on their integration in ATP frameworks. We present LaDose, a latent diffusion framework for fully volumetric dose prediction. An autoencoder learned on dose distributions, trained so that perturbed latents decode to plausible doses, compresses dose maps 128-fold and enables 3D diffusion on a single GPU. Prescription values are represented as exact scalar vectors that bypass the image encoder, avoiding resampling and latent compression. Multiscale classifier-free guidance allows for the modulation of clinical trade-offs at inference, without retraining or additional labeled data. On 900 clinically delivered head-and-neck volumetric modulated arc therapy plans from five centers, LaDose reached a median absolute error of 1.18 Gy (1.4--1.8\% of the prescription) and was the only one of six frameworks free of systematic bias with respect to the reference distribution. Guidance scales acted predictably on target coverage and organ-at-risk sparing. Integrated with direct aperture optimization, the pipeline generated plans in 15.2 minutes; in a blinded review, senior clinicians rated 14 of 15 plans acceptable without editing.

0.12

3D CT-to-PET Translation via Latent Brownian Bridge Diffusion

cs.CV cs.AI ·2026-09-11 ·Sarita Mourya, Francesco Di Feola, Pierangelo Veltri, Paolo Soda
similarity 0.65
Abstract

Computed tomography (CT) and positron emission tomography (PET) provide complementary anatomical and functional information for cancer diagnosis and treatment planning. However, the widespread use of PET is limited by high radiation exposure, elevated costs, and restricted availability. To address these limitations, deep learning-based CT-to-PET translation has emerged as a promising approach for synthesizing PET-like information directly from CT images, although accurately modeling the large cross-modal gap remains challenging. In this work, we propose a 3D CT-to-PET translation framework based on latent Brownian Bridge Diffusion (BBDM). The method consists of two stages. First, a Variational Autoencoder (VAE) is trained on paired CT-PET patches, integrating contrastive learning to improve latent alignment between anatomical and metabolic representations. Second, a BBDM is trained in the latent space to translate CT latent representations into their corresponding PET counterparts. The translated PET latents are then decoded and stitched to reconstruct the final 3D PET volume. We evaluate the proposed approach on two publicly available datasets. Quantitative results based on image fidelity and lesion-level PET-specific metrics demonstrate improved performance compared with competing methods. In particular, the proposed approach improves PET signal fidelity, better preserves clinically relevant uptake patterns, and shows improved performance in preserving small-lesion metabolic activation, paving the way for virtual imaging applications.

0.22

Impact of Pretraining Length, Model Size, and Annotation Budget on Downstream Clinical Tasks of Foundation Models in Radiotherapy

cs.CV ·2026-11-17 ·Alejandro Cortina Uribe, Sidsel Højklint Poulsen, Ivan Richter Vogelius, Jens Petersen
code/data similarity 0.65
Abstract · 9 pages, 5 figures, 2 tables. Supplementary Material included

Cone-beam CT (CBCT) scans acquired during radiotherapy capture changes such as tumor shrinkage and radiation-induced tissue changes, offering great potential for foundation models that help monitor individual treatment responses. However, to our knowledge, no large-scale pretrained models for radiotherapy CBCT exist, and evaluating modeling choices is difficult since downstream tasks are not known in advance. We examine how pretraining length, model size, and finetuning strategy relate to downstream clinical performance, and how metrics tracked during pretraining correlate with it. We pretrain vision transformer encoders with masked image modeling on about 150,000 raw 3D radiotherapy CBCT scans, tracking representation quality via validation loss, effective rank, and linear probing. Downstream tasks are classification of feeding tube presence ($n=51,135$ CBCTs, $1,562$ patients) and clinical characteristics (T-, N-stage, sex, age, tumor site; $n=1,278$ patients) in head and neck cancer. We compare low-rank adaptation (LoRA) and full finetuning against no pretraining, varying annotated dataset size. Pretext validation loss was the most consistent predictor of downstream performance; effective rank and linear probing were also associated, but not consistently across model sizes and finetuning methods. LoRA performed comparably to full finetuning while training only 0.5-2.5% of the parameters. Fully finetuned models consistently outperformed models without pretraining across all sizes, with AUROC 0.91-0.97 vs. 0.58-0.85 for feeding tube classification and 0.99-1.00 vs. 0.71-0.97 for sex classification. Model size had only a small, inconsistent effect, suggesting smaller models suffice at lower computational cost. Code and pretrained models (non-commercial license): https://github.com/alexCortinaU/RT-CBCT_SSL

0.48

A Specialized Large Multimodal Model for Interpreting PET/CT in Head and Neck Cancer

cs.CV cs.LG ·2026-09-02 ·Haengbok Chung, SunGyu Kim, Joo hyun Lee, Sangjin Bae, Min Jeong Cho +2 more
similarity 0.65
Abstract

Background: Diagnosing head and neck cancer using PET/CT is clinically challenging and time-consuming due to the anatomical complexity of the region, motivating computer-aided diagnosis (CAD). Generalist Large Multimodal Models (LMMs) remain limited in medical contexts by insufficient domain-specific knowledge, privacy and security concerns, and verbosity, motivating specialized standalone LMMs. Purpose: We evaluated the feasibility of a specialized LMM for automated PET/CT interpretation in head and neck cancer using a large-scale multi-institutional PET/CT dataset, a tailored training curriculum, and autoregressive training. Methods: LLaVA-NeXT was fine-tuned using a two-level curriculum with image-conversation pairs curated by two radiologists from public data. The dataset included clinically important annotations such as primary tumor presence and metastatic lymph node location. Level 1 used 28,000 image-conversation pairs to learn basic information, including modality type and hypermetabolism. Level 2 used 12,975 pairs to learn primary tumor presence and the existence and anatomical location of cervical lymph node metastases. External validation included four institutions with diverse imaging devices. Results: The specialized LMM substantially outperformed ChatGPT and LLaVA-NeXT. In Level-2 external validation, ROUGE-L, ROUGE-S, Cosine Similarity, Precision, Recall, and F1 were 0.8751, 0.8794, 0.8324, 0.8794, 0.8711, and 0.8751, while generalist models consistently scored below 0.1. Primary tumor classification accuracy was 83.14 +/- 1.15% internally and 69.03 +/- 0.81% externally. For lymph node localization, the corresponding scores were 0.6389, 0.6257, 0.5287, 0.5782, 0.6371, and 0.6648. Conclusion: Specialized LMMs show promising results for fast, accurate PET/CT-based diagnostic support and medical education, highlighting their potential for clinical translation.

0.16

Latent-to-Latent Flow for Volumetric Stochastic Segmentation

cs.CV cs.AIcs.LG ·2026-09-07 ·Omar Todd, Sooha Kim, Raghav Mehta, Katherine Mackay, David Bernstein +3 more
top venue similarity 0.64
Abstract · Accepted at MICCAI 2026

Uncertainty arising from inter-observer variability in medical image segmentation plays an important role in developing treatment plans. Research in this area is inhibited by the lack of multiple annotations for large-scale medical datasets, especially for volumetric data, which suffers from additional scaling and computational complexity challenges. Flow matching has emerged as a powerful framework for generative modelling and has also been demonstrated to maintain strong performance when working with latent representations of images. In this work, we introduce a latent-to-latent flow technique for stochastic segmentation of medical volumes via encoded representations of both the image and label space. We evaluate our method on two challenging applications covering delineation uncertainty for radiotherapy planning and multiple organ structure segmentation, improving efficiency up to 14x compared with full resolution models while maintaining clinically relevant performance.

0.33

Anatomy-Change-Aware Bidirectional Selective State-Space Memory for Clinically Deployed Thoracic Radiotherapy Auto-Contouring

eess.IV cs.CV ·2026-09-11 ·Galib Ahmed, Istiak Ahmed, Aritra Islam Saswato, Asib Mostakim Fony, Kazi Shahriar Sanjid +13 more
similarity 0.64
Abstract

We developed DAMM-Net++, a 2.5D architecture for thoracic OAR and target volume segmentation that addresses three persistent challenges in radiotherapy auto-contouring: inter-slice surface incoherence, systematic failure on small low-contrast targets, and the absence of per-case reliability signals. The central component is an anatomy-change-aware bidirectional selective state-space memory that models through-plane anatomical change and selectively propagates context along the axial slice sequence. A boundary-aware decoder sharpens near-surface predictions, and an uncertainty head provides calibrated per-voxel confidence for clinical triage. We evaluated 2,146 patients across four centers, an independent external cohort of 112 patients, and a multicenter reader study involving 17 radiation oncologists on 305 cases. The model achieves a mean Dice of 0.955 and HD95 of 3.78 mm, with the largest gains on low-contrast organs-at-risk (OARs) and target volumes where through-plane context is most critical. The uncertainty head is well-calibrated and supports case-level triage. In the reader study, AI assistance reduced contouring time by 75-80% across experience levels and raised junior-reader IoU from 0.861 to 0.925, matching the unedited model. External validation showed a modest internal-to-external drop (< 5%) with calibrated uncertainty transferring without recalibration. The complete deployment pipeline from DICOM ingestion to TPS-compatible RTSTRUCT export has been integrated into the clinical workflow at a partner hospital, where it is used to assist with contouring. These results suggest that anatomically motivated inter-slice memory, paired with uncertainty-guided review, offers a clinically viable path for thoracic auto-contouring.

0.20

SGRNet: Spatially Guided Radiology Network for Structured Radiological Reporting of Head and Neck Cancer

cs.CV ·2026-08-29 ·Ayush Gupta, Vinkle Srivastav, Prateek Upadhya, Amit Gupta, Krithika Rangarajan +1 more
similarity 0.63
Abstract

Automated radiological report generation can alleviate clinical workloads and eliminate observer variability. However, standard free-text generation models pose hallucination risks in dense regions and fail under data scarcity. We address these challenges in Head and Neck Cancer (HNC) from contrast-enhanced CT (CECT) imaging. To enforce factual safety, we reformulate report generation as an anatomically grounded, multi-label, structured reporting task, predicting localized tumor involvement across a hierarchical clinical schema. To bridge the visual gap from missing metabolic imaging (e.g., PET), we introduce SGRNet (Spatially Guided Radiology Network), incorporating two low-cost spatial priors: automated organ segmentations and weakly supervised tumor localization maps modeled via 3D Gaussian heatmaps. These priors are dynamically integrated via spatial feature modulation to guide the network toward subtle tumor-induced structural alterations. Evaluated on a multi-centric dataset of 184 paired HNC CECT volumes and reports, on five clinically salient, densely packed anatomical subsites, SGRNet achieves a mean Average Precision (mAP) of 0.60, an 8.8 percentage-point absolute improvement over strong volume-only 3D baselines.

0.22