CS Paper Radar

Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients

Yixin Yang, Yueyang Sun, Weichen Liu, Xianbing Zhao, Sicen Liu

cs.AI ·submitted 2026-08-22 12:49 UTC · v1

Abstract

Accurate assessment of patients in intensive care units (ICUs) is essential for timely clinical intervention and improved patient outcomes. Multimodal electronic health records (EHRs), including structured physiological time series and longitudinal clinical notes, provide complementary information for critical care prediction. However, in real-world clinical settings, individual modalities may be partially observed or entirely unavailable, resulting in substantial performance degradation for existing multimodal models. To address this challenge, we propose a multimodal prompt-learning framework for robust clinical prediction under diverse missing-modality scenarios. The proposed framework introduces four complementary types of prompts: generative prompts, missing-signal prompts, missing-type prompts, and temporal prompts. Generative prompts construct surrogate latent representations for unavailable modalities, while missing-signal prompts distinguish observed representations from generated ones. Missing-type prompts condition the model on different modality-availability configurations, whereas temporal prompts perform condition-specific aggregation over temporally encoded clinical sequences. Together, these prompts enable the model to capture missingness-aware intramodal dependencies and cross-modal interactions within a unified architecture. Extensive experiments demonstrate that our method outperforms existing approaches across evaluation metrics on two missingness settings. Ablation and robustness analyses further verify the complementary contributions of the four prompt types and the effectiveness of the proposed framework for clinical prediction from incomplete multimodal EHR data.

Comment
13 pages, 6 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Design Choices for Multimodal Transformer Learning with Missing and Longitudinal Healthcare Data

cs.LG cs.CV ·2026-10-10 ·Rita Cordeiro Mendes, Maria Manuel Pereira, Carlos Santiago, Catarina Barata
similarity 0.71
Abstract

Multimodal learning offers the potential to improve predictions by integrating complementary information from heterogeneous data sources. However, multimodal datasets are frequently incomplete, raising important questions about how models should handle missing information. Although recent approaches increasingly rely on Transformer-based architectures for multimodal fusion, incomplete multimodal data introduces challenges such as missing modalities and intra-modal missingness, which motivate key architectural design choices, such as tokenization granularity. In addition, many datasets are longitudinal, with observations collected across multiple time points. The sequential nature of the Transformer architectures provides a natural framework for modeling such data, but introduces additional considerations, such as how temporal information should be represented. In this study, we systematically investigate these design choices using three different cancer datasets. We compare modality-level and feature-level tokenization and evaluate attention masking and imputation as strategies for handling missing information, both naturally occurring in the datasets and synthetically induced. We further investigate approaches for incorporating temporal information in our training process and how to model it. Experiments are conducted with unimodal and multimodal configurations, as well as different levels of missingness for binary survival classification and multiple time encoding strategies for discrete-time survival prediction. Our results show that while multimodal outperforms unimodal results, missing data heavily impacts predictions and thus effective design choices such as tokenization granularity, and intra-missing handling is essential for reliable prediction.

0.17

TRACE: Tracking Multimodal Clinical Trajectories for Inpatient Risk Prediction

cs.AI ·2026-10-19 ·Minjun Kim, Jong Hak Moon
similarity 0.70
Abstract · 14 pages, 4 figures

Accurate inpatient risk prediction requires modeling heterogeneous clinical evidence that accumulates irregularly over time. This challenge spans in-hospital mortality, prolonged length of stay, and readmission, each with task-specific observation windows. Existing multimodal approaches combine structured health record data, chest x-rays, and clinical text, but primarily focus on modality fusion or missing-modality compensation, with limited attention to temporal trajectories over an admission. We propose TRACE (TRAjectory modeling of multimodal Clinical Events), a daily-scale framework organizing clinical events into day-level trajectories, separating day-invariant global context from day-specific local events drawn from EHR data, chest x-rays, radiology reports, and discharge summaries. The trajectory is encoded using temporal attention, allowing unavailable modalities to emit no tokens. A phenotype-guided contrastive objective further organizes admissions under heterogeneous disease distributions. Across cohorts from MIMIC-IV, eICU, and MIMIC-III comprising nearly 180,000 total stays, we evaluate TRACE on mortality, length of stay, and readmission prediction. TRACE achieves the highest AUROC across all nine dataset-task settings and the highest ACC in eight, outperforming baselines. Ablation studies validate our attention masks and contrastive learning, while trajectory-level attribution localizes risk-associated hospital days and modalities.

0.29

Represent, Then Generate: Multimodal-Conditioned Time-Series Generation under Irregular Missingness

cs.LG ·2026-08-12 ·Haochen Zhang, Jiaheng Guo, Yu-Chao Huang, Nicholas Konz, Tianlong Chen
similarity 0.69
Abstract · 17 pages, 5 figures

Continuous physiological time series underpin modern clinical monitoring, yet many of the most informative signals are invasive, expensive, or simply unavailable for a given patient. Conditional generation offers a remedy: an absent signal can be synthesized from co-recorded signals and routine clinical variables. Existing generators, however, are built around a single conditioning modality and degrade when forced to handle the heterogeneous, irregularly missing mix of time-variant signals and static covariates seen in practice. We propose ReCoGen (Represent Conditions, then Generate), a two-stage framework that decouples multimodal condition representation from target generation. Stage I trains one masked autoencoder per modality, distilling each time-variant condition into a compact and missingness-tolerant token sequence. Stage II trains a flow-matching generator that fuses these tokens with static conditions to synthesize the target signal. Across three physiological benchmarks, including continuous glucose monitoring on AI-READI and arterial blood pressure generation on MIMIC-III and MIMIC-IV, ReCoGen attains the best downstream utility on all sixteen (dataset, task, metric) settings, surpassing six representative conditional generators; on thirteen of them its utility also reaches or exceeds the utility measured on the real signal, a reference we read as an approximate anchor rather than a ceiling. Ablations trace the gains to the conditioning path: learnable cross-attention over the frozen per-modality encoders, and a dual token-plus-AdaLN route for the static conditions. ReCoGen thus turns routinely collected signals into informative surrogates for invasive or unavailable ones, a step toward less invasive, lower-cost continuous clinical monitoring.

0.10

Evidence-Conditioned Completion and Synthesis of Electronic Health Records

cs.AI stat.AP ·2026-11-14 ·Sicheng Shu, Parker Knight, Huitao Li, Yuan Luo, Yifan Peng +3 more
similarity 0.68
Abstract

Patient information can guide electronic health record (EHR) generation, but how its amount shapes predictive utility and disclosure risk remains poorly characterized. Here we present EHRDiB, which unifies visit completion and synthesis by distinguishing original missingness from withheld inputs and combining a shared multimodal representation with an evidence-conditioned latent diffusion bridge. We compared EHRDiB with three completion and eight synthesis methods on MIMIC-IV and numerical and discretized Duke Health representations. Evaluation covered downstream predictive utility, generative diversity, and distributional fidelity for both tasks, and disclosure risk for synthesis. EHRDiB achieved the smallest utility gaps from real-data-trained classifiers in seven of eight completion comparisons. In synthesis, AUPRC gaps were 45.2% smaller for ten MIMIC-IV principal-diagnosis targets and 53.3% smaller for numerical Duke Health data relative to the best comparator. Synthesis with calibrated thresholding yielded the lowest marginal distribution error across all three datasets, with Coverage above 0.95 for both output rules. Across twelve evidence-fraction intervals in MIMIC-IV synthesis, membership-inference AUROC increased from 0.516 to 0.598 as the mean target evidence fraction rose from 0.20 to 0.50; utility gaps narrowed relative to the lowest evidence configuration but did not decrease monotonically. Our study provides a unified framework and empirical foundation for EHR generation that considers both its utility for clinical analysis and its information disclosure risk.

0.20

A Leakage-Aware Multimodal Evaluation Framework for Early Intraoperative Acute Kidney Injury Prediction

cs.LG cs.AI ·2026-09-22 ·Quang Minh Nguyen, Duc Minh Le, Ho Nhat Minh Nguyen, Thuy Quynh Nguyen, Trong Nghia Nguyen
published similarity 0.67
Abstract · accepted on Conference on Optimization, Modeling, Simulation, and Analytics (COMOSA 2026)

Postoperative acute kidney injury (AKI) after major non-cardiac surgery carries substantial morbidity, yet early intraoperative risk stratification remains difficult. In this retrospective cohort study, we propose SynerT, a waveform-only hybrid temporal backbone that combines a causal dilated TCN with a hierarchy of dilated recurrent layers to encode early intraoperative physiologic trajectories for AKI risk prediction. Building on SynerT, we further design two model variants that extend the backbone with structured clinical context: SynerT-MM, a late-fusion multimodal extension that integrates hemodynamic burden summaries and preoperative covariates, and SynerTStack, a leakage-safe stacked ensemble that combines cross-validated predictions from SynerT-MM with strong tabular baselines at the meta-learning stage. All models are evaluated under a strict leakage-aware framework on VitalDB, a high-fidelity perioperative database, with prediction restricted to information available within the first 60 intraoperative minutes. Among 2,413 waveform-usable cases (180 AKI-positive; 7.46% prevalence), SynerT fell well below strong structured-data baselines, demonstrating that waveform-only temporal modeling is insufficient under strict early constraints. SynerTMM recovered discrimination by incorporating hemodynamic burden summaries and preoperative covariates, and SynerT-Stack achieved the best overall performance across AUROC, AUPRC, and F1-max. Cross-fitted Platt recalibration substantially corrected calibration defects in both multimodal variants, and decision-curve analysis confirmed the recalibrated stacked model delivered the strongest net clinical benefit across low-to-intermediate thresholds.

0.20

DeliMF: Gated Cross-Attention Fusion of Structured Time Series and Clinical Text for Early Intensive Care Delirium Prediction

cs.LG ·2026-10-17 ·Yibo Yang, Zerui Huang, Haomiao Yang, Zixia Wang
similarity 0.67
Abstract · Preprint

Delirium affects a large fraction of intensive care unit (ICU) patients and foreshadows longer stays, lasting cognitive decline, and higher mortality. Flagging the patients who will become delirious within the first day of admission would give clinicians a window to intervene. Yet state-of-the-art predictors rely only on the structured electronic health record (EHR) of vital signs, laboratory values, and medications, while overlooking the free-text clinical notes written in the same window, which carry information the numbers cannot encode. To this end, we present DeliMF, a multimodal framework that fuses the structured time series with clinical text. It encodes the structured signals with a temporal Transformer and the time-stamped radiology reports with a clinical text encoder, and fuses the two through a gated cross-attention module. The gate learns a per-patient weight on the text; when a patient has no note the gate closes and DeliMF reduces to its structured-only form, giving robustness to missing text by construction. On an incident-only MIMIC-IV cohort, DeliMF improves significantly over the baselines, particularly on patients who carry clinical text, while losing nothing on those without. Ablation studies confirm the contribution of multimodal fusion and gated cross-attention, and a case study further shows that fusing clinical text with the structured stream recovers delirium cases that the structured data alone misses. We hope DeliMF encourages early ICU prediction to make fuller use of the overlooked clinical text and contributes to the early identification and timely intervention of delirium.

0.24

FedCoRe: Target-Adaptive Completion for Missing Modalities in Healthcare Federated Learning

cs.CV cs.AIcs.LG ·2026-08-18 ·Holger R. Roth, Ziyue Xu, Peter Cnudde
published similarity 0.67
Abstract · Accepted to the 7th Workshop on Distributed, Collaborative & Federated Learning, DeCaF 2026, MICCAI, Strasbourg, France

Federated multimodal models often assume every site has every modality, although hospitals differ in access to EHRs, chest radiographs, and ECGs. We study this setting on a MIMIC-derived respiratory deterioration task with simulated FL clients and introduce FedCoRe (Federated Cross-Modal Representation Completion). FedCoRe learns representation- or logit-space corrections rather than generating synthetic ECGs or CXR images. When a client observes a modality that may be missing at deployment, it evaluates the same example with and without that modality to obtain paired supervision. Only clients with such pairs update the completion module, and validation may retain the unchanged prediction. We freeze the trained multimodal predictor during evaluation so that measured differences come only from completion. Hiding ECG reduced AUROC by about 0.085; paired-example FedAvg restored 0.0415 AUROC, or 49.0% of the lost performance. We therefore report two distinct effects: paired-example FedAvg partially recovers the missing-ECG gap, while validation-selected completion is a task-specific classifier-logit correction rather than literal ECG recovery. For CXR, effect-aware completion recovers 52.8% of the loss in a controlled test where CXR is hidden. Paired-example FedAvg transfers part of this effect, but validation keeps the no-completion baseline for deployment cases whose inputs lack CXR. Thus, FedCoRe should be read as a validation-gated completion/correction framework: it can recover missing-modality signal in supported settings, but it should be deployed only when paired examples and validation evidence support that modality.

0.14

TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment

cs.CV cs.AI ·2026-08-31 ·Merna Bibars, Bolaji Omofojoye, Allan I. Levey, Rachel Hershenberg, Gari D. Clifford +1 more
similarity 0.67
Abstract · 19 pages, submitted to IEEE Transactions on Affective Computing

Depression and anxiety in older adults with Mild Cognitive Impairment (MCI) are frequently underdiagnosed due to limited access to care. Multimodal analysis of remote clinical interviews is a scalable screening approach, but existing methods have three limitations. First, they do not correct temporal misalignment across multimodal features extracted at different resolutions, inducing spurious cross-modal associations. Second, remote recordings exhibit uneven modality dropout, but missing values are often zero-filled, making them indistinguishable from valid near-zero measurements. Finally, they do not jointly attribute predictions to modalities, questions, and interview moments, limiting fine-grained clinical interpretation. We propose a Temporally-Aligned, Missingness-Aware, Interpretable (TAMI) multimodal fusion framework. TAMI aligns speech, language, facial, and physiological features within question-answer segments on a shared timeline, encodes modality-level missingness over time, and conditions fusion on question context. In interviews with 49 older adults with MCI, TAMI achieved area under the receiver operating characteristic curve (AUROC) scores of 0.68 (depression) and 0.69 (anxiety). Fine-grained temporal alignment of multimodal features produced the largest performance gain ($Δ{\geq}0.1$). Multi-level interpretability analysis revealed that depression classification relied on eyegaze and open-ended questions, while anxiety classification depended on eyegaze and head pose, with attribution uniformly distributed across questions. Using only responses to the open-ended questions (5.1min), the depression model achieved an AUROC score of 0.67, which was not significantly different from using the full interview (19min) ($p>0.05$). Our findings support designing interview protocols centered on open-ended questions for depression screening in older adults with MCI.

0.30