CS Paper Radar

Missingness-Aware Conformal Prediction Under Cross-Hospital Distribution Shift

Liang You, Dongwen Ou, Hengyu Shi, Siyuan Dai

cs.LG stat.AP ·submitted 2026-09-25 04:04 UTC · v1

Abstract

Clinical measurements are recorded for some patients but not others, at rates that differ across hospitals, and marginal conformal coverage does not ensure coverage within groups defined by missingness. We propose a missingness-aware conformal calibration procedure for mortality prediction under cross-hospital distribution shift. It selects a measurement on an independent sample, groups patients by whether that measurement is recorded, and applies Mondrian calibration within each group, so no calibration outcome is reused. We evaluate the procedure across hospitals in eICU and across care units within one MIMIC-IV hospital, using three predictors. Relative to pooled calibration, it reduces the average worst-group coverage gap on its selected groups in all six settings, with a median reduction of 1.9 percentage points; paired site-bootstrap intervals exclude zero in five. These gains do not extend uniformly. Calibration by predicted risk achieves smaller gaps on a broader panel of missingness groups, and when eICU hospitals are evaluated separately, the gain shrinks for all three predictors and reverses in sign for one. We explain this discrepancy with a hospital-level decomposition. Pooling reweights hospitals through a covariance between group shares and coverage errors, and lets errors of opposite sign cancel: weighting explains the reversal, and cancellation accounts for most of the attenuation for the other two predictors. Constructed population distributions show that pooled and within-hospital evaluations can rank calibration methods oppositely even without sampling noise. Pooled improvement alone therefore cannot establish better coverage within hospitals, even when the calibration groups are fixed.

Comment
33 pages, 2 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Stochastic Grouping Conformal Prediction for Effective Subgroup Reliability

cs.LG cs.AI ·2026-10-08 ·Meihui Zhong, Wenxin Tai, Ting Zhong, Fan Zhou
theory similarity 0.67
Abstract · 9 pages

Conformal prediction offers a distribution-free coverage guarantee, making it especially attractive for clinical applications. Standard conformal prediction, however, provides such guarantees only at the population level, and its prediction sets can exhibit coverage disparities across clinically important subgroups. A natural remedy is to calibrate within predefined groups. However, this can require access to sensitive subgroup attributes and is prone to a worst-group bottleneck: protecting the most difficult subgroup can inflate prediction sets for all, increasing cognitive burden on decision makers. To this end, we propose Stochastic Grouping Conformal Prediction (SGCP), a conformal framework for subgroup-reliable uncertainty quantification. It learns a stochastic grouping map that allows each sample to draw calibration information from others with similar calibration behavior, yielding a local score law that boosts reliability across subpopulations. We prove that SGCP retains the standard coverage guarantee. Experiments on synthetic and real-world benchmarks show that it consistently reduces subgroup coverage gaps while achieving smaller or comparable prediction set sizes relative to existing baselines.

0.18

Alert-List Concordance for Evaluating Imputation Strategies in ICU Sepsis Prediction

cs.LG q-bio.QM ·2026-10-15 ·Priyanka Siddappa, Muntaqim Ahmed Raju, Shifat Haider Al Amin, Ruizhe Ma
published similarity 0.60
Abstract · 8 pages, 2 figures. Accepted at the 14th Workshop on Data Mining in Biomedical Informatics and Healthcare (DMBIH 2026), held in conjunction …

Preprocessing is an under-measured reliability risk in clinical prediction: a model can keep a similar AUROC and still change who receives an alert. Imputation quality is commonly evaluated using reconstruction error or aggregate discrimination, neither of which directly measures agreement on which patients are flagged. At a matched alert rate, pipelines with similar aggregate performance can produce different patient lists. This question is especially relevant under the frequent, structured missingness of real ICU records. We therefore introduce alert-list concordance, which asks whether two pipelines identify the same high-risk patients, measured by the Jaccard similarity of their top-k% alert sets with patient-level bootstrap confidence intervals. On PhysioNet/CinC 2019 (40,336 patients, 71.9% missingness), we compare nine imputation strategies and a native-missingness baseline using a fixed XGBoost configuration, fitted and Platt-calibrated separately for each strategy. After Bonferroni correction, 22 of the 28 reconstruction-method pairs showed no statistically significant difference in patient-level AUROC, yet their alert lists overlap at Jaccard 0.518 to 0.750. Similar aggregate discrimination can therefore accompany different patient selections. ForwardFill is a separate outlier, pairing higher discrimination with the lowest overlap in the matrix, 0.40 to 0.45. The broad pattern holds within each hospital system and under six patient-level aggregation rules. At the row level, eight of ten methods fall within an AUROC range of 0.011. In this row-level pipeline, deep imputers did not outperform mean imputation on discrimination, and the sequential implementations cost 5 to 12 times mean imputation in total pipeline wall-clock time on our hardware.

0.54

Conformal Prediction for Spatially Dependent Data via Sequential Whitening

cs.LG stat.MLstat.APstat.ME ·2026-10-07 ·Ayush Baran Sen, Arkajyoti Saha
similarity 0.60
Abstract

Split conformal prediction uses prediction errors on held-out (calibration) data to determine how wide the prediction intervals should be. It guarantees distribution-free finite-sample coverage when these errors and the error at the target site are exchangeable. This assumption may fail under spatial dependence and nonrandom sampling geometry. Existing spatial methods use fitting residuals to remove the predictable part of spatial variation from calibration and target errors. However, the spatial variation that only the calibration residuals can predict remains in both the target and calibration errors, reducing the efficiency and stability of the interval. We address this by additionally conditioning on the calibration residuals sequentially, which scales to large networks through nearest-neighbour approximations. Under a correct working covariance and an elliptical residual law, the resulting interval has exact finite-sample coverage under any spatial design, and under further conditions it is asymptotically oracle efficient. We also bound coverage loss under covariance misspecification and develop a diagnostic that identifies regions at risk of undercoverage. In simulated data, our method produces narrower and more stable intervals than global and localized state-of-the-art alternatives. In a national PM2.5 application, it produces narrower intervals within the network and identifies regions at risk of coverage failure.

0.20

Group-Conditional Conformal Risk Control: Exact Guarantees, Limits, and Overlapping Extensions

cs.LG stat.MLmath.ST ·2026-11-07 ·Tareq Aldirawi, Wenge Guo
similarity 0.59
Abstract · 68 pages, including 40 pages of appendices

We study group-conditional conformal risk control (CRC) for disjoint and overlapping groups under bounded monotone losses. For disjoint groups, we establish exact finite-sample conditional risk control, two-sided bounds on the group-restricted risk, and fundamental limitations arising from local support and calibration-rank granularity. For overlapping groups, we study two exactly valid approaches: group-restricted calibration followed by union deployment, and direct calibration of the deployed prediction-set family. We characterize the union's limiting reconciliation gap, identify conditions for its disappearance, and establish sharp lower bounds for structured losses in specified product-form models. We also demonstrate a strict size advantage for direct calibration in an ordinal model. Experiments show that direct calibration can reduce prediction-set size, but its efficiency depends on group structure, loss type, and shape fitting. Results on historical diabetic-retinopathy scores further illustrate these tradeoffs. Overall, local support, reconciliation slack, and threshold heterogeneity determine efficiency, with neither approach uniformly dominating.

0.28

Conformal Risk Minimization for Semi-Supervised Domain Adaptation via Optimal Transport

cs.LG ·2026-08-24 ·Manos Giannopoulos, Yi Shen, Michael M. Zavlanos
similarity 0.59
Abstract · 18 pages, 2 figures, 6 tables

In high-stakes healthcare applications, machine learning models are frequently trained on data from one patient population and deployed on another, creating a distribution shift that degrades both accuracy and reliability. Semi-Supervised Domain Adaptation (SSDA) addresses this by leveraging labeled data from some source domain to improve model performance on a target domain where labels are scarce. However, existing SSDA methods optimize primarily for point-prediction accuracy and offer no principled uncertainty quantification --- a prerequisite for clinical trust. Conformal Prediction (CP) can address this limitation by providing prediction sets with rigorous, distribution-free coverage guarantees. However, applying CP post-hoc to a pre-trained model can yield prohibitively large prediction sets, as SSDA pre-training methods do not account for the nonconformity score geometry that determines conformal set size. Conformal Risk Minimization (CRM) has been used to resolve this issue in the fully supervised setting by integrating the CP objective directly into model training, but it requires a large labeled dataset to compute nonconformity thresholds during training, precisely the data that is scarce in the SSDA regime. We propose an end-to-end framework that integrates CRM into the SSDA training objective, enabling effective CRM in the limited-labeled-target-data regime. The key idea is to utilize Optimal Transport (OT) to generate pseudolabels for unlabeled target instances, providing the additional training signal needed by CRM to operate using only a small labeled target set. This results in a model jointly optimized for domain invariance and conformal efficiency, producing prediction sets that are compact, coverage-valid, and support domain-specific constraints such as excluding mutually contradictory diagnoses in skin lesion classification.

0.11

Beyond Imputation: Mask-Adaptive Conformal Prediction via Tree Embeddings on General Missing Data Mechanisms

cs.LG stat.ML ·2026-11-01 ·Jiarong Fan, Juhyun Park, Thi Phuong Thuy Vo, Nicolas J-B. Brunel
similarity 0.59
Abstract

Conformal prediction (CP) provides a distribution-free framework for uncertainty quantification with finite-sample marginal coverage guarantees. Yet, missing values introduce distributional heterogeneity, where marginal coverage can hide systematic undercoverage for certain missingness patterns. At the same time, existing methods based on distances or smoothness require a continuous representation of partially observed covariates. Although imputation restores continuity, it obscures the missingness patterns and introduces task-dependent overhead. We develop a new CP framework that is \emph{adaptive to missingness patterns}, without imputing covariates. We rely on data augmentation by feature embedding that accounts for missingness. Our framework represents each input with missing values by a tree embedding together with its raw missingness mask, and then learns a regularized quantile score function. Varying the choices for the penalty function and the augmentation enables us to characterize three practically relevant validity levels: exact coordinate-wise validity, exact validity on observed masks, and approximate missingness-conditional guarantees for general mechanisms (MCAR, MAR, MNAR). Experiments on synthetic and real datasets support these guarantees and show that the proposed method reduces miscoverage risk while maintaining informative intervals, especially in non-MCAR settings. These results highlight the practical value of uncertainty quantification that adapts to missingness.

0.18

Evidence-Conditioned Completion and Synthesis of Electronic Health Records

cs.AI stat.AP ·2026-11-14 ·Sicheng Shu, Parker Knight, Huitao Li, Yuan Luo, Yifan Peng +3 more
similarity 0.58
Abstract

Patient information can guide electronic health record (EHR) generation, but how its amount shapes predictive utility and disclosure risk remains poorly characterized. Here we present EHRDiB, which unifies visit completion and synthesis by distinguishing original missingness from withheld inputs and combining a shared multimodal representation with an evidence-conditioned latent diffusion bridge. We compared EHRDiB with three completion and eight synthesis methods on MIMIC-IV and numerical and discretized Duke Health representations. Evaluation covered downstream predictive utility, generative diversity, and distributional fidelity for both tasks, and disclosure risk for synthesis. EHRDiB achieved the smallest utility gaps from real-data-trained classifiers in seven of eight completion comparisons. In synthesis, AUPRC gaps were 45.2% smaller for ten MIMIC-IV principal-diagnosis targets and 53.3% smaller for numerical Duke Health data relative to the best comparator. Synthesis with calibrated thresholding yielded the lowest marginal distribution error across all three datasets, with Coverage above 0.95 for both output rules. Across twelve evidence-fraction intervals in MIMIC-IV synthesis, membership-inference AUROC increased from 0.516 to 0.598 as the mean target evidence fraction rose from 0.20 to 0.50; utility gaps narrowed relative to the lowest evidence configuration but did not decrease monotonically. Our study provides a unified framework and empirical foundation for EHR generation that considers both its utility for clinical analysis and its information disclosure risk.

0.20

Learning to Adapt and Calibrate: Score Distribution Alignment for Few-Shot Uncertainty Prediction in Medical VLMs

cs.CV ·2026-09-09 ·Xuan Cuong Ngo, Ngan Le
similarity 0.57
Abstract

Uncertainty estimation for medical vision--language models (VLMs) using conformal prediction has gained increasing attention due to its distribution-free coverage guarantees. However, standard conformal prediction relies on exchangeability between calibration and test data and typically requires a sufficiently large calibration set to obtain reliable coverage. These assumptions are difficult to satisfy in few-shot transfer settings, where only a small labeled support set is available to adapt a pretrained VLM to a new medical task, while an unlabeled query set is used for evaluation. Supervised fine-tuning on the support set changes the model parameters and consequently shifts the nonconformity score distribution, breaking exchangeability between calibration and query samples and leading to unreliable coverage under distribution shift. Existing transductive conformal adaptation methods often preserve validity by avoiding supervised updates. While this helps maintain conformal assumptions, it underutilizes the scarce labeled support data and limits task adaptation, which is the primary objective in few-shot learning. In this setting, conformal prediction should serve as an uncertainty estimation layer that supports the adapted model, rather than preventing adaptation itself. To this end, we propose AlignCP, a framework that reconciles supervised few-shot adaptation with conformal uncertainty estimation under non-exchangeability. AlignCP learns a reweighted calibration distribution that reduces the score-level discrepancy between the labeled support set and the unlabeled query set. By aligning the one-dimensional nonconformity score distributions, AlignCP aims to close the coverage gap induced by adaptation without requiring query labels.

0.20