CS Paper Radar

Leakage- and Label-Aware Benchmarking of Class-Imbalance Strategies for Multimodal Chest X-Ray and EHR Classification

Dang-Khoa Bui, Quang-Huy Tran, Minh-Khoi Nguyen-Bui, Duc-Tuan Ngo, Thanh-Trong Tran, Tuan-Khoi Nguyen, Hoang-Anh Ngo

cs.LG cs.CV ·submitted 2026-11-05 12:47 UTC · v1
benchmark

Abstract

Benchmarks of class-imbalance strategies for multimodal chest X-ray (CXR) and electronic health record (EHR) prediction are often evaluated on record-level splits of cohorts in which one patient contributes several radiographs. We audit such a benchmark on 29,083 MIMIC-CXR radiographs of 10,479 patients with eight diagnosis targets. First, the record-split benchmark was misleading: a lookup table that copies the label of a development-set radiograph with the same EHR vector nearly matched the best model, and on unseen patients every model fell to the level of predicting every case positive. Second, tracing the labels back to MIMIC-IV explained why: each patient's labels are keyword matches on the diagnosis codes of a single admission, copied, with one EHR vector per patient, to all of the patient's radiographs, so about a third of the radiographs carry labels that differ from the codes of their own admission. Third, under patient-separated evaluation with admission-aligned and rule-refined labels, the initial conclusions do not hold: in the evaluated configurations no imbalance strategy raised the observed mean macro F2 over no augmentation by more than 0.01, tuned tree ensembles showed no consistent advantage over tuned logistic regression, and the released EHR variables stayed at the level of the all-positive baseline, whereas the diagnostic history available at imaging time and image embeddings exceeded it, including, with rule-refined labels, a CheXpert-trained encoder that never saw these radiographs; RAD-DINO had seen 96.6% of them during self-supervised pretraining. Adding either encoder to the diagnostic history gave no statistically significant F2 gain; exploratory AUROC gains were significant with RAD-DINO but not with DenseNet. We provide reusable diagnostics, a leakage-aware protocol and a checklist.

Comment
10 pages, 1 figure, 6 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation

cs.CL cs.AI ·2026-09-16 ·Daniel P. Jeong, Charles Q. Li, Hossein Hosseiny, Nitya M. Bhalla, Fatma Uyar Morency +3 more
similarity 0.67
Abstract · Preprint

Radiologists follow heterogeneous reporting practices. Two radiologists examining the same image and identifying the same clinical findings might nevertheless compose superficially distinct reports, varying in terminology, shorthand, formatting, and level of detail. These variations in reporting norms represent an under-appreciated obstacle in efforts to evaluate AI-based radiology report generation (RRG) models, where machine-generated reports are typically assessed based on their concordance with human-generated references. In this paper, we quantify the sensitivity of established evaluation metrics to variations in reporting practices, revealing impacts large enough to alter the rankings of models. We introduce a radiologist-informed taxonomy of variations in radiology reporting practice and a method (ReRef) that rewrites reference reports along the axes of our taxonomy while preserving clinical interpretation. For instance, when comparing the performance of nine RRG models on MIMIC-CXR using RadCliQ-v1, condensing the discussion of normal findings in the reference reports causes Libra to drop from first to second place while CheXOne rises from third to first. Our results suggest that many current metrics fail to decouple clinical interpretation from conformity to reporting practices and that choosing the ``right'' references that accurately reflect the desired reporting practices can be important in practice. To support future research, we release MIMIC-CXR-Ext-ReRef, a radiologist-validated dataset of 120 (original, alternative) reference report pairs derived from MIMIC-CXR.

0.10

Cross-dataset transportability of pediatric chest X-ray deep learning across three countries: discrimination, calibration, operating-point failure, and limited-label recovery

eess.IV cs.CV ·2026-09-04 ·Nazim-E-Alam
similarity 0.67
Abstract

Background and Objective: External evaluation of medical-imaging AI is often collapsed into discrimination. We evaluated a computational protocol that separately tests discrimination, probability calibration, fixed operatingpoint transport, shortcut-associated signal, and limited-label recoverability for pediatric pneumonia classification across datasets from three countries. Methods: After exact-duplicate removal, 5,824 Guangzhou radiographs supported leakage-controlled source development and internal testing. A frozen three-seed DenseNet121 dual-view ensemble was evaluated zero-shot on BDCXR-3257 from Bangladesh (n = 3, 257) and an untouched harmonized VinDr-PCXR/PediCXR test cohort from Vietnam (n = 1, 077). Matched seed-42 variants tested architectural robustness. Secondary BDCXR analyses used a fixed 651-image adaptation pool and 2,606-image hold-out; 163, 326, and 651 labels represented 5%, 10%, and 20% of complete BDCXR. Results: Internal AUROC was 0.976 with 95.1% sensitivity. BDCXR and VinDr-PCXR AUROC were 0.798 and 0.742, while frozen-threshold sensitivity fell to 6.2% and 0%. Source-to-BDCXR AUROC degradation occurred for a full-image baseline (0.961 to 0.749), ungated dual-view model (0.977 to 0.766), and gated MixStyle model (0.966 to 0.789). With 163 BDCXR labels, Platt recalibration preserved AUROC while increasing held-out sensitivity to 88.3%, but specificity was 47.9% and the alert rate was 78.5%. Two hundred repeated 163-label fits confirmed sensitivity recovery but substantial specificity variability. Conclusions: Cross-dataset shifts across countries affected ranking, probability alignment, and source-defined decision behavior differently. Transport studies should evaluate these components separately and quantify the operational burden of apparent recovery.

0.06

From Adults to Children: Benchmarking Foundation Models for Chest X-Ray Classification under Domain Shift

cs.CV ·2026-10-15 ·Sara Inácio, Hugo Proença, João C. Neves
publishedbenchmark similarity 0.65
Abstract · Accepted at WACV 2027

Foundation models have shown significant potential across diverse fields and have been applied to medical imaging, but their robustness across patient populations remains unclear. In this work, we investigate foundation models' ability to generalize from adult to pediatric chest X-ray classification. To the best of our knowledge, this is the first benchmark to compare multiple foundation models under a common adult-to-pediatric evaluation setting while also analyzing the effect of adaptation depth. We harmonize VinDr-CXR and PediCXR into a five-label classification task and evaluate six foundation models and two CNN baselines. All models are adapted using adult data only and tested in both the in-domain and pediatric domains using linear probing, partial fine-tuning, and full fine-tuning. The benchmark reveals a clear and consistent trend: all models suffer a substantial performance degradation when transferred from adult to pediatric data, with drops exceeding 10 percentage points. While Macro-AUROC values reach approximately 92% on the adult test set, performance decreases to around 78% on the pediatric domain. Another important finding is that increasing the depth of adaptation improves pediatric performance on average, although the effect varies across models. Moreover, strong in-domain performance does not necessarily indicate greater robustness to domain shift.

0.40

Retrieval-Grounded Multi-View Reasoning for Clinically Reliable Radiology Report Generation

cs.CV ·2026-11-03 ·Wanshu Fan, Zeyu Wang, Zheng Tong, Qi Ai, Wencai Weng +6 more
similarity 0.65
Abstract

Chest radiograph report generation is hampered by retrieved reports that resemble the current image but are clinically inconsistent with it, and by diagnostic-label imbalance. We present RG-MVR, a retrieval-grounded framework comprising Retrieval-Augmented Evidence Grounding, which combines dense memory summaries with visually gated sparse retrieved evidence; Adaptive Multi-view Evidential Reasoning, which weights image, memory, and retrieved evidence per case; and a Clinical Prompt Learner, which models finding mention separately from positive, negative, or uncertain status. On MIMIC-CXR, RG-MVR achieved a clinical-label F1 of 0.483 and a BLEU-4 of 0.118 (IU X-ray: 0.202 and 0.110). In an exploratory external cohort of 22 examinations, clinical-label agreement declined markedly (F1, 0.071) and unsupported statements remained common, although physicians found fewer unsupported temporal comparisons (9 vs. 20 cases) and device-related statements (7 vs. 15) than with the baseline. Evidence-aware retrieval and structured diagnostic prompting may reduce specific reporting errors, but broader external validation is required.

0.22

A radiographic world model for clinical reasoning and evidence generation

cs.AI ·2026-09-07 ·Suyang Xi, Songtao Hu, Shansong Wang, Mojtaba Safari, Luke del Balzo +6 more
highly cited similarity 0.65
Abstract

Medical imaging artificial intelligence (AI) is commonly developed as separate mappings from radiographs to diagnostic outputs or from clinical descriptions to generated images, although both arise from the same underlying radiographic state. A world-model formulation instead seeks to learn an internal representation of this state that can support both clinical readout and conditional simulation of radiographic observations. Here we introduce MedDream, a radiographic world model that learns a shared continuous latent state from paired chest radiograph-text observations for diagnostic reasoning and report-conditioned evidence generation. MedDream was pretrained on 2.65 million leakage-controlled chest radiograph-text pairs curated from 4.40 million candidates. Across eight clinical datasets and two independent reader cohorts, MedDream outperformed leading diagnostic and generative comparators. For diagnostic reasoning, MedDream showed strong generalization across disease recognition, label-scarce adaptation, severity assessment, and localization, while MedDream-supported review increased mean resident concordance with independent radiologist consensus from 56.3% to 63.0%. For evidence generation, MedDream produced radiographs that preserved clinically relevant pathology and improved downstream performance on held-out real data, with synthetic augmentation increasing external VinDr-CXR macro-AUROC from 76.4% to 81.4%. More importantly, conditioning generation on prespecified subgroup performance gaps enabled targeted evidence construction, increasing weighted F1 by 3.1 percentage points in Asian patients, whereas matched-volume unguided augmentation decreased it by 2.3 points. These findings establish radiographic world models as a path toward medical AI that learns clinically meaningful internal states for interpreting, simulating, and constructing evidence for clinical use.

0.36

GLR-MM: Graph-Based Global-Local Reconstruction for Robust Multimodal Chest X-ray and EHR Representation Learning under Missing Modalities

cs.LG cs.AI ·2026-09-20 ·Surbhi Sharma, Nikhil Manali, Devesh Maheshwari
published similarity 0.64
Abstract · Accepted in MICCAI

Clinical multimodal models must often predict before all chest X-ray (CXR) and electronic health record (EHR) inputs are available. Existing approaches align observed representations, model missingness, or reconstruct across modalities, but do not jointly exploit within-patient and clinically similar inter-patient evidence. We propose GLR-MM, a Graph-Based Global-Local Reconstruction framework for early ICU mortality prediction. It maps five CXR-EHR modalities to a shared space, reconstructs missing embeddings through complementary local cross-modal and global graph-attention branches, adaptively fuses their estimates, and optimizes class-balanced prediction, reconstruction, and contrastive objectives. On 9,620 MIMIC-derived ICU stays, we evaluate 10%, 30%, and 50% random modality missingness with shared deterministic masks. MUSE performs better under mild and moderate missingness, whereas GLR-MM achieves higher AUROC and AUPRC at 50% by 0.0088 and 0.0249, respectively. These results indicate that graph-guided reconstruction is most useful when inputs are severely incomplete.

0.24

Evaluation Choices Shape Biomedical ML Claims: A Pediatric Pneumonia Benchmark Case Study

cs.CV cs.LG ·2026-09-29 ·Bhanu Prakash Vangala, Sowmya Guda, Latha Peddi, Navya Vangala
benchmark similarity 0.64
Abstract

Biomedical machine learning papers often compress model performance into one headline number. That number can look like a property of the model even when it depends strongly on how the benchmark was evaluated. We study this problem on the widely used Kermany pediatric chest radiograph dataset using nine image classifiers and a controlled evaluation protocol. Under the same protocol, the eight pretrained backbones differ by only 0.026 AUROC. In contrast, changing whether the backbone is frozen or fine-tuned changes AUROC by 0.044 on average, and changing the decision threshold changes balanced accuracy by 0.090 on average. The official test split is also measurably different from the training pool: a partition classifier distinguishes them at AUC 0.697, rising to 0.898 for normal radiographs. Most strikingly, a classifier using only file properties, with no image anatomy, reaches 0.992 balanced accuracy within the training pool but falls to 0.496 on the official test split. Validation-fitted thresholds and calibration also transfer imperfectly. These results show that a high benchmark score can support different conclusions when the split, training policy, threshold, metric, calibration, and uncertainty are not communicated with it. We end with a seven-item reporting recommendation in which each item is tied to an effect measured in the study

0.11

Rethinking weakly supervised localization on chest radiographs: position nulls, protocol dependence, and PACE

cs.CV cs.LG ·2026-09-29 ·Shixin Qiu
benchmark similarity 0.64
Abstract

Weakly supervised localization on chest radiographs is reported as a single overlap accuracy, and those numbers are not comparable across papers. We measure what one contains. Against position-null controls that keep a readout's own box and move only its position, class activation mapping and its gradient-weighted variant beat the null on two to four of seven annotated findings on ChestX-ray14 and two of eight on CheXlocalize. The null is beatable: three board-certified radiologists clear it on four to seven. Three rarely stated choices each move a result by more than most reported method effects: the results regime, the extraction threshold, and the mask-to-box rule, whose cost is predicted by how fragmented the thresholded mask is. Predicted box area alone reverses a paired comparison. These measurements produced PACE, for \emph{population anomaly and class evidence}, which multiplies the finding identity from the label-trained head by the spatial evidence from a frozen radiograph backbone that never sees a pathology label. It is the case study that made the protocol dependence visible, and we report its costs beside its gain: in an exploratory matched-area analysis it improves strict-overlap accuracy on both corpora and beats a label-derived second map, but it is the least faithful of the three readouts to the classifier's causal map, its advantage depends on the box rule, and, this benchmark releasing no localization validation set, every design choice behind it was made on the boxes it is scored on. Its one clean transfer beats one baseline and loses to the other.

0.27