CS Paper Radar

Beyond Accuracy: Benchmarking Uncertainty Quantification in Retinal Disease Screening

Mahvash Jebeli, Mohammad Eslami, Nazlee Zebardast, Lucia Sobrin, Tobias Elze

eess.IV ·submitted 2026-11-17 09:57 UTC · v1
benchmarkdataset

Abstract

Reliable uncertainty quantification (UQ) is essential for evaluating retinal AI systems beyond classification accuracy, yet it remains unclear whether existing UQ strategies generalize across diseases, model architectures, and reliability criteria. We introduce a large-scale, multi-disease benchmark comprising over $242,000$ fundus photographs from $32$ datasets covering diabetic retinopathy, age-related macular degeneration, and glaucoma. We evaluate six representative UQ strategies across three Vision Transformer configurations, assessing discrimination, probabilistic calibration, selective prediction, and conformal prediction-set coverage. Our findings reveal no universally superior model or UQ strategy: methods that improve reliability in one disease or architecture may underperform in another, while improvements in discrimination, calibration, and selective prediction do not necessarily align. Notably, computationally intensive methods do not consistently outperform simple confidence scores, and domain-specific foundation models and calibration-oriented loss functions do not guarantee greater reliability. In contrast, conformal prediction provides a complementary framework for achieving near-nominal marginal prediction-set coverage across diverse settings, subject to its statistical assumptions. These findings challenge universal UQ recommendations and highlight the need for disease-specific, model-specific, and multidimensional reliability evaluation before clinical deployment. We release standardized dataset splits, trained checkpoints, evaluation code, and an interactive research demonstration to support reproducible benchmarking.

Closest papers

Nearest neighbours by meaning across the whole index.

Uncertainty of Vision Medical Foundation Models

cs.LG ·2026-08-31 ·Haoxu Huang, Narges Razavian
similarity 0.78
Abstract

Accurate uncertainty estimation is essential for machine learning systems de- ployed in high-stakes domains such as medicine. Traditional approaches primarily rely on probability outputs from trained models (point predictions), which provide no formal guarantees on prediction coverage and often require additional calibra- tion techniques to improve reliability. In contrast, conformal prediction (region prediction) offers a principled alternative by generating prediction sets with finite- sample validity guarantees, ensuring that the ground truth is contained within the set at a specified confidence level. In this study, we explore the impact of pre-training approach, dataset scale and domain on both point and region-level uncertainty quantification, by studying domain-specific vision medical foundation models vs. general domain vision foundation models. We conduct a comprehensive evaluation across foundation models trained on retinal, histopathological, and Chest X-Rays data, applying various calibration techniques. Our results demonstrate that (1) pre-training on higher-quality domain-specific datasets along with self-supervised learning leads to better-calibrated point predictions than general domain pre-training, (2) stan- dard re-calibration methods alone cannot fully mitigate uncertainty discrepancies across models trained on different data sources, (3) domain-specific foundation model can lead to more efficient conformal prediction. These findings highlight the importance of careful model selection and the inte- gration of both point and region prediction to enhance the reliability and trust- worthiness of medical AI systems. Our work underscores the need for a holistic approach to uncertainty quantification in recent development of medical vision foundation model, ensuring robust and interpretable AI-driven decision-making.

0.06

From Canon to Nikon: Camera Domain Shift and Potential Equity Implications in Retinal AI

cs.CV cs.CY ·2026-10-21 ·Lisa Y. W. Tang
similarity 0.73
Abstract

Diabetic retinopathy is a leading cause of preventable blindness, yet access to regular screening remains limited worldwide. AI offers a scalable approach, but its performance can be sensitive to image quality and acquisition conditions. In a Brazilian cohort of 8,524 patients, retinal images acquired with the Nikon NF5050 were more than twice as likely to have inadequate image quality compared with those acquired with the Canon CR-2, despite refurbished Canon systems costing more than twice as much as Nikon systems. This potential cost-quality trade-off highlights an important consideration for the reliability of AI systems deployed across heterogeneous imaging systems. Using a frozen pretrained retinal foundation model, we systematically evaluate whether camera-associated performance differences persist across 12 retinal disease classification tasks, four input resolutions, and five downstream classifiers. We find that camera-associated performance differences vary substantially across tasks, and that neither higher input resolution nor classifier choice consistently mitigates cross-camera variation. These findings identify camera platform as a potentially overlooked source of performance variability, with implications for the safety and reliability of retinal AI when deployed across heterogeneous imaging systems. We release all source code, evaluation scripts, and per-task results on GitHub to support reproducible benchmarking.

0.24

FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

cs.CV cs.AI ·2026-09-27 ·David Restrepo, Chenwei Wu, Luis Filipe Nakayama, Miguel L. Martins, Stergios Christodoulidis +2 more
top venuebenchmarkdatasetcode/data similarity 0.73
Abstract · Accepted at NeurIPS 2026, Evaluations & Datasets Track. Interactive benchmark dashboard: https://huggingface.co/spaces/dsrestrepo/FOCUS-Reti…

Progress in AI-based retinal image analysis has accelerated with the adoption of foundation models, yet evaluating their real-world reliability remains challenging. Performance reported on a single dataset does not capture how models behave under dataset shift, across clinical definitions, or for different patient subgroups. This limitation is particularly critical in medical imaging, where robustness, calibration, and fairness are essential for safe deployment. We introduce FOCUS (Foundation Ophthalmic Cross-Dataset Understanding under Shift), a cross-dataset benchmark evaluating retinal fundus models across vision-only encoder models (VM), vision-language dual-encoder models (VLM), and multimodal large language models (MLLM). FOCUS harmonizes binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy tasks across ten public datasets spanning diverse geographies, acquisition conditions, and label protocols. The benchmark evaluates models through a unified analysis layer that measures ranking performance, calibration, subgroup disparities, and image-quality robustness. We present a large-scale evaluation covering 532 base configurations and 228 MLLM configurations adapted via LoRA fine-tuning. Results show no model family consistently dominates across tasks and datasets: general VM encoders achieve the strongest average ranking performance, medical MLLMs are competitive but variable, and dual-encoder VLMs benefit substantially from lightweight adaptation. Fine-tuning improves in-domain performance but exhibits heterogeneous transfer to external datasets, particularly in calibration. FOCUS provides a practical framework and public benchmark to assess generalization, reliability, and robustness beyond single-dataset leaderboards.

0.52

Evidential Deep Learning with conditional entropy for glaucoma detection

cs.CV cs.LG ·2026-11-10 ·Hai Siong Tan, Kuancheng Wang
published similarity 0.72
Abstract · 15 pages, 4 figures. To appear in TAAI 2026

We propose a novel conditional entropy function as a basis for quantifying predictive uncertainty in Evidential Deep Learning (EDL) for glaucoma detection from retinal fundus images. Using the EyePACS-AIROGS-light-V2 and RIM-ONE-DL datasets for validation, we found that our proposed EDL model regularized by conditional entropy function produced uncertainty estimates with substantially lower calibration errors than Deep Ensembles, MC Dropout, post-hoc calibration and the conventional EDL framework based on Dempster uncertainty. To further probe our models' interpretability, we analyzed Grad-CAM heatmaps using a focus index representing spatial concentration. Compared with the other uncertainty quantification methods, EDL models exhibited greater separation between the index distributions of correctly and incorrectly classified fundus images. In particular, false negatives were associated with localized Grad-CAM activations that were displaced from the optic disc, consistent with its established clinical importance in glaucoma assessment. Overall, our findings suggest that our proposed EDL framework for glaucoma detection provides not only enhanced uncertainty calibration but also admits a more physiologically consistent interpretability.

0.32

Lesion-Aware Explainable AI for Diabetic Retinopathy Grading from Fundus Images

cs.CV cs.AI ·2026-11-14 ·Truong Thinh Nguyen, Hai Phong Nguyen, Truong Thanh Hung Nguyen, Giang Son Tran, Hung Cao
similarity 0.70
Abstract · 12 pages, 6 figures, 9 tables. Truong Thinh Nguyen and Hai Phong Nguyen contributed equally. Corresponding author: Truong Thinh Nguyen

Post-hoc explanations of deep-learning models for diabetic retinopathy (DR) grading are difficult to trust because different attribution methods highlight different retinal regions for the same image, and a visually plausible heatmap does not show that the highlighted pixels drive the prediction. We propose the Consensus-Calibrated Explanation Map (CCEM), which fuses Grad-CAM++, Ada-SISE, and SmoothGrad Integrated Gradients (SG-IG) using image-specific weights derived from annotation-free reliability cues: deletion sensitivity, inter-method agreement, compactness, retinal containment, and peak structure. CCEM explains a frozen ordinal grader, an EfficientNet-B7 with CBAM and a scalar regression head, pre-trained on EyePACS 2015 in Stage 1 and fine-tuned on APTOS 2019 in Stage 2 with quadratic weighted kappa (QWK)-optimized thresholds; it reaches a QWK of 0.920 on the hidden Kaggle test set and 0.559 on the external MAPLES-DR cohort. Explanations are evaluated on MAPLES-DR, 198 MESSIDOR images with expert pixel-level lesion annotations that are never used for training, threshold fitting, or fusion. On the 193 images with annotated lesions, CCEM obtains the highest Soft Pointing Game score (69.61%; 95% CI 63.7-75.3), defined as the ratio of the strongest lesion-supported saliency to the global map maximum rather than a lesion detection rate, together with the lowest deletion AUC (0.570) and the best deletion-insertion balance (0.115); each gain is significant against the strongest single explainer on paired tests. Annotation-free, per-image reliability weighting therefore turns three disagreeing explainers into one map whose peak evidence is more lesion-supported and more faithful to the model than that of any single explainer.

0.13

A Decision-Aware Evaluation Protocol for Uncertainty in Medical Image Segmentation

cs.CV ·2026-10-15 ·Saket Maganti, Shreyank N Gowda
similarity 0.69
Abstract

Medical segmentation uncertainty is often evaluated with calibration error or pixel-level error-detection scores, but these metrics do not show whether uncertainty improves downstream review decisions. We propose a decision-aware evaluation protocol that separates uncertainty estimation from deferral policy and evaluates their combination under fixed review budgets: which pixels, regions, or images should be accepted, and which should be reviewed. The protocol reports fixed-budget error reduction, risk-coverage behaviour, region-level review proxies, structural failure modes, probability-derived baselines, and sensitivity to review assumptions. We instantiate it across retinal vessel, skin lesion, and polyp segmentation with multiple backbones and uncertainty estimators. Under matched inference budgets and canonical multi-layer MC Dropout, test-time augmentation (TTA) gives a stronger error-detection signal across matched MC--TTA comparisons, including architecture and dropout-placement checks. However, estimator rankings are not universal. For example, probability-derived scores remain competitive, deep ensembles can outperform TTA, learned deferral helps mainly when the base estimator is weak, and rankings can change with review unit, uncertainty target, correction assumptions, and cost model. Structural and second-observer analyses further show that boundary and thin-structure failures are partly tied to annotation ambiguity. These findings support evaluating uncertainty as an estimator-policy-review configuration, not as a standalone score map. Code: https://anonymous.4open.science/r/Uncertainity-in-Medical-Image-Segmentation/README.md

0.34

Detecting Glaucoma Across Multi-ethnic Myopic and Non-Myopic Populations Using an Uncertainty-Aware Vision Transformer: A Multicentre Model Development and Validation Study

cs.CV cs.AI ·2026-09-24 ·Raghavan Lavanya, Yangqin Feng, Ten Cheer Quek, Quan V. Hoang, Linda Yi-Chieh Poon +34 more
similarity 0.69
Abstract

Background: Artificial intelligence (AI)-based glaucoma detection from colour fundus photographs (CFP) offers scalable screening, but performance may decline on external datasets because of differences in ground-truth definitions, populations, and coexisting conditions such as high myopia (HM). We developed and validated a Vision Transformer-based deep learning (DL) model for glaucoma detection across multi-ethnic cohorts with and without HM. Methods: A ViT-B/16 model with predictive uncertainty estimation was developed using 56,483 CFPs (57.1% with myopia; 14.4% with HM). Glaucoma labels were standardised using clinical, imaging, and perimetry data. The model was validated on 16 independent datasets across three continents, including four datasets with explicit HM labels. Findings: Internal AUROC was 98.7% (95% CI 98.2-99.1%), with sensitivity 94.5% and specificity 97.3%. Across 16 external datasets from eight countries, AUROCs ranged from 86.4% to 99.6%. In HM eyes, internal AUROC was 97.8% (95% CI 96.1-99.2%), with sensitivity 94.8% and specificity 93.7%. External HM AUROCs were 86.5% in the Beijing Eye Study and 93.3%, 91.8%, and 85.5% in hospital-based datasets from Taiwan, Thailand, and South Korea. In an exploratory HM clinical evaluation, the model had higher CFP-only diagnostic accuracy than ophthalmologists and trained graders (92.0% vs 70.0%; p=0.008) and performed comparably to glaucoma specialists using full clinical information. Interpretation: The model showed robust glaucoma detection across myopic and non-myopic multi-ethnic populations and may support AI-assisted screening in settings with high HM prevalence.

0.28

A Disease-specific Retinal Aging Biomarker for Predicting Progression to Late Age-related Macular Degeneration

cs.CV ·2026-11-12 ·Hexin Dong, Benjamin Hou, Puzhen Wu, Mingquan Lin, Emily Y. Chew +2 more
similarity 0.68
Abstract · 41 pages, 13 figures (including supplementary material)

Age-related macular degeneration (AMD) is a leading cause of irreversible vision loss in older adults. While chronological age is a primary risk factor, it inadequately reflects individual variations in retinal aging, limiting effective AMD risk stratification and timely intervention. Here, we propose and validate fundus-based AMD age (FAA), which quantifies disease-specific retinal aging in AMD and captures accelerated AMD-related retinal aging, thereby improving prognostic precision over chronological age. Trained on 36,782 images from 2,192 participants in the Age-Related Eye Disease Study (AREDS) dataset and validated across AREDS and AREDS2 cohorts, our model demonstrates significant improvements over traditional chronological age-based approaches. Specifically, integrating the proposed FAA into Cox proportional-hazards models significantly enhances prediction performance. FAA improved the C-index from 0.836 to 0.888 and reduced the 5-year Brier score from 0.050 to 0.046 in AREDS, with consistent gains observed in AREDS2 (C-index 0.715 to 0.756; 2-year Brier 0.078 to 0.076). FAA showed improved prognostic performance across the two datasets and under different phenotype definitions (reading center-defined vs. specialist-defined grading), supporting its potential utility for AMD risk stratification. These findings highlight the advantage of incorporating FAA into AMD risk prediction frameworks, supporting further evaluation of FAA for AMD risk stratification.

0.21