CS Paper Radar

Evaluation of Clinically Steerable Retinal Image Generation from Foundation Model Latent Spaces

Zuzanna A. Wakefield-Skórniewska, Bartłomiej W. Papież

cs.CV ·submitted 2026-08-13 16:40 UTC · v1
published

Abstract

Medical foundation models learn latent representations of clinically meaningful phenotypes, yet their ability to support controllable image generation remains largely unexplored. We evaluate four retinal foundation models within the representation tokenizer framework and examine whether demographic and clinical information encoded in latent representations from foundation models is preserved during synthetic image generation. We show that generated representations and images faithfully inherit phenotype information when evaluated within their originating foundation models, consistently outperforming conventional latent diffusion on multiple downstream prediction tasks. However, these gains largely disappear when evaluated using classifiers trained on real images, revealing a previously uncharacterised synthetic-to-real representation gap. These findings demonstrate that foundation-model latent spaces provide a powerful substrate for controllable retinal synthesis while highlighting the need to better align synthetic representations with real-image distributions.

Comment
MICCAI 2026 Workshop SASHIMI Submission

Closest papers

Nearest neighbours by meaning across the whole index.

Exploring the Latent Structure of AMD Severity Using Retinal Foundation Models

eess.IV cs.CV ·2026-10-09 ·Niccolò Fineschi, Jovana Panic, Atsushi Watanabe, Hideaki Hayashi, Kazuya Nishimura +5 more
published similarity 0.69
Abstract · 7 pages, 5 figures. Accepted for publication at IEEE EMBS BHI 2026

Retinal foundation models have demonstrated strong performance across a wide range of ophthalmic tasks, yet relatively little is known about how disease-related information is represented within their learned embeddings and whether this distribution influences downstream prediction performance. In this work, we use Age-related Macular Degeneration (AMD) as a case study to investigate how disease severity is represented within the embedding space of RET-CLIP, a retinal vision-language foundation model, using bilateral fundus images from the AREDS dataset, benchmarked against non-retinal encoders. We first characterize the distribution of AMD severity within the learned representation space and then evaluate how this structure influences classification performance across three clinically relevant task formulations of increasing granularity. Finally, we analyze the most influential latent dimensions combining feature attribution, statistical phenotype association, and qualitative image analyses. Our results show that RET-CLIP represents AMD severity as a continuous spectrum rather than a set of discrete categories, with healthy and advanced disease stages exhibiting clearer separation than intermediate ones. This organization was reflected in downstream performance, which decreased from a balanced accuracy of 0.844 and 0.814 for binary AMD and Late-AMD detection respectively to 0.446 for six-class grading. Furthermore, the most influential latent dimensions were statistically associated to distinct retinal phenotypes, including drusen accumulation, pigmentary alterations, and advanced retinal lesions. These findings suggest that the structure of retinal foundation-model representations influences the difficulty of clinically relevant prediction tasks and demonstrate the potential of representation-level analysis for understanding disease organization and improving interpretability.

0.18

What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization

cs.CV cs.AI ·2026-09-21 ·Niklas Bubeck, Yundi Zhang, Vasiliki Sideri-Lampretsa, Julian McGinnis, Jiancheng Yang +2 more
similarity 0.67
Abstract

Latent diffusion models now dominate medical image generation, and every such pipeline rests on a \emph{tokenizer} that compresses images into the latent codes for image generation to operate on. Thereby, the tokenizer choice bounds every downstream task from reconstruction fidelity and generation quality to the representations available for downstream analysis. Yet, medical imaging pipelines routinely utilize tokenizers from natural imaging on the hypothesis that their behavior carries over. However, this is an assumption never tested in the medical imaging regime, where datasets are orders of magnitude smaller and images exhibit far lower inter-sample variance. We present a systematic evaluation of medical image tokenizers evaluating thirty configurations across ten model families on twelve datasets at three compression factors, spanning reconstruction, generation, latent geometry, downstream classification, and memorization. We find that (1) performance on image reconstruction and generation strongly correlate, unlike prior reports on natural images; (2) modern tokenizers use nearly all of their codebook entries, but still leave most of the latent space unused; (3) training-set memorization is mild and is further suppressed by stronger latent space compression; and (4) discrete quantization can largely preserve downstream classification, with lookup-free schemes being the main exception.

0.24

CAMEL: Cross-Architecture Masked Embedding Learning for Ultra-Lightweight Retinal Foundation Models

cs.LG cs.AIcs.CV ·2026-10-29 ·Andre Guarnier De Mitri, Giovani Decico Lucafo, Miller Matheus Lima Anacleto Rocha, Diego Furtado Silva
similarity 0.66
Abstract · 8 pages, 6 figures, pre-print

Recent progress in deep learning has substantially advanced automated grading of ophthalmologic pathologies, supported by large annotated datasets and improved representation learning. However, a significant gap remains between state-of-the-art performance and clinical scalability. In particular, foundation models have reshaped image-based medical screening and diagnosis. These large-scale architectures trained on extensive datasets learn rich, generalizable feature representations that can be adapted to a wide range of downstream tasks. In ophthalmology, RETFound, one of the most relevant foundation models for retinal images, performs strongly across classification, segmentation, and risk prediction tasks. Despite their effectiveness, foundation models are often computationally prohibitive, hindering their deployment in real-world clinical environments where hardware constraints and latency are critical. To bridge this gap, we introduce CAMEL (Cross-Architecture Masked Embedding Learning), a novel cross-architecture distillation framework that uses an asymmetric masking strategy during embedding learning to effectively transfer knowledge from massive teacher foundation models to compact, lightweight student models. Our experimental evaluation across four datasets shows that CAMEL preserves accuracy while drastically reducing computational complexity, enabling lightweight yet highly accurate models suitable for resource-limited clinical settings.

0.21

Where to Predict in Retinal OCT? Anatomy-Guided Target Selection for I-JEPA

cs.CV ·2026-11-12 ·Yunchu Feng, Wai Tak Lau, Kaveri A. Thakoor
published similarity 0.65
Abstract · 18 pages. Accepted as a poster at the GenAI for Health Workshop at NeurIPS 2026 (non-archival)

Self-supervised learning aims to learn meaningful representations from unlabeled data, potentially allowing models to capture subtle anatomical variations across patients and reduce the amount of labeled data needed. Image-based Joint-Embedding Predictive Architecture (I-JEPA) learns by predicting target-region representations from visible context rather than reconstructing pixels. However, its uniform target placement does not account for the anatomical structure of retinal OCT, where retinal tissue occupies only part of each B-scan. We investigate anatomy-guided target selection using image intensity and predicted retinal segmentation while retaining the I-JEPA architecture and latent prediction objective. We compare anatomy-guided rectangular targets, anatomy-shaped targets and coverage-constrained placement with uniform masking, and evaluate the learned representations using linear probing for glaucoma classification on FairVision. We found that rectangular targets guided by image intensity achieved the highest linear-probe AUC in our comparisons. We show that anatomically meaningful target selection is promising for retinal OCT representation learning, while greater anatomical specificity did not consistently improve performance.

0.35

TheseusSyn: Unified Medical Image Synthesis via Context-Conditioned Masked Autoregression

eess.IV ·2026-10-28 ·Shibo Li, Venet Osmani
similarity 0.64
Abstract · 11 pages, 2 figures

High-fidelity medical image synthesis is promising for data-efficient learning and privacy-preserving model development. However, most existing approaches rely on symbolic or label-based conditioning, which constrains generative diversity and limits case-specific contextual fidelity. Inspired by the Ship of Theseus paradox, we propose TheseusSyn, a novel context-conditioned synthesis paradigm that generates images through iterative content replacement. Our framework employs a multi-codebook discrete tokenizer to convert medical images into sequences of token indices, extracted with a global representation token that serves as a contextual constraint. A masked autoregressive generative model then iteratively masks and predicts tokens conditioned on both local neighborhoods and the global context. By progressively replacing image content until the entire token sequence is renewed, TheseusSyn produces synthetic samples that preserve anatomical coherence and semantic consistency while introducing controlled diversity. We evaluate the proposed framework on multiple datasets from MedMNIST by constructing synthetic datasets with varying masking strategies and training ResNet-18 classifiers exclusively on either real or synthetic data. Experimental results demonstrate that classifiers trained on TheseusSyn-generated images achieve competitive performance on official test sets, indicating that the synthesized data preserve diagnostically relevant features and semantic consistency. Systematic analysis of masking strategies reveals their impact on structural fidelity and downstream classification accuracy. These results suggest that context-conditioned masked autoregression provides an effective and generalizable solution for label-free medical image synthesis, offering a scalable alternative to conventional data augmentation and class-conditioned generative models in medical imaging applications.

0.23

Learning Pathology-Sensitive Representations for Unsupervised Retinal OCT Anomaly Detection

cs.CV cs.LG ·2026-10-23 ·Tania Haghighi, Sina Gholami, Behafarin Emam, Jennifer I Lim, Theodore Leng +3 more
similarity 0.64
Abstract

Deep learning approaches for retinal optical coherence tomography (OCT) predominantly rely on predefined disease categories, limiting their ability to detect rare or previously unseen abnormalities. While visual foundation models provide transferable representations, their effectiveness in identifying pathological deviations from normal retinal anatomy remains insufficiently understood. In this work, we introduce an unsupervised framework for retinal OCT anomaly detection and localization that adapts pretrained visual representations to normal retinal anatomy through anatomy-aware reconstruction and contrastive triplet learning guided by synthetic structural perturbations, without requiring disease-specific annotations. We systematically investigate the influence of pretraining strategies, anatomy-aware representation adaptation, and anomaly scoring mechanisms on the detection and localization of unseen retinal abnormalities. Our findings demonstrate that adapting pretrained representations to normal retinal structure improves anomaly sensitivity, while pixel-space reconstruction errors and feature-space deviations provide complementary signals for identifying pathological changes. These findings underscore the importance of anatomy-aware representation learning and offer insights into developing generalizable retinal anomaly detection frameworks beyond predefined disease categories.

0.30

Biological Congruence: Evaluating Microscopy Generation Across Representations

cs.CV ·2026-11-01 ·Duway Nicolas Lesmes-Leon, Shumpei Takezaki, Fabian Schmeisser, Ryoma Bise, Gillian Lovell +3 more
published similarity 0.64
Abstract · Accepted at WACV2027

Generative models increasingly produce realistic microscopy images, yet conventional evaluation primarily characterizes visual fidelity and distributional coverage rather than the biological information represented by synthetic data. We introduce biological congruence as an evaluation principle describing whether biologically relevant information forms a coherent description across representations and remains consistent with its empirical biological context. To study this principle, we develop a text-conditioned diffusion framework that jointly generates microscopy images and instance masks, exposing the same generated biological state through conditioning, visual appearance, and instance structure. Across prompt--mask, prompt--image, image--mask, and population-level analyses, we find strongly feature-dependent behavior: density and confluency are consistently preserved, morphology exhibits recurrent disagreement, while nearest-neighbor distance reveals that strong image--mask agreement can coexist with weaker conditioning and population fidelity. Stress testing increasingly unsupported phenotype combinations further shows that target controllability can increase while untargeted phenotypes reorganize and the complete generated state moves toward configurations better supported by real data. These findings demonstrate that generative quality, controllability, cross-representation agreement, and population fidelity capture complementary properties, motivating biological congruence as a relational perspective for evaluating scientific generative models.

0.37

Disentangling Co-Occurring Retinal Pathologies with Saliency-Guided Sparse Expert Routing

cs.CV cs.LGeess.IVeess.SP ·2026-08-10 ·Nagur Shareef Shaik, Jeongwoo Park, Yeong-Jin Kim, Jaeuk Jung, Hyunjung Oh +1 more
publishedbenchmark similarity 0.63
Abstract · Accepted at 2026 IEEE International Workshop on Machine Learning for Signal Processing

Retinal fundus images frequently exhibit multiple co-occurring pathologies, yet standard deep learning classifiers apply static, identical computation to every image regardless of the underlying disease distribution. We propose a novel architecture that resolves this via sparse conditional computation, pairing a Guided Context Gating (GCG) spatial attention front-end with a sparsely-routed Mixture-of-Experts (MoE) block operating over feature tokens. Crucially, this routing yields an interpretable, data-driven decomposition. Expert allocation is significantly disease-dependent (p < 0.001), with the healthy Normal state and morphologically distinct pathologies (e.g., ERM, AMD) isolating to dedicated experts. On a five-class, patient-disjoint 5-fold cross-validation benchmark, our model achieves 0.912 +/- 0.008 macro AUC and 0.653 +/- 0.014 macro F1. Furthermore, Grad-CAM++ and post-MoE t-SNE visualizations confirm that expert routing aligns with localized lesions and geometrically maps co-occurring cases between their constituent clusters, positioning sparse MoE as an interpretable approach to multi-disease retinal screening.

0.20