CS Paper Radar

Sparse Autoencoders for Expert-Assisted Interpretation of Histopathology Foundation Model Embeddings

Šimon Šmída, Wanda Benesova

cs.CV ·submitted 2026-10-29 19:26 UTC · v1

Abstract

Histopathology foundation models learn rich embedding representations, but their internal structure remains difficult to inspect. Concept-level interpretability is particularly relevant in computational pathology because pathologists reason through recurring morphological and tissue-level patterns, including tissue architecture, cellular morphology, staining variation, and artifacts. In this work, we investigate whether sparse autoencoders (SAEs) can decompose embeddings from histopathology foundation models into sparse latent features suitable for expert-assisted interpretation. We train vanilla and Batch-TopK SAE variants on NCT-CRC embeddings, using CONCH as the primary foundation model and UNI2-h as an additional robustness evaluation, and characterize the resulting latent features through reconstruction fidelity, sparsity, coherence, label enrichment, and downstream class-signal preservation. The learned sparse representations preserve most downstream tissue-class information, and their latent features retrieve more coherent and label-concentrated patch groups than raw embedding dimensions or PCA components. Discussions with pathology experts further motivated a WSI-level latent visualization workflow that places feature activations into larger tissue context. Overall, the additional UNI2-h evaluation suggests that the observed behavior is not specific to CONCH, supporting the use of sparse autoencoders for expert-assisted analysis of histopathology foundation-model embeddings, while emphasizing that latent features should be treated as candidate concepts requiring pathological assessment rather than automatically validated histopathological concepts.

Closest papers

Nearest neighbours by meaning across the whole index.

EXPOSE: Explainable and Domain-Robust Embeddings from Pathology Vision Foundation Models using Sparse Autoencoders

cs.CV cs.LG ·2026-08-28 ·Anja Witte, Maximilian Lennartz, Jan Baumbach, Guido Sauter, Stefan Bonn +2 more
code/data similarity 0.74
Abstract

Vision Foundation Models (VFMs) are widely used in computational pathology but remain sensitive to domain shifts arising from variations in staining, tissue preparation, and scanner hardware. A key limitation is that VFM embeddings entangle biological with domain-specific information, hindering cross-domain generalization. We propose Explainable Probing of Cross-Domain Sparse Embeddings (EXPOSE), a framework that uses Sparse Autoencoders (SAEs) as an explainable bottleneck to identify and suppress domain-specific components in VFM embeddings. We train a sparse representation of VFM features, use a linear classifier to identify domain-specific latent dimensions, and mask these features prior to downstream relapse prediction without retraining the backbone model. Experiments on a large prostate cancer dataset with multiple acquisition domains show that SAE features capture both domain- and task-specific information, which are partially disentangled in the latent space. Removing domain-specific features improves cross-domain performance and increases embedding robustness as measured by the Domain Robustness Index (DoRI). Code is available at https://github.com/imsb-uke/expose .

0.25

HERO: Histology Encoder for Robust Representation in Oncology

cs.CV ·2026-09-28 ·Zhi Li, Eghbal Amidi, Yating Cheng, Tyson Dawson, Gorkem Can Ates +9 more
benchmarkcode/data similarity 0.69
Abstract · 22 pages, 4 figures, 13 tables; author information updated; scientific content unchanged

Foundation models trained on large pathology image corpora now provide strong, transferable representations for computational pathology. Over the past few years a series of such models has been released, each trained on more slides than the last; on standard classification and segmentation benchmarks, the leading models are now separated by small margins. In clinical use, however, the foundation model is applied to images from hospitals, scanners, and staining protocols outside its training data. Encoders generally embed these acquisition factors alongside biological information, which may introduce downstream errors and hinder safe clinical adoption. A pathology foundation model should therefore be robust to acquisition shift without giving up representation quality, yet robustness is seldom the axis along which models are compared. In this report, we introduce HERO (Histology Encoder for Robust Representation in Oncology), a ViT-G/14 pathology foundation model trained with the DINO and iBOT objectives and refined with high-resolution Gram anchoring on a morphology-balanced corpus of 500 million tiles from approximately 575,000 clinical whole-slide images. Across the evaluated public benchmarks, HERO shows the strongest robustness to center, scanner, and stain variation among the compared state-of-the-art foundation models, performs comparably on tile-level classification, segmentation, and gene-expression prediction, ranks first on average across 39 evaluated slide-level clinical tasks, and, under an equal-weighted framework-level analysis, has the best average rank across the six benchmark frameworks.

0.22

MOSAIC: Towards a Universal Latent Space for Computational Pathology Foundation Models

cs.CV cs.AI ·2026-10-19 ·Dibaloke Chanda, Praveen Madiraju
publishedcode/data similarity 0.69
Abstract · Accepted at WACV 2027

Pathology foundation models (PFMs) differ substantially in architecture, pretraining objective, training data and representation dimensionality, yet it remains unclear whether they learn fundamentally different feature spaces or alternative coordinate systems of a shared underlying representation. We present MOSAIC, a systematic study of latent-space similarity, alignment and interoperability across $12$ patch-level and $6$ slide-level PFMs. We first quantify representational agreement using complementary similarity measures and analyze how architecture, pretraining objective, network depth and magnification influence latent geometry. We then evaluate multiple alignment strategies for mapping independently learned representations into a common latent space. The resulting representations are evaluated through cross-model agreement, retrieval, geometry preservation, and downstream multiple-instance learning across molecular and morphological prediction tasks. MOSAIC provides a unified framework for studying whether heterogeneous PFMs can move beyond representational similarity toward a shared and functionally transferable pathology representation space. Our code, benchmarks and additional results are available at https://researchsubmissions66.github.io/MOSAIC/.

0.42

Concept-based explanation of gene expression prediction from H&E images

cs.CV ·2026-08-17 ·Amos Muench, Jonathan Thielmann, Reduan Achtibat, Maximilian Dreyer, Philip Bischoff +7 more
similarity 0.67
Abstract

Recent advances in pathology foundation models have enabled accurate prediction of spatial transcriptomics (ST) from routine H&E images. However, existing explainability methods for vision transformer (ViT)-based models are largely limited to local heatmaps and do not reveal how morphological concepts contribute to ST predictions. Here, we introduce an explainable framework that combines relevance propagation and concept discovery to link transcriptional programs to tissue morphology. We developed a ViT-based framework for virtual ST from H&E images that combines ViT-aware layer-wise relevance propagation with relaxed archetypal TopK sparse autoencoder-based concept discovery. This approach provides both local explanations and global insights into the morphological patterns associated with transcriptional programs. We applied the framework to colorectal cancer ST data from the HEST-1k cohort and evaluated its generalizability in TCGA COAD. Our architecture accurately predicts clinically relevant ST signatures and accompanying molecular phenotypes. Measured and predicted gene expression profiles reveal substantial spatial heterogeneity of the colorectal cancer subtypes iCMS2 and iCMS3 across a large number of samples. Spatially resolved and aggregated iCMS classification achieve weighted F1 scores of 0.872 and 0.819 (0.770 in TCGA COAD), respectively, and both stratify patient outcome. Beyond prediction, our framework establishes a relevance-based concept atlas linking molecular phenotypes to histopathological representations. Comparison of activation- with relevance-derived concepts demonstrates that relevances provide a more direct link between tissue morphology and downstream predictions. We establish a general strategy for concept-based explanation of spatial prediction, and our framework is readily applicable to a broad range of ViT-based pathology models.

0.07

Label-Free Coreset Selection with Foundation Models for Efficient Annotation in Computational Pathology

cs.CV ·2026-10-05 ·Tuo Yin, Jennifer Dhont
code/data similarity 0.67
Abstract · 32 pages, 7 figures

Computational pathology has the potential to improve clinical outcomes through a demonstrated increase in diagnostic and prognostic accuracy. However, the development and validation of deep learning algorithms still require annotated data, a costly procedure involving expert pathologists who already face critical workforce shortages. Existing coreset selection methods to optimize annotation efforts currently all rely on hyperparameters tuned on natural-image benchmarks that do not transfer to histopathology and are cumbersome to use in clinical practice. In this study, we present GCcore, a novel label-free coreset selection method that embeds every image of a dataset with any pathology foundation model and greedily selects the samples that collectively maximize the global coverage of the embedding space. The proposed method provides a lower-bound guarantee on the global coverage of the returned coreset for any coreset size, while being completely hyperparameter-free and deterministic. We demonstrate GCcore's superior performance over 14 baselines including state-of-the-art methods across 10 tasks and datasets spanning whole slide image classification, tile classification, and tissue segmentation, where it ranks first on six and within the top three on nine, while also demonstrating how existing methods can shift by up to five rank positions depending on their hyperparameter settings. Code is publicly available at https://github.com/OncoAI-ULBHUB/GCcore.

0.23

CMCNet: Aligning Ultrasound Image Embeddings with Textual TI-RADS Representations for Fine-Grained Thyroid Classification

cs.CV cs.AI ·2026-08-14 ·Bingxin Yu, Xueli Wang, Jerry Zhou, Wenyan Wang, Li Wen +4 more
datasetcode/data similarity 0.66
Abstract

Ultrasound is the primary imaging modality for assessing thyroid nodules, and the ACR TI-RADS framework standardizes diagnosis through five ultrasound feature categories that are aggregated into five risk levels (TR1-TR5). Although widely adopted in clinical practice, most deep learning approaches focus on binary malignancy classification, while multi-class prediction and explicit utilization of feature-level supervision remain underexplored, largely due to limited annotated data. In this study, we introduce the STN dataset of 600 thyroid nodules with paired transverse and longitudinal ultrasound images, bounding box annotations, and complete labels for all five TI-RADS feature categories. Following the clinical decision process, we investigate how structured feature information can guide representation learning during training while requiring only images at inference. We demonstrate that text embeddings derived from standardized feature descriptions form a stable surrogate representation for TI-RADS risk levels. Based on this observation, we propose CMCNet, which aligns image embeddings to fixed textual embeddings via a Center-Margin Contrastive Loss that simultaneously promotes intra-class compactness and inter-class separation. Experimental results show that this embedding alignment strategy is more data-efficient and robust than direct multitask learning, and consistently outperforms InfoNCE, center loss, a strong multitask baseline, and a VQA-style multimodal model, particularly in imbalanced settings. The dataset is freely available at doi: 10.5281/zenodo.19125693 and the source code is available at: https://www.healthinformaticslab.org/supp/.

0.20

Glyph: An Interpretability Engine for AI-Driven Scientific Discovery in Clinical Data

cs.AI ·2026-11-06 ·Robbie Holland, Ashwin Kumar, Mateus A. Esmeraldo, David Le, Andrea Fisher +17 more
code/data similarity 0.66
Abstract

Automated scientific discovery has the potential to advance understanding of disease mechanisms and accelerate identification of early disease markers. Manual data annotation is infeasible at population scale, motivating methods that automatically extract pre-defined phenotypes in multimodal and unstructured clinical data. Such pre-defined feature sets determine and limit which associations these approaches can discover. This limitation extends to agentic AI systems that use these tools to query population-scale data. In contrast, foundation models (FMs) trained on large volumes of clinical data can perform tasks beyond their training objectives. This suggests their latent embeddings may represent novel or unexplored features that could yield insights into disease mechanisms, yet such latent features remain difficult to interpret. To address this challenge, we introduce Glyph, an automated multistage framework that extracts features from multimodal and unstructured clinical data using FMs and sparse autoencoders, interprets them as coded clinical concepts, and evaluates their value for clinical scientific discovery. We applied Glyph to abdominal CT volumes, CT radiology reports and laboratory measurements from 41,628 patients across different institutions. We identified 397 predictive concepts supported by 4,463 Bonferroni-significant associations. Moreover, a majority of discovered associations were automatically reproducible in held-out cohorts across different sites. Glyph concepts provided complementary associations to literature-derived markers, leading to gains in predictive power. We envision that interpretability can extend beyond a mechanistic understanding of neural networks to connecting implicitly learned features with clinical knowledge and establishing their significance for patient outcomes. We release Glyph at https://github.com/StanfordMIMI/Glyph.

0.29

Sparse concept attribution for histomorphological hypothesis generation from whole-slide classifiers

eess.IV q-bio.QM ·2026-09-02 ·Tristan Lazard, Kenza Bouzid, Julius Hense, Shruthi Bannur, Daniel Coelho de Castro +4 more
benchmark similarity 0.65
Abstract

Histology images contain rich morphological information and can provide insights into pathological processes. However, deriving hypotheses relating morphological phenotypes to clinical attributes is bottlenecked by a manual image interpretation step. Here, we demonstrate that this process can be automated through interpretable deep learning. We present SCOPE, a method to interpret slide-level classifiers by combining pathology-specific vision--language models with sparse concept attribution onto a generalist histomorphological concept bank. To measure whether such explanations recover known morphology, we introduce MorphoRecoveryBench, a benchmark of seven tasks with pathologist-curated reference descriptions. On this benchmark, dense concept attribution is indistinguishable from a random baseline, whereas sparse attribution recovers substantial known morphology; decomposing the pooled slide embedding reaches similar explanation correctness at a fraction of the computational cost. Post-hoc interpretation of whole-slide classifiers can thus generate morphological hypotheses at scale, for expert validation.

0.35