CS Paper Radar

Multi-Level Evidence Aggregation for Robust Facial Phenotype Retrieval in Rare Genetic Disorder Prioritization

Alexander Hustinx, Carolin Kaffiné, Behnam Javanmardi, Tzung-Chien Hsieh, Peter Krawitz

cs.CV cs.IR ·submitted 2026-08-11 15:12 UTC · v1

Abstract

AI-assisted facial phenotyping supports rare genetic disorder prioritization by retrieving visually similar diagnosed cases from facial image reference databases such as the GestaltMatcher Database (GMDB). Existing GestaltMatcher-based retrieval frameworks compare each test image with individual gallery images in a facial phenotype embedding space. However, this pointwise formulation does not fully exploit available evidence, because patients may have multiple images and disorders may be represented by multiple diagnosed gallery patients. We propose an inference-time multi-level evidence aggregation framework that improves facial phenotype retrieval without modifying the underlying GestaltMatcher-Arc encoder. The framework combines embedding-level patient aggregation of multiple images from the same individual, patient-weighted disorder centroids, and hybrid individual-centroid scoring to integrate test-patient observations, disorder-level gallery evidence, and local nearest-neighbor evidence. We evaluated the approach on GMDB v1.1.4 across disorders represented during training (GMDB-Freq), unseen disorders (GMDB-Rare), and multi-image patient subsets, using a unified gallery containing both GMDB-Freq and GMDB-Rare disorders. Multi-level evidence aggregation improved mean per-disorder top-$N$ retrieval accuracy across all evaluation subsets. Top-1 accuracy increased from 38.52% to 48.82% on GMDB-Freq and from 19.38% to 23.79% on GMDB-Rare. On multi-image subsets, top-1 accuracy increased from 46.12% to 60.94% on GMDB-Multi-Freq and from 18.54% to 26.71% on GMDB-Multi-Rare. These findings show that inference-time aggregation can improve next-generation facial phenotype retrieval without retraining the encoder, supporting a shift from isolated single-image matching toward multi-level aggregation of patient and disorder evidence for rare-disorder prioritization.

Comment
including supplementary notes: 32 pages, 16 figures. Preprint submitted to journal for peer-review

Closest papers

Nearest neighbours by meaning across the whole index.

Synthetic Image Oversampling for Rare-Class Representation Learning in Deep Facial Phenotyping

cs.CV ·2026-10-16 ·Miles Liao, Zhanliang Wang, Hongzhuo Chen, Kai Wang
similarity 0.73
Abstract · BIBM 2026 ML4BMI Workshop

Recent deep facial phenotyping methods can support the recognition of genetic syndromes, but patient images are often scarce and unevenly distributed across disorders. Synthetic facial images offer a way to supplement the training data without collecting additional patient photographs. Their usefulness, however, depends on whether they improve recognition of real patients: generated images may introduce artifacts or repeat features already represented in the training set. It is therefore necessary to examine both how much synthetic augmentation to use and which disorders benefit. We investigate disease-specific synthetic oversampling using images from the Phenotypic Disease Image Database to augment a subset of the GestaltMatcher Database. We compare increasing augmentation levels in an image-based model, GestaltMatcher-Arc, and a multimodal model, GestaltMML, while keeping the real training and patient-disjoint evaluation partitions fixed. Repeated training runs assess overall and disease-specific accuracy, and an exploratory embedding analysis examines changes in class compactness and separation. Augmentation improves mean recognition accuracy in both training setups, although gains remain modest in GestaltMatcher-Arc and continue across the tested levels in GestaltMML. These aggregate improvements are accompanied by uneven disease-level responses, including reduced accuracy for some disorders. In the embedding analysis, disorders with fewer real training images tend to gain more class separation, but geometric changes do not consistently track classification performance. Together, the findings support synthetic oversampling for limited-data facial phenotyping and emphasize the need to assess its amount and benefit for individual disorders within each training setup.

0.15

FaceKit: a Toolkit for Interpretable Facial Phenotyping, Synthetic Image Generation and Privacy Analysis in Rare Diseases

cs.CV q-bio.QM ·2026-10-07 ·Hongzhuo Chen, Zhanliang Wang, Florent Pollet, Mian Umair Ahsan, Joshua Bie +7 more
hot direction similarity 0.66
Abstract

Many rare genetic diseases are associated with recognizable craniofacial features. However, traditional approaches for describing facial morphology rely largely on qualitative clinical observation and free-text descriptions, which are often subjective, non-standardized, and difficult to reproduce across observers and institutions. Although the Human Phenotype Ontology (HPO) provides controlled terms for describing facial features, these terms are typically categorical rather than quantitative and may vary depending on examiner experience and interpretation. Here, we present FaceKit, a computational framework for quantitative facial phenotyping from frontal facial photographs. FaceKit extracts standardized measurements of facial landmarks and derived 120 morphological features, then reports feature-level z-scores representing deviation from population reference distributions. The reference distributions are built from the FairFace dataset spanning diverse ancestral groups. We evaluated FaceKit on a curated subset of the GestaltMatcher Database covering 50 rare-disease cohorts. In addition to quantitative facial analysis, FaceKit includes synthetic facial image generation to support rare disease model development and data augmentation. We also performed privacy evaluation to assess whether synthetic images reveal identifiable information from real patient photographs and could compromise patient privacy. Across disease case studies, FaceKit-derived quantitative measurements captured known facial features associated with rare genetic disorders and provided objective support for clinical phenotyping. Together, these results establish FaceKit as a useful tool for quantitative phenotyping, and has the potential to improve rare disease diagnosis, support genotype-phenotype studies, and enable more reproducible clinical characterization across diverse patient populations.

0.31

ProtoGestalt: Interpretable Prototypes for Facial Phenotyping and Diagnosis of Rare Genetic Disorders

cs.CV ·2026-11-05 ·Ranga Baminiwatte, Alexander Hustinx, Fabio Hellmann, Aaron J Masino
similarity 0.63
Abstract

Rare genetic disorders are difficult to diagnose due to their diversity and overlapping clinical features. Facial morphology can provide diagnostic clues, yet most diagnostic models offer limited interpretability and are not designed to support broad screening for undiagnosed disease. We present ProtoGestalt, an interpretable prototype-based framework for facial phenotyping, rare-disease screening, and syndrome identification. The method extends prototype learning by constraining learned visual features to select facial regions, allowing predictions to be explained through localized and additive evidence. We assess ProtoGestalt's ability to distinguish syndromic from unaffected faces on multiple control datasets and class prevalence values. We further evaluate it's diagnostic performance on syndromes observed during training. The learned prototype-based representations provide significant separation between syndromic and case-matched control images and retain useful performance as the proportion of controls increases. The framework also demonstrates the ability to differentiate among syndromes. Although reasonably accurate specialized models for facial classification of rare disorders already exist, ProtoGestalt offers an interpretable alternative that links predictions to anatomically meaningful facial regions, potentially supporting clinical decision-making and the discovery of phenotype-gene associations.

0.17

MedHEB: Benchmarking Medical Embeddings Across Heterogeneous Clinical Evidence

cs.CV ·2026-11-02 ·Yingshu Li, Shaoyang Zhou, Zhanyu Wang, Yunyi Liu, Xinyu Liang +4 more
benchmark similarity 0.59
Abstract

Clinical evidence is inherently relational: medical images are interpreted with prior studies, cross-modal examinations, reports, and localized findings, and retrieving relevant evidence is therefore central to clinical interpretation. Recent foundation models increasingly encode these heterogeneous sources into shared embedding spaces, yet existing embedding evaluation often focuses on isolated matches, leaving unclear whether current models retrieve relevant evidence across changes in modality, appearance, and granularity. We introduce MedHEB, a unified benchmark with 107 retrieval tasks from 54 datasets, including both in-distribution and held-out-source evaluation, across three axes: evidence form (2D images, 3D volumes, text), pairing type (image--image, image--text, text--text, and modality-bridging visual matching), and relevance granularity (global, question-conditioned, region-level). Under a single ranking protocol, MedHEB varies clinically relevant evidence definitions to test whether embedding proximity remains meaningful across heterogeneous evidence, including modality-bridging visual retrieval and region-level text-to-region retrieval. Across representative models, performance is uneven across retrieval families: models strong on conventional image--text or text--text retrieval are less reliable on modality-bridging and region-level tasks. Current embedding spaces capture some clinical relationships well but do not yet provide balanced retrieval across heterogeneous evidence. We further provide MedEmb, an MLLM-based reference embedding model trained on the MedHEB ID split, as a reproducible baseline for unified 2D--3D--text medical retrieval. We release MedHEB with preprocessing scripts, evaluation code, access manifests, and MedEmb to support research on clinically meaningful medical embeddings.

0.23

LENS-GRF: Permutation-Invariant Lesion Evidence Network with Gated Residual Fusion for Acne Severity Grading and Multi-Rater Clinical Oracle Analysis

cs.CV cs.AI ·2026-09-25 ·Muhammad Muhtasim Shahriar, M. F. Mridha, Md. Rakibul Islam
similarity 0.58
Abstract · Under Review at Results in Engineering (Elsevier)

Localized lesion evidence may improve automated acne severity grading, but detector errors and differences in clinical grading criteria can limit generalization. We evaluated LENS-GRF, a lesion evidence network with permutation-invariant aggregation and gated residual fusion that combines a global Vision Transformer with lesion-region embeddings derived from YOLO11s proposals. Fully automated four-class grading was evaluated on 292 held-out ACNE04 images and, without external adaptation, on 200 PLSBRACNE01 participants represented by 600 facial views. Annotation-assisted inference used registered lesion boxes from three dermatologists in a matched subset of 148 participants. Performance measures included accuracy, macro-averaged precision, sensitivity, and one-versus-rest specificity. Paired comparisons used exact McNemar tests with Holm adjustment. On ACNE04, automated LENS-GRF achieved 80.82% accuracy, 78.80% macro precision, 78.92% sensitivity, and 92.78% specificity. With supplied lesion boxes, internal accuracy averaged 95.89% across three seeds. On the full external cohort, automated LENS-GRF achieved 35.00% accuracy, compared with 42.50% for an independently trained global baseline. External macro precision, sensitivity, and specificity for LENS-GRF were 50.52%, 34.31%, and 78.19%, respectively. In the matched subset, accuracy using Doctor A's lesion annotations was 47.97%, compared with 35.14% for automated inference (unadjusted $p = 0.0094$; Holm-adjusted $p = 0.0846$). Strong internal performance with supplied lesion annotations did not translate into reliable automated grading on the external cohort. More reliable lesion detection, modelling that accounts for differences in grading criteria, and independent validation are needed before clinical use.

0.31

NeuroCBIR: A Fast and Accurate Image Retrieval System for Whole-Brain and Region-Specific MRI

cs.CV cs.LGq-bio.NC ·2026-10-05 ·Felix Nieto-del-Amor, Jingru Fu, J. -Sebastian Muehlboeck, Eric Westman, Daniel Ferreira +1 more
code/data similarity 0.57
Abstract · Neuroimaging, Content-Based Image Retrieval, MRI, Zero-Shot Learning

Content-based image retrieval (CBIR) in neuroimaging enables the identification of structurally similar brain scans, supporting diagnosis, prognosis, and treatment planning; however, existing methods are often limited to small datasets, single brain regions, or coarse class labels, thereby restricting their clinical utility and generalizability. Here, we present NeuroCBIR, a framework for fast and flexible retrieval of both whole-brain and region-specific 3D T1w MRI scans. A total of 103 cortical and subcortical regions are extracted to enable both whole-brain and region-level queries. NeuroCBIR leverages latent representations learned by a variational autoencoder (VAE) combined with contrastive learning, producing scan-specific embeddings that capture anatomical patterns. These embeddings were evaluated for subject re-identification, zero-shot age prediction, and zero-shot multi-class pathology stratification. Re-identification performance was high across both whole-brain and brain-region levels (mean average precision across the top-5 retrieved images (mAP@5) >= 98.4%), with robust generalization across datasets and acquisition conditions. While NeuroCBIR is not trained for age prediction or pathology stratification, zero-shot evaluations for these two tasks demonstrate that the embeddings encode meaningful information for downstream tasks. Embedding extraction on a 4-core CPU required approximately 18.7 s per scan, whereas similarity search was effectively instantaneous (less than 0.01 s). NeuroCBIR is publicly available for brain MRI with more than 26,000 precomputed T1w MRI embeddings. It supports reproducible research, region-specific flexibility, and clinically meaningful personalized diagnostic support. The software is available at https://github.com/minnelab/NeuroCBIR.

0.26

GAMF-Net: Grouped Availability-Aware Multimodal Fusion for Smartphone Skin Lesion Classification

cs.CV ·2026-10-20 ·Yue Zhuo, Jingjia Yan, Fangyu Liu, Lingjun Liu, Zongjun Chen +2 more
published similarity 0.57
Abstract · 14 pages, 4 figures. Accepted at MMM 2027

Smartphone clinical images provide a low-cost data source for computer-aided skin lesion classification in resource-constrained settings, but images alone cannot fully capture patients' medical histories and lesion symptoms. To address the lack of structured organization among heterogeneous clinical variables, the ambiguity between observed and imputed values, and the risk that clinical information may interfere with visual evidence, we propose GAMF-Net, a grouped availability-aware multimodal fusion network. Ten clinical variables are divided into three semantic groups---demographics, history-related risk, and lesion presentation---and each group jointly encodes field values and field-level availability masks. The image branch produces the base classification logits, while each clinical group generates image-conditioned class-level residual evidence. Only available groups participate in availability-normalized equal-weight fusion; when all clinical information is unavailable, the model reduces exactly to the image-only pathway. We evaluate the method on PAD-UFES-20 under a patient-disjoint five-fold protocol with separate training, validation, and test partitions. GAMF-Net achieves an Accuracy of $74.17\pm2.85\%$, a balanced accuracy (BACC) of $70.11\pm2.11\%$, a Macro-F1 of $68.85\pm0.85\%$, and a Macro-AUROC of $93.42\pm1.20\%$, outperforming all image-only and multimodal baselines reimplemented under a unified protocol. Compared with the strongest baseline, MetaBlock, the corresponding improvements are 2.37, 2.59, 4.14, and 1.02 percentage points, respectively. Ablation results show that clinical information substantially improves class-balanced performance, constrained equal-weight fusion is more effective than dynamic fusion, and the lesion-presentation group provides the most informative clinical complement.

0.22

OraRefer: Segmentation-Guided Multi-Photograph Fusion for Oral Cancer Screening with Guaranteed Patient-Level Sensitivity

cs.CV eess.IV ·2026-11-05 ·Rupam Mukherjee, Mehakdeep Singh, Shirin Dasgupta, Subhamoy Mandal
similarity 0.56
Abstract

Oral cancer and the oral potentially malignant disorders (OPMD) that can precede it are visible on the oral mucosa, which makes smartphone photographs a promising tool for screening. Deep learning models for this task classify individual photographs. However, screening refers patients, not photographs: a patient is usually photographed several times, and the referral decision must miss no more than a set share of patients with disease. We propose OraRefer, a patient-level screening model that reads each photograph with a segmentation-guided foundation model that conditions its class decision on the predicted lesion, combines all photographs of a patient with risk factors, and refers through a threshold with a finite-sample sensitivity guarantee. The model localizes the mouth itself, so inference needs no annotation. On 3000 photographs of 714 patients, patient-level fusion reaches an AUROC of 0.919 and outperforms attention-based multiple-instance learning. At a 90% sensitivity target, OraRefer detects 95% of patients with OPMD or cancer and misses 3 of 61 patients with cancer, at a specificity of 0.610. Because the threshold is calibrated without retraining, each clinic can set its own guaranteed operating point on a local calibration sample.

0.16