CS Paper Radar

GAMF-Net: Grouped Availability-Aware Multimodal Fusion for Smartphone Skin Lesion Classification

Yue Zhuo, Jingjia Yan, Fangyu Liu, Lingjun Liu, Zongjun Chen, Zhisheng Luo, Zhonghua Xie

cs.CV ·submitted 2026-10-20 12:35 UTC · v1
published

Abstract

Smartphone clinical images provide a low-cost data source for computer-aided skin lesion classification in resource-constrained settings, but images alone cannot fully capture patients' medical histories and lesion symptoms. To address the lack of structured organization among heterogeneous clinical variables, the ambiguity between observed and imputed values, and the risk that clinical information may interfere with visual evidence, we propose GAMF-Net, a grouped availability-aware multimodal fusion network. Ten clinical variables are divided into three semantic groups---demographics, history-related risk, and lesion presentation---and each group jointly encodes field values and field-level availability masks. The image branch produces the base classification logits, while each clinical group generates image-conditioned class-level residual evidence. Only available groups participate in availability-normalized equal-weight fusion; when all clinical information is unavailable, the model reduces exactly to the image-only pathway. We evaluate the method on PAD-UFES-20 under a patient-disjoint five-fold protocol with separate training, validation, and test partitions. GAMF-Net achieves an Accuracy of $74.17\pm2.85\%$, a balanced accuracy (BACC) of $70.11\pm2.11\%$, a Macro-F1 of $68.85\pm0.85\%$, and a Macro-AUROC of $93.42\pm1.20\%$, outperforming all image-only and multimodal baselines reimplemented under a unified protocol. Compared with the strongest baseline, MetaBlock, the corresponding improvements are 2.37, 2.59, 4.14, and 1.02 percentage points, respectively. Ablation results show that clinical information substantially improves class-balanced performance, constrained equal-weight fusion is more effective than dynamic fusion, and the lesion-presentation group provides the most informative clinical complement.

Comment
14 pages, 4 figures. Accepted at MMM 2027

Closest papers

Nearest neighbours by meaning across the whole index.

Masked Privileged-Information Distillation for Multimodal Skin Lesion Classification Under Missing Clinical Metadata

cs.CV cs.LG ·2026-10-02 ·Anirban Barua, Md Mahir Abrar Khan, Ayman Iktidar, Md. Sajjatul Islam
similarity 0.73
Abstract

Multimodal skin lesion classification combines clinical images with patient metadata to improve diagnostic accuracy. However, complete metadata available during training may be only partially accessible at deployment, and resource-constrained settings additionally require computational efficiency. We address these challenges with a privileged-information distillation framework in which a multimodal teacher trained on complete metadata supervises a 9.2x smaller student trained with randomly masked clinical fields. Clinical fields are masked as whole groups at a per-sample rate drawn from U(0,1), so one training run covers the full metadata availability range. Fusion is residual, with metadata added as a gated correction to an unconditional image base. On the PAD-UFES-20 dataset, distillation under masked training improves balanced accuracy over cross-entropy training at every availability level, by an average of +4.7 points versus +1.8 points without masking. The masked student loses only 7.8 balanced-accuracy points as metadata decreases from complete to absent, compared with 36.6 points for the same student trained on complete metadata, highlighting the role of masked training in graceful degradation beyond distillation alone. Grad-CAM visualizations further show that the masked student's attention generally remains lesion-centered as metadata is withdrawn. The resulting compact model targets point-of-care settings, where clinical metadata is often incomplete.

0.23

Beyond Blur: A Semantic Tri-view Pipeline for Teledermatology Gradability via Skin Micro-relief

eess.IV cs.CVcs.HCcs.LG ·2026-09-02 ·Robert Engel
published similarity 0.68
Abstract

Smartphone skin photographs are indispensable to teledermatology, yet assessing the diagnostic suitability of submitted cases (gradability) remains a critical bottleneck in mobile care workflows. Dermatologists routinely review multiple photographic views (regional, angled, and close-up) to identify consistent textural detail rather than relying on a single image. We present the Semantic Tri-view Pipeline, an interpretable architecture for automated teledermatology gradability screening that formalizes epidermal micro-relief as a computable biomarker of image quality. Using an expert-annotated subset of the public SCIN dataset, we train a lightweight DeepLabV3+ model to segment micro-relief fidelity. These spatial masks are then aggregated across up to three case views with a logistic regression classifier, leveraging viewpoint redundancy to support robustness under uncontrolled smartphone acquisition. This approach learns context-aware, clinically intelligible heuristics, such as penalizing high-fidelity texture in regional distance views. Evaluated at a predefined 90% sensitivity operating point, the system's apparent errors largely reflect subjective clinical variance on borderline cases where clinicians rely on non-visual metadata. On SCIN, performance improves from an AUC of 0.81 (80.6% PPV) on variance-heavy majority-consensus cases to 0.96 (97.7% PPV) on optically unambiguous unanimous cases. Overall, this work delivers an interpretable, privacy-by-design, edge-ready system that can provide real-time feedback during case submission to filter ungradable photo sets before review.

0.40

Bridging Research and Practice: A Systematic Evaluation of Generalist and Dermatology-Specific Models in Clinical Skin Lesion Classification

cs.CV ·2026-10-02 ·Emanoel dos Santos, Kelvin Cunha, Rodrigo Mota, Fabio Papais, Thales Bezerra +6 more
top venuebenchmark similarity 0.67
Abstract · 10 pages, 1 figure, 3 tables, approved at MICCAI 2026

The application of machine learning to dermatology has grown substantially in recent years, moving beyond proof-of-concept studies toward potential applications. However, clinical dermatology remains a challenging and still open problem. Diagnostic assessment is often ambiguous, and skin lesions exhibit high variability, compounded by differences in acquisition modality, device quality, and patient demographics. These factors hinder the development of robust models suitable for safe and equitable clinical use. To support translation into practice, it is essential to systematically evaluate how contemporary models generalize across heterogeneous data sources. In this work, we benchmark a diverse set of architectures on recent dermatology datasets, spanning dermoscopic images and smartphone-based clinical photographs. We assess the robustness of recent general-purpose and medical vision-language models, as well as foundation models, and compare them against task-specific dermatology classifiers, including embedding-based approaches and convolutional neural networks. Our study provides an evaluation of model performance under distribution shifts, modality changes, and demographic variability. By quantifying the gap between current state-of-the-art models and the requirements of clinical deployment, we aim to contribute to the development of reliable, accessible, and clinically applicable AI systems for dermatology.

0.33

MIFR: A Modality-Invariant and Fair Representation Framework for Skin Disease Classification

cs.CV cs.LG ·2026-08-19 ·Asonyu Senge Njih, Yvan Guifo Fodjo, Vianney Kengne Tchendji, Jerry Lacmou Zeutouo, Kerol Djoumessi
published similarity 0.65
Abstract · Accepted for publication at the First Workshop on Advancing African Medical AI through Global Integration (AFRICAI)-MICCAI 2026

Skin diseases represent a major global public health burden, yet machine learning tools developed to assist in their diagnosis suffer from two critical limitations: reliance on only one modality for diagnosis and systematic performance disparities across skin tones. While existing approaches address each challenge separately, this work proposes a modality-invariant framework with fair representation (MIFR) for skin disease classification. The architecture pairs clinical photographs with dermoscopic images using ViT-based encoders, projecting each input into a high-dimensional embedding space via modality-specific projection heads. The resulting model is trained with a five-component multi-objective loss including weighted cross-entropy for classification, confusion and skin-type classification losses for fairness, per-modality supervised contrastive loss for class alignment, and a modality-invariance loss for clinical and dermoscopic modality alignment. Experiments on the HIBA+Derm7pt paired dataset and the external PAD-UFES-20 and ISIC 2019 datasets showed that modality-invariant representation learning provides competitive predictive performance compare to relevant baseline models and competitive fairness on the internal dataset. t-SNE visualizations confirmed that clinical and dermoscopic embeddings of the same disease are geometrically aligned, validating the joint objectives.

0.35

FedDermaSeg: Federated Learning for Dermatological Image Segmentation

cs.CV cs.AIcs.LG ·2026-10-06 ·Anabik Pal, Ganesh Patidar, Bikash Santra
similarity 0.64
Abstract

Skin cancer is a major global health concern, and early detection and accurate lesion delineation are important for effective diagnosis and treatment planning. Automated skin lesion analysis can assist dermatologists, with lesion segmentation serving as a fundamental step in computer-aided diagnostic systems. Conventional deep learning-based segmentation models typically rely on centralized training, where images and their corresponding segmentation masks are collected on a central server. Such data aggregation raises privacy concerns in medical applications and requires substantial centralized computational resources. To address these limitations, we investigate the feasibility of federated learning for privacy-preserving skin lesion segmentation. The training and validation sets of the ISIC 2018 Skin Lesion Segmentation Challenge dataset are used to simulate a distributed learning environment and develop a federated segmentation model. The resulting model is evaluated on the ISIC 2018 test set and the PH2 dataset to assess its performance and generalizability. Experimental results demonstrate that the federated model achieves performance comparable to centralized training while consistently improving upon the locally trained models. These findings demonstrate the potential of federated learning for collaborative skin lesion segmentation without requiring centralized aggregation of medical images.

0.13

Multi-Dataset Diagnostic Utility of Clinical Visual Concepts in AI Systems for Dermatology

cs.CV ·2026-10-06 ·Linda Wermelinger, Simone Lionetti, Fabian Gröger, Nipun Ranasekara, Philippe Gottfrois +4 more
publisheddatasetcode/data similarity 0.63
Abstract · Accepted at the MICCAI ISIC Workshop 2026. 11 pages, 3 figures, 3 tables. Code and dataset: https://github.com/Digital-Dermatology/SkinLex

The clinical integration of AI systems in digital dermatology relies heavily on human trust. Clinically interpretable visual concepts can act as intermediate representations enhancing trust and reliability. However, research in this domain is currently limited by scattered, heterogeneous dataset annotations. In this work, we introduce SkinLex, a harmonized dataset of 48 clinical morphological attributes across four public datasets (SkinCon, DermaCon-IN, MM-Skin, and PASSION) for a total of 20,411 records. Supervised nine-partition classification of skin conditions shows that limiting features to specific visual groups, like shapes or colors alone, reduces diagnostic accuracy. Bootstrapped backward elimination reveals that the set of 48 visual concepts has some degree of redundancy for algorithmic nine-partition diagnosis on the examined dataset. This demonstrates that coarse diagnosis on the selected dataset requires a relatively small but varied combination of clinical concepts, and motivates further research to improve concept taxonomy. Results can be translated into clinical benefits by reducing inputs for concept-based models, improving efficiency for annotation and modeling, and further enhancing interpretability. Code and prompt templates are available at https://github.com/Digital-Dermatology/SkinLex.

0.41

LENS-GRF: Permutation-Invariant Lesion Evidence Network with Gated Residual Fusion for Acne Severity Grading and Multi-Rater Clinical Oracle Analysis

cs.CV cs.AI ·2026-09-25 ·Muhammad Muhtasim Shahriar, M. F. Mridha, Md. Rakibul Islam
similarity 0.63
Abstract · Under Review at Results in Engineering (Elsevier)

Localized lesion evidence may improve automated acne severity grading, but detector errors and differences in clinical grading criteria can limit generalization. We evaluated LENS-GRF, a lesion evidence network with permutation-invariant aggregation and gated residual fusion that combines a global Vision Transformer with lesion-region embeddings derived from YOLO11s proposals. Fully automated four-class grading was evaluated on 292 held-out ACNE04 images and, without external adaptation, on 200 PLSBRACNE01 participants represented by 600 facial views. Annotation-assisted inference used registered lesion boxes from three dermatologists in a matched subset of 148 participants. Performance measures included accuracy, macro-averaged precision, sensitivity, and one-versus-rest specificity. Paired comparisons used exact McNemar tests with Holm adjustment. On ACNE04, automated LENS-GRF achieved 80.82% accuracy, 78.80% macro precision, 78.92% sensitivity, and 92.78% specificity. With supplied lesion boxes, internal accuracy averaged 95.89% across three seeds. On the full external cohort, automated LENS-GRF achieved 35.00% accuracy, compared with 42.50% for an independently trained global baseline. External macro precision, sensitivity, and specificity for LENS-GRF were 50.52%, 34.31%, and 78.19%, respectively. In the matched subset, accuracy using Doctor A's lesion annotations was 47.97%, compared with 35.14% for automated inference (unadjusted $p = 0.0094$; Holm-adjusted $p = 0.0846$). Strong internal performance with supplied lesion annotations did not translate into reliable automated grading on the external cohort. More reliable lesion detection, modelling that accounts for differences in grading criteria, and independent validation are needed before clinical use.

0.31

Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging

cs.CV cs.AI ·2026-09-16 ·Johannes Kaiser, Florian Braunmiller, Daniel Rückert, Georgios Kaissis
similarity 0.63
Abstract

AI models for multimodal medical imaging must balance modality-specific specialization with cross-modal shared representations, a trade-off that pure Mixture-of-Experts (MoE) architectures currently fail to satisfy. Expert-based routing improves in-domain learning but may sacrifice cross-modal signals, which appear particularly important for rare (low-prevalence) pathologies in our experiments. To resolve this, we introduce Generalist-Specialist-MoE (GS-MoE), a two-branch (MoE) architecture that couples a cross-modal generalist model with distinct modality-specific specialists (experts) via domain-constrained feature fusion. On RadImageNet (1.35M images, 165 pathologies, three modalities), GS-MoE recovers detection of six low-prevalence pathologies on which every baseline scores F1 $=$ 0, with per-class gains up to +0.60 F1. It attains this while even slightly exceeding dense and specialist-only MoE aggregate baselines (MCC 0.770), while using ${\sim}53\%$ fewer active parameters at inference than the strongest investigated dense model.

0.12