CS Paper Radar

Boot-and-Feedback Framework for Generalist-Expert Model Collaboration in Breast Ultrasound Diagnosis

Ming Cheng, Hongyu Sun, Zhaolin Chen, Jun Liu, Hossein Rahmani, Qiuhong Ke

cs.CV cs.MM ·submitted 2026-08-25 02:07 UTC · v1
published

Abstract

Breast ultrasound (BUS) is widely used for breast cancer diagnosis yet remains operator-dependent. While deep learning shows promise, ensuring diagnostic reliability and interpretability is challenging. Recent Multimodal Large Language Models (MLLMs) often generate spurious descriptions due to limited domain knowledge, which mislead downstream expert models and compromise clinical validity. To address these challenges, we propose the Boot-and-Feedback (BooF) model collaboration framework for synergistic MLLM-expert interaction. Specifically, in the Boot Stage, the MLLM is guided by the BI-RADS lexicon and preliminary benign-malignant vision-expert predictions, enabling it to transfer general reasoning to BUS analysis while avoiding hallucinations. Subsequently, the Feedback Stage integrates these descriptions with visual features via a lightweight Attention-Gated Cross-Modality Fusion Module. This allows the expert to leverage textual feedback while adaptively filtering noise. Extensive experiments on multiple BUS datasets demonstrate that BooF substantially outperforms state-of-the-art methods in terms of diagnostic accuracy and interpretability.

Comment
5 pages, 2 figures. Published in ICASSP 2026
Journal ref
ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2026
DOI
10.1109/ICASSP55912.2026.11460659

Closest papers

Nearest neighbours by meaning across the whole index.

Explainable Multimodal AI Approach to Reduce Unnecessary Breast Biopsies

cs.AI ·2026-10-26 ·Sadam Hussain, Mansoor Ali, Daly Betzabeth Avendaño-Avalos, Jose Gerardo Tamez-Peña
similarity 0.73
Abstract

Recently, there has been rise in multimodal vision-language models for computer-aided diagnosis (CAD), yet their application in breast diagnosis, particularly for predicting biopsy outcomes remains limited. This study presents a multimodal vision-language approach, incorporating Grad-CAM as an Explainable AI (XAI) technique, to predict benign tumors in patients recommended for biopsy, especially those classified as BI-RADS 4 and 5. A novel multimodal framework was devel- oped using Vision Transformer (ViT) for imaging feature extraction and DeBERTa-V3-Large for textual embeddings. The model was trained and evaluated on a comprehensive in-house dataset comprising 1,632 patients, each with four mammography views and a corresponding radiology report. The features of both modalities were extracted independently and fused by late fusion before classification. Grad-CAM was used to generate visual explanations highlighting key regions influencing model predictions. The proposed model demonstrated high sensitivity 90%, 95%, 99%, and 100% with corresponding specificities of 40%, 36%, 18%, and 11%, respectively. The best AUC achieved was 0.72 by multimodal approach on the test set, outperforming the image only (0.66) and text-only (0.68) baselines. The proposed multimodal architecture, enhanced with Grad-CAM, demonstrates a strong potential to identify benign tumors among high-risk patients, offering interpretable insights and thereby helping reduce unnecessary biopsies, patient anxiety, and healthcare costs.

0.10

Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation

cs.CV cs.AIcs.LG ·2026-09-14 ·Xinyue Xu, Hongbin Lin, Juangui Xu, Hualiang Wang, Lehan Wang +3 more
similarity 0.72
Abstract

Medical imaging modalities such as ultrasound and X-ray are widely used in clinical practice, where diagnosis follows a structured, evidence-driven workflow aligned with standardized criteria. While multimodal large language models (MLLMs) show promise for automated medical report generation, most existing systems rely on end-to-end multimodal fusion without modeling clinically defined intermediate attributes, leading to limited grounding and interpretability. To address this issue, we propose CORAL (COncept-grounded ReAsoning with Localization), a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process. CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module. The resulting textual concept tokens are combined with mask-modulated visual features within an MLLM to enable structured report generation and diagnostic prediction. Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs, indicating that concept-grounded reasoning better aligns generation with clinical decision processes.

0.32

Architecture-aware Robustness Evaluation of Explainable Deep Learning for Breast Cancer Diagnosis

cs.CV ·2026-09-18 ·Balenthira Thanusanth, Selvarajah Thuseethan, Roshan G. Ragel, Bimali S. Weerakoon, Ayesh Jayasinghe +1 more
code/data similarity 0.71
Abstract · 12 pages, 7 figures

Explainable Artificial Intelligence (XAI) has become essential in medical image analysis to ensure transparency of deep learning (DL)-based diagnostic systems. However, selecting appropriate XAI techniques for breast cancer recognition remains largely ad hoc, with limited systematic evaluation across different DL architectures. This study presents a systematic architecture-aware evaluation protocol to assess the effectiveness of nine widely used XAI techniques across four categories of DL models: very deep, lightweight, transformer-based and hybrid neural networks. The evaluation is conducted on a breast ultrasound dataset comprising 780 images using clinically aligned spatial metrics, including Pointing Game, Intersection over Union and Mean Coverage, to quantify agreement between generated explanations and expert-annotated lesion regions. Results indicate that explanation quality is primarily influenced by the interaction between model architecture and XAI method, rather than any single technique consistently outperforming others. Hybrid architectures produce more spatially coherent explanations, while lightweight and transformer-based models exhibit greater variability across methods. The findings show that no single technique generalises across architectures and evaluation criteria, emphasising the need for joint selection of DL models and XAI techniques. Explainability depends on both model design and explanation strategy and should not be considered independently. This work provides a structured evaluation protocol and practical guidance for selecting XAI techniques in breast cancer diagnosis, supporting more transparent clinical decision-support systems. \textcolor{blue}{Code is publicly available at https://github.com/Nishan-Charlie/Explainable-AI}

0.29

Multimodal AI in Healthcare: Review of Vision-Language Foundation Models for Real-World Medical Applications

cs.CV ·2026-11-15 ·Taha Razzaq, Murtaza Taj, Asim Iqbal
published similarity 0.69
Abstract · 20 Pages, 3 figures, 3 Tables

The emergence of foundation models has marked a transformative shift in AI, enabling robust generalization across diverse downstream tasks through putative zero-shot learning. Large Language Models and Vision Language Models have demonstrated strong capabilities in tasks such as image interpretation, report generation, and question answering by effectively learning from multimodal data-images paired with associated text often with minimal supervision. In the healthcare domain, this ability to align visual and textual information reduces the reliance on extensive manual annotations, as models can leverage existing clinical reports and imaging data to learn meaningful representations. This integration holds promise for improving diagnostic support, treatment planning, and overall patient care, even in data constrained settings. In this review, we provide a definitive taxonomy of the medical VLM landscape, tracing the evolution from early Contrastive Alignment and Generative MLLMs to the cutting edge frontiers of Dense Pixel Grounding, Sparse Mixture-of-Experts (MoE), and Reasoning Incentivized (RL) architectures. We critically examine the "medical bottleneck" identifying the persistent challenges of data scarcity, the "hallucination" risks in generative diagnostics, the computational strain of 3D volumetric processing, and the lack of standardized, clinically grounded evaluation metrics.

0.45

Interpretable Deep Learning Models for Breast Cancer Diagnosis Using Medical Imaging

cs.CV cs.AI ·2026-10-27 ·Sami Alnawali, Muhammad Atif Saeed, Abdul Hadi, Azka Atiq, Akhtar Jamil +1 more
benchmark similarity 0.69
Abstract · 37 page, 45 Figures

Deep learning shows promise for breast cancer diagnosis, but limited interpretability hinders clinical adoption. We benchmark ten ImageNet-pretrained models (ResNet50, VGG19, InceptionV3, DenseNet121, Xception, MobileNetV2, EfficientNet, ConvNeXt, ViT, and Swin Transformer) for benign-versus-malignant classification on the BreakHis histopathology dataset using a patient-level split, and explain their predictions with Grad-CAM, LIME, Integrated Gradients, and LRP. DenseNet121 performed best, with 99.4% test accuracy and an AUC-ROC of 0.999, and significantly outperformed all other models (p < 0.001). Transformer-based models trailed the CNNs. The explanations consistently highlighted nuclei-rich and glandular regions of diagnostic relevance. Combining transfer learning with explainable AI can thus yield accurate, interpretable tools for breast cancer diagnosis.

0.19

SMILE: A Structured Multi-Agent Clinical Reasoning for Liver Ultrasound Understanding

cs.CV cs.CLcs.LG ·2026-11-02 ·Haris Ghafoor, Minjee Seo, Seonaeng cho, Keungmo Yang, Jaejun Lee +1 more
publishedcode/data similarity 0.69
Abstract · Accepted at Winter Conference on Applications of Computer Vision (WACV) 2027

Automated liver ultrasound report generation improves the diagnosis of liver diseases by producing clinically structured descriptions of anatomical and disease-related findings. However, current generative vision-language models (VLMs) often rely on a single reasoning pathway that maps ultrasound appearance directly to textual descriptions. In addition to the inherent vision-language modality gap, this design is limited for liver ultrasound because disease interpretation is constrained by subtle visual evidence, heterogeneous disease patterns, and limited paired supervision. To address these limitations, we propose SMILE, a Structured Multi-agent clInicaL rEasoning framework that uses multiple complementary specialist agents to interpret ultrasound images and consolidates their outputs through a judge-guided consensus mechanism. Our approach employs three specialized agents: an image interpretation agent for morphological descriptions, a clinical-stage reasoning agent, and a visual question answering (VQA) agent for targeted queries. These agent responses are then aggregated and scrutinized by a judge VLM using a consensus filtering prompt to produce a unified, clinically coherent output. Across a public liver ultrasound dataset and an internal clinical dataset, SMILE consistently outperforms state-of-the-art VLMs, achieving up to 19.0\% improvement in ROUGE-1 and 18.8\% improvement in clinical F1-score, while reaching VQA F1-scores of 0.82 and 0.65 on public and internal datasets, respectively. These results demonstrate the potential of SMILE for data-scarce and limited-supervision clinical settings. Our models, datasets, and code are available \href{https://github.com/CMME-Lab/SMILE}{here}.

0.42

ProtoCAM: Interpretable Few-Shot Mask-Guided Prototypical Learning for Breast Lesion Classification in Ultrasound Imaging

cs.CV cs.AI ·2026-09-11 ·Ashkan Ebadi
similarity 0.68
Abstract · 11 pages, 3 figures

Breast ultrasound imaging plays an important role in the early detection and diagnosis of breast cancer, particularly for patients with dense breast tissue. However, developing reliable deep learning models for ultrasound analysis is challenging due to limited annotated medical data and the need for interpretable predictions. To address these challenges, this paper proposes ProtoCAM, an explainable few-shot learning framework for breast lesion classification that integrates mask-guided feature encoding, prototypical metric learning, and gradient-based visual explanations. The proposed approach leverages lesion masks to guide feature extraction and constructs class prototypes within an embedding space to enable robust classification under limited training samples. The framework was evaluated on the BUSI dataset using a stratified group k-fold cross-validation protocol to prevent patient-level data leakage. Experimental results demonstrate ProtoCAM's high performance in low-data scenarios. In a 3-way 5-shot setting, the proposed method achieves a macro F1-score of 0.910, representing a substantial improvement over standard supervised CNN models. Among the evaluated backbone networks, ResNet18 achieved the best performance, reaching a macro F1-score of 91.65% under a 15-shot configuration, providing interpretable insights into the classification decisions. These results highlight the potential of explainable few-shot learning frameworks for reliable computer-aided breast cancer diagnosis in data-scarce medical imaging environments.

0.10

Beyond Explanation: Debugging Medical Imaging Models via Concept Intervention

cs.CV cs.LG ·2026-10-06 ·Samrajya Thapa, Daniel J. Quest, Timothy L. Kline, Carrie L. Langstraat, Emanuel C. Trabuco +1 more
published similarity 0.68
Abstract · Accepted at the 5th Workshop on Applications of Medical AI (AMAI), MICCAI 2026

Medical imaging models often operate as black boxes, limiting interpretability and systematic debugging. We introduce an easy-to-use, plug-and-play framework for concept-based interpretation and model refinement. By aligning a single-modality encoder to BioMedCLIP, we construct a Concept Bottleneck Model (CBM) that enables concept-level interventions. These interventions allow us to isolate causal versus spuriously correlated concepts, validate insights with domain experts, and generate counterfactual samples for targeted fine-tuning. We evaluate our framework on a Mayo Clinic ultrasound dataset and the CheXpert 5x200 chest X-ray dataset. Results demonstrate that concept intervention enables reliable model diagnosis while maintaining, and occasionally improving predictive performance via guided fine-tuning. Our findings highlight the practical value of this framework for controlled, interpretable refinement of clinical deep learning models.

0.19