CS Paper Radar

Can Multimodal Decision Models Classify Breast Ultrasound Images? Benchmarking CLEF and CLEF-Flash Across Prompts and Datasets

Tran Cao Minh, Hai Tran, Ha Minh Tan

eess.IV cs.CV ·submitted 2026-10-09 14:41 UTC · v1

Abstract

Multimodal decision models return probabilities over predefined alternatives, offering a structured interface for image classification. This study benchmarks inference with CLEF and CLEF-Flash on 2818 breast ultrasound images from Dataset B, BUSI, and BUS-BRA. Dataset B and BUS-BRA use binary benign-malignant classification, whereas BUSI includes normal, benign, and malignant categories. Model parameters remain unchanged, and original images are submitted without user-defined preprocessing. Three prompts provide minimal labels, an explicit classification task, or sonographic criteria. We report accuracy, macro F1, balanced accuracy, macro precision, throughput, and latency. CLEF-Flash with the clinical prompt achieves the highest accuracy on all three datasets: 74.85% on Dataset B, 66.28% on BUSI, and 51.63% on BUS-BRA. On BUSI, however, CLEF with the task-guided prompt achieves the highest macro F1 (64.51%) and balanced accuracy (68.72%). CLEF-Flash has lower latency in every matched comparison. Clinical instructions improve CLEF-Flash accuracy relative to the minimal prompt but reduce CLEF accuracy on all datasets. Confusion matrices reveal frequent benign-to-malignant errors, particularly on BUS-BRA, where all configurations fall below the 67.63% accuracy of a majority-class predictor. These findings demonstrate dataset-and prompt-dependent performance and motivate class-specific evaluation before applying multimodal decision models to breast ultrasound analysis.

Closest papers

Nearest neighbours by meaning across the whole index.

Single-Pass Decision Scores Enable Rapid Classification with Adjustable Thresholds in Medical Multimodal Large Language Models

cs.CV ·2026-10-14 ·Patrick Wienholt, Frederik Hauke, Christiane Kuhl, Jakob Nikolas Kather, Sven Nebelung +1 more
code/data similarity 0.72
Abstract · 29 pages (18 pages main text, 11 pages Supplementary Information), 5 figures, 3 tables

Multimodal large language models (MLLMs) answer binary clinical questions with a yes or no token, leaving the decision threshold implicit and fixed. We derive a continuous score from a single forward pass by renormalizing the probabilities of the yes and no tokens, allowing the threshold to be adapted to clinical needs without retraining. We evaluated this approach with MedGemma and Gemma-3 (4B and 27B) on six medical imaging datasets. The default threshold was frequently misaligned with the task: for identifying normal chest radiographs, MedGemma-27B labeled every image as abnormal, whereas a validation-selected threshold yielded a sensitivity of 0.945 and a specificity of 0.696. Normalization achieved the highest AUROC on four of six datasets (CheXpert: 0.819 vs. 0.777 for raw and 0.743 for text-elicited probabilities), and a single decision required 53-79 ms on one GPU. This simple post-hoc method turns medical MLLMs into fast classifiers with adjustable operating points. Code: https://github.com/TruhnLab/llm-threshold-calibration

0.46

A Concept-informed Interpretable Artificial Intelligence Framework for Breast Ultrasound Diagnosis

cs.CV ·2026-10-12 ·Shengzhou Zhong, Ziying Wang, Liang Li, Lihua Yan, Shuochun Chen +12 more
code/data similarity 0.66
Abstract · 31 pages, 17 figures, 4 tables

Most existing artificial intelligence systems for breast ultrasound rely primarily on image-driven prediction, with limited incorporation of the knowledge from the Breast Imaging Reporting and Data System (BI-RADS) and the intermediate assessment processes underlying radiologists' workflow, thereby constraining their interpretability and alignment with clinical practice. We curated, to our knowledge, the largest multicenter collection of breast ultrasound images paired with structured BI-RADS descriptors, comprising 9,941 patients across 20 independent datasets. Across 14 testing datasets, IBUS demonstrated consistent generalization performance across different clinical centers and imaging conditions. Incorporating BI-RADS descriptors enabled fine-grained risk stratification, while cross-checking predicted BI-RADS grades against malignancy probabilities identified internally discordant predictions that may warrant additional clinical review. In a reader study involving eight radiologists, IBUS assistance increased diagnostic sensitivity among both junior and senior readers, with the potential of guiding radiologists to make targeted revision in risk stratification. Overall, IBUS exhibits its potential as a second-stage decision-support system that complements radiologists by providing additional risk estimates and interpretable evidence for diagnostically challenging cases. The demo video of IBUS platform is publicly available at https://github.com/zsz1997/IBUS.

0.30

Explainable Multimodal AI Approach to Reduce Unnecessary Breast Biopsies

cs.AI ·2026-10-26 ·Sadam Hussain, Mansoor Ali, Daly Betzabeth Avendaño-Avalos, Jose Gerardo Tamez-Peña
similarity 0.66
Abstract

Recently, there has been rise in multimodal vision-language models for computer-aided diagnosis (CAD), yet their application in breast diagnosis, particularly for predicting biopsy outcomes remains limited. This study presents a multimodal vision-language approach, incorporating Grad-CAM as an Explainable AI (XAI) technique, to predict benign tumors in patients recommended for biopsy, especially those classified as BI-RADS 4 and 5. A novel multimodal framework was devel- oped using Vision Transformer (ViT) for imaging feature extraction and DeBERTa-V3-Large for textual embeddings. The model was trained and evaluated on a comprehensive in-house dataset comprising 1,632 patients, each with four mammography views and a corresponding radiology report. The features of both modalities were extracted independently and fused by late fusion before classification. Grad-CAM was used to generate visual explanations highlighting key regions influencing model predictions. The proposed model demonstrated high sensitivity 90%, 95%, 99%, and 100% with corresponding specificities of 40%, 36%, 18%, and 11%, respectively. The best AUC achieved was 0.72 by multimodal approach on the test set, outperforming the image only (0.66) and text-only (0.68) baselines. The proposed multimodal architecture, enhanced with Grad-CAM, demonstrates a strong potential to identify benign tumors among high-risk patients, offering interpretable insights and thereby helping reduce unnecessary biopsies, patient anxiety, and healthcare costs.

0.10

OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System One

cs.LG ·2026-09-30 ·Luyao Tang, Cheng Chen
code/data similarity 0.65
Abstract · We introduce OmniMed-Jev, a decision-native interface based on Jev that outputs Choice, Noul or Score decisions with calibrated probabilitie…

Medical models are judged not only on correctness, but on whether reported confidence matches actual accuracy. Generalist multimodal medical models have expanded what a single model can perceive, yet they still express bounded decisions such as diagnoses, findings or cell counts as generated text, so the reported probability reflects the next token rather than the decision itself. Motivated by decision-native interfaces such as Jev, we introduce OmniMed-Jev, which represents each medical decision as a Choice, Noul or Score decision over a runtime-supplied candidate set and returns a full distribution over that set: mutually exclusive classes, binary presence of a finding, or a bounded ordered value. The design is omni in three respects: it accepts diverse imaging modalities, covers different prediction tasks, and expresses them through one candidate-conditioned probability model, so heterogeneous outputs become comparable probabilities rather than task-specific strings. In an interface-controlled comparison against a generative baseline trained on the same backbone, data and schedule, OmniMed-Jev's reported probabilities track observed correctness far more closely, reducing calibration error by up to an order of magnitude and reliability error by up to two, while point-prediction performance remains comparable; counting is the one family where the generative baseline stays ahead. Making the decision distribution the model's output is not a format change but what turns reported numbers into probabilities that mean what they say. These results support explicit decision modeling as a way to make reported confidence meaningful within the evaluated tasks, and they are not evidence of clinical readiness: the comparison cannot separate the interface from associated training differences, which we state alongside the results. Code is available at github.com/lytang63/OmniMed-Jev.

0.55

Boot-and-Feedback Framework for Generalist-Expert Model Collaboration in Breast Ultrasound Diagnosis

cs.CV cs.MM ·2026-08-25 ·Ming Cheng, Hongyu Sun, Zhaolin Chen, Jun Liu, Hossein Rahmani +1 more
published similarity 0.65
Abstract · 5 pages, 2 figures. Published in ICASSP 2026

Breast ultrasound (BUS) is widely used for breast cancer diagnosis yet remains operator-dependent. While deep learning shows promise, ensuring diagnostic reliability and interpretability is challenging. Recent Multimodal Large Language Models (MLLMs) often generate spurious descriptions due to limited domain knowledge, which mislead downstream expert models and compromise clinical validity. To address these challenges, we propose the Boot-and-Feedback (BooF) model collaboration framework for synergistic MLLM-expert interaction. Specifically, in the Boot Stage, the MLLM is guided by the BI-RADS lexicon and preliminary benign-malignant vision-expert predictions, enabling it to transfer general reasoning to BUS analysis while avoiding hallucinations. Subsequently, the Feedback Stage integrates these descriptions with visual features via a lightweight Attention-Gated Cross-Modality Fusion Module. This allows the expert to leverage textual feedback while adaptively filtering noise. Extensive experiments on multiple BUS datasets demonstrate that BooF substantially outperforms state-of-the-art methods in terms of diagnostic accuracy and interpretability.

0.52

When Adaptation Hurts: Connecting Representational Drift to OOD Failures in MedSAM Fine-Tuning

cs.CV ·2026-08-21 ·Marko Haralović, Sounic Akkaraju, Carlo Baretta, Vasil Zapryanov, Alexia Briassouli
publishedcode/data similarity 0.64
Abstract · Accepted at the SAFER Workshop, MICCAI 2026, non camera-ready version

Foundation models for medical image segmentation, like prompt-based MedSAM, generalize well across domains and modalities, often in zero or few-shot setups. However, their performance depends on the quality of prompts and the adaptation of the models to custom datasets. This work systematically examines how MedSAM generalizes across diverse medical imaging benchmarks, with six adaptation strategies: full-model and encoder-only LoRA, shallow and deep visual prompt tuning (VPT), and decoder-only and full fine-tuning. Models are trained on the International Skin Imaging Collaboration Challenge (ISIC 2018) dataset and evaluated under clean and increasingly noisy prompts on IN and Out-of-Distribution (OOD) datasets: close-OOD PH2 (dermoscopy), far-OOD BUSI (Breast Ultrasound Images Dataset) and CBIS-DDSM (Curated Breast Imaging Subset of the Digital Database for Screening Mammography). We show that adaptation improves performance on IN and close-OOD data but often reduces performance on far-OOD data. Full fine-tuning provides the best tradeoff, while encoder-only LoRA is the strongest parameter-efficient alternative, outperforming standard LoRA and VPT under far-OOD shifts. Using Centered Kernel Alignment (CKA), we show that far-OOD degradation is strongly associated with drift in decoder representations, whereas encoder similarity alone does not explain robustness. This suggests encoder-only LoRA provides stronger robustness than standard LoRA by adapting the encoder to distribution shift in visual features, while preserving the decoder pathway. We further show that random 0-100 pixel jitter on prompts produces more robust and better performing models. We thus conclude that robust MedSAM adaptation requires the combined consideration of prompt noise exposure, domain shift, and representation preservation. We release our code: https://github.com/ImSounic/medsam-vpt

0.68

A Controlled Audit of Architectural Complexity in Uncertainty-Aware Multi-Organ Ultrasound Classification

cs.CV ·2026-08-28 ·Yang Song, Pengbo Sun, Shichang Feng, Ye Zhu, Xin Xu +1 more
similarity 0.64
Abstract · Supplementary material is available as an ancillary file on this arXiv page

Multi-organ ultrasound classifiers increasingly combine attention, mixture-of-experts routing, uncertainty gating, and evidential deep learning (EDL) objectives to address heterogeneous anatomy and acquisition. Yet a plausible design rationale does not by itself establish that an added component improves the trained system. We contribute a controlled complexity-audit framework, applied to the deployment decision between the maximal evidential candidate Full-EDL and simpler alternatives. Six candidates were evaluated on the primary dataset and three in an internal replication, using ten matched seeds, frozen image-level partitions, capacity- and optimisation-aware comparisons, symmetric temperature scaling, paired decision rules, and a separate out-of-distribution (OOD) veto. Retaining Full-EDL did not establish a reliable macro-F1 gain on either dataset, while the simplified alternatives remained inconclusive under the non-inferiority margin. Simple cross-entropy with temperature scaling (Simple-CE+TS) met the calibrated negative log-likelihood criterion on both datasets and showed favourable selective-risk ordering. The raw calibration advantage of evidential training disappeared after temperature scaling and did not recur on the second dataset. The gate had negligible observable influence at the audited checkpoints, and deleting the Full-only chain revealed no stable task or calibrated-loss benefit. Simple-CE nevertheless triggered the OOD veto against the fetal probe but not the lung probe, precluding an unconditional OOD-safety claim. We therefore selected Simple-CE+TS for the evaluated in-distribution objective while retaining Full-EDL as the maximal reference. Components should earn retention through functional and retraining-based evidence, and calibration and distribution-shift reliability should be evaluated separately.

0.19

Open ultrasound foundation model for robust segmentation and clinical measurement across heterogeneous settings

cs.CV ·2026-09-16 ·Chao Qin, Fahad Shahbaz Khan, Salman Khan, Sarim Ather, Siddiq Anwar +2 more
code/data similarity 0.63
Abstract · The PDF includes the Supplementary Information. Code is available at https://github.com/AlfredQin/sonobase

Ultrasound is the most widely deployed imaging modality worldwide, yet clinical AI remains fragmented into narrow single-task models that fail when device, operator, or anatomy changes. Here we present SonoCorpus, an open resource unifying 456,963 images and 1,626,085 expert masks from 53 public datasets spanning 24 clinical applications and 17 countries, and SonoBase, an interactive segmentation foundation model pretrained on it. Across fifteen evaluation datasets introducing new organs, devices, operators, and geographies, SonoBase outperforms SAM2, MedSAM2, and the concept-promptable MedSAM3 on every dataset and matches per-dataset specialist models trained on the same data; on fully external data it exceeds the accuracy these baselines achieve on their own in-distribution benchmarks. Ejection fraction derived from its segmentations falls within inter-observer variability (6.63\% error), with fewer misclassifications at the defibrillator-candidacy threshold than either promptable baseline (13\% versus 18--42\%); fetal head-circumference (1.81~mm) and gestational-age (1.2 days) errors fall below inter-observer variability. Where a baseline fails outright, one in four test cases, SonoBase recovers a usable segmentation in 81\% of them, including on handheld probes operated by minimally trained users in two low- and middle-income countries (Sierra Leone and Tanzania). Five labeled examples can help the model adapt to a new setting, and the identical training protocol transfers well to newer models such as SAM3, locating the advantage in ultrasound-specific pretraining rather than any single architecture. To ensure reproducibility and enable the community to build on SonoBase as a platform, we release all checkpoints, optimizer states, data-split indices, deduplication hashes, and starter code at https://github.com/AlfredQin/sonobase.

0.36