CS Paper Radar

Evaluating Cross-Signal Coherence in Mammography Vision-Language Models

Rebeca Nam, Chase M. Fensore, Christian A. Fauria-Robinson, Judy Gichoya, Joyce C. Ho

cs.CV cs.CLcs.LG ·submitted 2026-11-09 15:33 UTC · v1

Abstract

As vision-language models (VLMs) move toward interpreting medical images directly, their outputs increasingly include not just a label but a stated confidence and a written justification. Yet the field still predominantly evaluates VLMs using just accuracy. We evaluate the coherence between answer, confidence, and explanation across four VLMs on breast density and BI-RADS categorization, using two mammography datasets. We perform controlled perturbations of the question, multiple choice option orders, and the images and assess changes in the three outputs. Our results suggest confidence does not decrease even as the mammography image is changed. Shuffling the options also changes most VLMs' answers. Notably, there are cases where a model's own stated reasoning contradicts its final answer, a distinction accuracy alone cannot capture but one that governs whether an AI-generated answer can be trusted.

Closest papers

Nearest neighbours by meaning across the whole index.

MammoSeek: A Mammography Agent with selective zooming for grounded reports

cs.CV ·2026-11-04 ·Haoyan Ding, Qian Wang, Lixia Wang, Debiao Li
similarity 0.68
Abstract · 5 pages, 2 figures

Radiologists read a screening mammogram by first forming a global impression of the four views and then selectively magnifying suspicious regions before assigning a BI-RADS assessment. MLLMs, in contrast, see the whole image at a fixed, reduced resolution, which makes sub-millimeter calcifications invisible, and they perform close to chance on mammography. Therefore, we present MammoSeek, a mammography agent that reproduces this reading process. After forming an impression of each breast, the agent performs selective zooming: it decides by itself which detector candidates deserve a closer look and inspects only those on full-resolution views. Guided by a BI-RADS assessment skill, it then assigns a category and generates a localized, lexicon-structured report for each breast; its assessment is combined with image-model scores in a decision layer to decide recall. On an internal (EMBED) and an external (VinDr-Mammo) dataset, \ours reaches exam-level AUCs of 0.824 and 0.818, far above other MLLMs, mammography agent and CNN classifiers. Selective zooming also outperforms both no zooming and forced zooming in localization, structured-report F1 and recall decisions at low cost.

0.29

PROVE: Proof-guided Regime-aware Operator Verification for Hallucination Detection in Medical Visual Question Answering

cs.CV cs.MM ·2026-08-28 ·Keyang Zhou, Siyi Li, Zhongnan Shi, Qichao Ying, Wei Tang +1 more
similarity 0.66
Abstract

In medical visual question answering (VQA), hallucinations of vision-language models (VLMs) may lead to confident but incorrect responses, raising the risk of diagnostic errors. Existing hallucination detection methods uniformly estimate the reliability of VLM outputs from response consistency or visual evidence. However, such uniform verification across questions ignores question-specific characteristics, resulting in missed overconfident errors and false alarms from over-verification. We present PROVE (Proof-guided Regime-aware Operator Verification), a black-box detector that adapts verification strategy to the evidential structure of each question. PROVE classifies questions into three verification regimes based on what kind of visual proof they demand, activates a regime-specific subset of five complementary operators, and adjusts operator importance per question through a lightweight calibration layer conditioned on deterministic question-answer features. PROVE uses question-specific evidence to reweight operators and produce a calibrated risk score. Evaluated on 8048 test samples across three medical VQA benchmarks and four frontier VLMs, PROVE achieves 0.821 AUROC, outperforming the strongest baseline by +0.159, with consistent gains across all models and benchmarks.

0.20

Bilateral Evidence Selection and Reasoning for Mammography Vision--Language Models

cs.CV cs.AI ·2026-11-19 ·Zhengbo Zhou, Dooman Arefan, Shandong Wu
similarity 0.66
Abstract

Medical vision--language models (VLMs) often need to identify small, subtle lesions within high-resolution images containing extensive normal anatomy and redundant visual information. Visual-token compression improves efficiency, but preserving salient regions alone can discard the context needed for diagnostic interpretation. This context can extend across images: in bilateral mammography, the opposite breast provides a patient-specific anatomical reference. Yet retaining such evidence does not ensure that the model uses it effectively. We propose Bilateral Evidence Selection and Reasoning (BESR), which connects evidence preservation with comparative assessment through a bilateral-guided token selector and a bilateral reward. Under a shared token budget, the selector jointly prioritizes lesion appearance and its local and bilateral context, while the reward encourages assessments that benefit from the preserved comparison. Experiments on a bilateral dataset constructed from CBIS-DDSM and on VinDr-Mammo demonstrate improved evidence retention and diagnostic performance under constrained visual-token budgets. Inference benchmarking further shows reduced prefill latency.

0.30

EVADE: Evidence-Verified Agentic Diagnosis with Escape

cs.CV ·2026-08-19 ·Mohaimenul Azam Khan Raiaan, Nur Mohammad Fahad
similarity 0.65
Abstract

Medical vision-language models (VLMs) can achieve high accuracy but remain unreliable: they are systematically overconfident, benefit little from test-time reasoning, and lack the ability to reliably calibrate trust in their own responses. We introduce EVADE (Evidence-Verified Agentic Diagnosis with Escape), an inferential, non-training method that enhances the safety of deploying a single frozen VLM. EVADE responds and, when uncertain, localises the region most diagnostically relevant, re-answers on a zoomed view, and commits only when both the entire image and the zoomed view responses agree; otherwise, it abstains. To directly address verification hallucination in single-model self-checking, our main idea is to verify gate consistency across different image views rather than re-reading the model's own text. Experimental evaluation on VQA-RAD, SLAKE, and PathVQA using Qwen2.5-VL-7B reports that EVADE is the only method that simultaneously improves both calibration and selective risk while maintaining accuracy, reducing expected calibration error (ECE) by up to 45% compared to zero-shot. Chain-of-thought, self-consistency, and self-verification all fail at least one axis. A grounding analysis reports that self-proposed regions perform better at diagnostic structure localisation than centres or random crops. However, a 7B VLM cannot use this localisation to revise answers. Therefore, reliability gains come from the consistency gate and calibrated abstention.

0.23

Rethinking Correctness for Uncertainty Estimation in Clinical Prediction with Vision-Language Models

cs.LG ·2026-09-14 ·Mingcheng Zhu, Jinning Liang, Tingting Zhu
similarity 0.64
Abstract · 5 pages, 3 figures

Vision-language models are increasingly explored for clinical prediction from electronic health records and medical images, where identifying unreliable predictions is important for safe deployment. Uncertainty estimation (UE) enables detecting such predictions, but its evaluation depends on a correctness criterion that determines whether each model output is correct. If this criterion disagrees with human judgement or distorts downstream UE performance, conclusions about model reliability can be misleading. We introduce a two-axis framework that evaluates correctness criteria by their agreement with human judgements and fidelity to human-referenced UE performance. We assess eight criteria across three clinical prediction tasks and three models using 450 predictions annotated by two reviewers. Across the audited tasks, canonical exact matching (EM) achieved the highest observed human agreement and lowest UE distortion, while the BERT-based matching (BEM) and LLM-judge also showed strong human agreement. Across four UE methods and 23,254 clinical predictions, criterion choice changed error-detection AUROC by up to 0.146 and reversed the relative ranking of UE methods. The LLM-judge also selectively accepted invalid or uncertain outputs, accepting 16 of 30 such human-identified errors. These results demonstrate that correctness assessment is an integral component of clinical UE evaluation and should be validated before UE methods are compared.

0.18

MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering

cs.CL ·2026-09-03 ·Erfan Nourbakhsh, Ke Yang, Anthony Rios
top venue similarity 0.63
Abstract · Accepted to EMNLP Findings 2026

Medical visual question answering (Med-VQA) is often assumed to require medical fine-tuning, large models, or complex multi-agent pipelines. We revisit this assumption with \textbf{MedProb}, a lightweight probing framework that predicts multiple-choice Med-VQA answers from frozen VLM representations without free-text generation. Across PATH-VQA, SLAKE, and VQA-RAD, MedProb recovers substantially more answer-relevant signal than prompting and performs stronger than medical VLMs and agentic systems. Probing also reduces the apparent gap between small and large models compared to prompting, suggesting that smaller VLMs contain more recoverable Med-VQA signal than generation-based evaluation reveals. Across 14 matched general-purpose and medical VLM pairs, medical adaptation does not consistently improve this linear decodability. Finally, free-text generation exhibits an answer-position bias of up to 10 percentage points, whereas MedProb also has positional bias, however, it is impacted differently than prompting. Our main results target the multiple-choice/multiclass Med-VQA setting; we additionally show the probe can be extended to open-ended generation via a rejection-sampling scoring procedure.

0.33

MammoMix: Leveraging Mixture of Experts for Robust Mammogram Breast Detection

cs.CV ·2026-08-11 ·Dinh Tan Nguyen, Hoang Quan Dang, Chen Zhang, Sai Ho Ling
published similarity 0.63
Abstract · Australasian Joint Conference on Artificial Intelligence 2025

Breast lesion detection in mammography remains a challenging task due to variations in image quality, lesion appearance, and population demographics across datasets. While current object detectors such as YOLO and DETR achieve strong results on individual datasets, their performance often degrades when trained on or applied across heterogeneous sources. To address this, we propose MammoMix, a novel framework based on Mixture-of-Experts (MoE) paradigm for robust and generalizable lesion detection. In MammoMix, each expert model is trained on a specific domain, allowing it to specialize in distinct characteristics of its source data. A gating mechanism adaptively weighs contributions from each expert based on input image, combining their outputs to enable domain-adaptive inference. To improve reliability, we further incorporate a calibration module, MoCAE, which adjusts confidence scores to reflect true predictive uncertainty. We evaluate MammoMix on 3 public mammography datasets: CSAW, DDSM, and DMID, covering diverse clinical settings. Results show that MammoMix outperforms baseline detectors in both average precision and reliability, particularly on datasets with greater variability. Our findings demonstrate that expert specialization and calibrated ensemble fusion significantly enhance model generalization and robustness. MammoMix offers a promising step toward dependable AI-assisted breast cancer screening across real-world clinical domains.

0.52

Uncertainty, Not Concealment: A Mechanistic Reappraisal of Sycophancy in Vision-Language Models

cs.CV cs.LG ·2026-11-06 ·Mohamed I. Eldagla, Zeyad H. ElShafei, Ahmed M. Anter
publishedcode/data similarity 0.62
Abstract · Accepted to the NeurIPS 2026 Med-Reasoner Workshop

Vision-language models (VLMs) in medical retrieval-augmented generation must be able to say when a retrieved report contradicts the X-ray, so understanding whether they can detect such conflicts is important for safe deployment. The sycophancy mask hypothesis holds that these models internally detect conflicts but refuse to report them, concealing a detected contradiction. We introduce a three-part diagnostic that separates concealment from uncertainty: representation probing, activation steering, and two-way forced choice. Across four instruction-tuned VLMs on chest X-rays, we confirm behavioral suppression of contradiction and find that a linear probe separates matched from mismatched inputs, beyond what a simple CLIP model achieves. Steering the learned conflict direction never elicits contradiction, and forcing a binary choice does not recover one; instead, cross-modal attention to the image drops on mismatched pairs. We conclude that these models fail to contradict because the mismatch registers as hedged, gated uncertainty rather than concealed knowledge, calling for re-grounding and uncertainty surfacing. Code is available at https://github.com/mohamed-eldagla/Uncertainty-Not-Concealment/.

0.31