CS Paper Radar

SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answering

Yaojun Hu, Danyang Tu, Yang Liu, Jiajin Zhang, Wei Fang, Zhiqiang Liu, Chunlai Dong, Yingda Xia, Haochao Ying, Jian Wu, Ling Zhang

cs.CV ·submitted 2026-08-26 10:58 UTC · v1
top venuebenchmark

Abstract

Volumetric medical VQA requires reasoning over long and redundant 3D visual token sequences, especially in multi-sequence MRI where complementary modalities provide diverse diagnostic cues but expose the decoder to many repeated anatomical regions. To investigate reasoning under multi-sequence visual redundancy, we first introduce BreMRIs-VQA, a clinically curated breast MRI benchmark with 1.19M QA pairs from 71.0K sequences and 12.9K patients, covering both free-text and multiple-choice questions. We further propose SeVeR, a selective visual exposure framework that compresses dense volumes into modality-wise prototypes and retrieves complementary multi-level evidence with change-aware gated attention during decoding, trained with a marginal-utility self-consistency objective that suppresses unhelpful retrieval. Experiments on BreMRIs-VQA and public benchmarks show that SeVeR improves both discriminative and generative performance while exposing substantially fewer visual tokens.

Closest papers

Nearest neighbours by meaning across the whole index.

InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision

cs.CL cs.CV ·2026-09-28 ·Guanghao Zhu, Zeyu Liu, Zhitian Hou, Pengkai Wang, Zhijie Sang +7 more
similarity 0.72
Abstract

Recent medical multimodal models have benefited from larger corpora, broader modality coverage, and stronger reasoning-oriented training, yet effective data design across continued pretraining (CPT) and post-training remains challenging. Medical sources vary substantially in structure, granularity, and information density, and their utility shifts as training progresses from broad knowledge acquisition to late-stage consolidation. Meanwhile, post-training is often dominated by short-form visual question answering, providing limited supervision for informative and answer-consistent explanations. We introduce InfiMed2, a family of 4B and 27B generalist medical multimodal foundation models built around stage-aware data design. We curate a 55.68B-token corpus that combines broad clinical knowledge with context-rich biomedical visual evidence through source-specific processing. Our CPT pipeline first adapts the vision encoder, then builds broad medical knowledge, and finally transitions to an evidence-focused data mixture during learning-rate decay. For supervised fine-tuning (SFT), we regenerate visual question-answering responses using answer stability, answer-masked reconstruction, and correctness-constrained selection to produce more informative and answer-consistent supervision. The 4B model is further optimized with reinforcement learning with verifiable rewards (RLVR). Across five medical multimodal benchmarks, InfiMed2-4B achieves 66.73% mean accuracy after RLVR, surpassing the larger Qwen3.5-9B, while InfiMed2-27B reaches 73.72%, the highest among the evaluated open-weight models.

0.22

MedQA-MM: Shortcuts Behind Medical Visual Reasoning

cs.CV cs.CL ·2026-09-03 ·Benlu Wang, Yifan Zhang, Jiaqing Yu, Chin Siang Ong, Juncheng Huang +5 more
top venuebenchmark similarity 0.70
Abstract · Accepted to EMNLP 2026 (Main Conference)

A benchmark score credits final answers, but not the route by which an item can be answered. In medical multimodal multiple-choice questions (MCQs), this distinction matters because a correct answer can be supported by the intended image finding or by benchmark-preserved cues in the wording of answers, non-visual clinical text, visible image text, artificial annotations, or device/context artifacts. We call the resulting score-level overinterpretation reasoning inflation. Here, a route is an observable input path that can support answer selection, not a claim about the model's hidden cognition. Across six medical multimodal MCQ datasets, we separate candidate cues from behavioral evidence through prompt- and image-side audits, modality ablations, and matched repairs that preserve the medical target and answer key. In a 13-configuration open-model panel, full-input accuracy is 62.63%, while text-only and options-only settings achieve 53.96% and 29.71%, respectively. Removing length-gap, absolute/conspicuous, and spatial/prepositional cues lowers accuracy by 6.58, 3.50, and 4.77 percentage points. We also construct MedQA-MM, a 1,000-item shortcut-mitigated subset, where text-only and options-only accuracy fall to 5.21% and 12.33%. This does not imply that models never use images; it shows that medical image-reasoning claims require route-level evidence.

0.58

Learning Where to Look: Anatomical Grounding and Guided Attention for Cardiac MRI Vision-Language Models

cs.CV ·2026-09-30 ·Bangwei Guo, Xiao Chen, Boris Mailhe, Jia Yao, Yiqing Wang +6 more
similarity 0.69
Abstract

Cardiac magnetic resonance imaging (CMR) enables assessment of cardiac anatomy, ventricular function, and myocardial tissue characteristics. Clinicians interpret these images by identifying cardiac structures and focusing on the regions relevant to each clinical question, motivating anatomically guided vision-language models (VLMs). Yet CMR-specific supervision for anatomical localisation and clinical question answering remains limited. To address this gap, we investigate fine-grained CMR visual question answering through anatomical grounding and guided attention. We construct 128,915 anatomical-grounding and 42,799 clinical QA pairs across short-axis cine, late gadolinium enhancement, and long-axis cine. These datasets support anatomical recognition, localisation, and clinical assessment without requiring paired reports for individual training images. To help the model learn where to look, we introduce Cardiac Anatomy-Routed Attention (CARA), which selects predicted anatomical priors according to the question and guides decoder attention with learned task-specific strengths. Combining anatomical grounding pretraining with CARA yields our model, CARA-VL. Experiments demonstrate CARA-VL's strengths in clinical assessment and regional localisation across CMR imaging settings, with promising generalization to an external clinical cohort. Together, our data and method provide a practical framework for studying and advancing cardiac visual understanding in VLMs. We will release the QA data derived from public datasets upon publication.

0.37

MedICL: Modality-Anatomy-Aware Visual In-Context Learning for Medical Visual Question Answering

cs.CV ·2026-10-14 ·Ji-jun Park, Soo-joon Choi, Seungmin Han, Haeun Kwon, Jiwon Jeong +1 more
similarity 0.69
Abstract

Training-free adaptation of medical vision-language models (LVLMs) is a deployment requirement, yet visual in-context learning (ICL) injects modality- and style-mismatched demonstrations into medical contexts, teaching a frozen model the wrong answering style. We present MedICL, which curates demonstrations instead of retraining the model. MADS selects modality-anatomy aligned demonstrations that cover complementary interpretation dimensions; W2C curates them with weak retrieval signals and a strong verification vote that checks demonstrations rather than the model's own outputs; ECC answers with W anchored demonstration sets, aggregates by abnormality-aware signatures, and re-answers once on conflict. On VQA-RAD, SLAKE, and PathVQA under the LLaVA-Med zero-shot protocol, MedICL improves the strongest zero-shot baseline by +11 - +14 points on open-ended and +7 - +14 on closed questions, recovers 71% of fine-tuned open-ended performance on VQA-RAD at zero training cost, and stays robust to shifted or corrupted demonstration banks.

0.14

Enhancing RAG-based Medical VQA with Gated Multimodal Representations

cs.CV cs.CL ·2026-10-16 ·Daniela Jimenez-Pano, Hugo Jair Escalante, Delia Irazú Hernández-Farías
benchmark similarity 0.69
Abstract · Preprint submitted to ECIR

Medical Vision-Language Models (Med-LVLMs) have shown strong performance in Medical Visual Question Answering (MVQA), yet they may still produce factually inconsistent responses. Retrieval-Augmented Generation (RAG) can mitigate this issue by incorporating external clinical evidence; however, retrieval is often guided by a single modality, potentially overlooking the joint semantics of medical images and clinical questions. We introduce a multimodal RAG framework that learns a joint query representation through a Gated Multimodal Unit, enabling evidence retrieval to leverage information from both modalities while keeping the underlying Med-LVLM frozen. We evaluate the proposed approach on benchmark datasets. Results show that the multimodal query representation consistently outperforms a reference Med-LVLM and remains competitive with state-of-the-art methods. Our findings demonstrate the potential of multimodal query representations for more effective evidence retrieval in Medical VQA.

0.17

Multimodal AI in Healthcare: Review of Vision-Language Foundation Models for Real-World Medical Applications

cs.CV ·2026-11-15 ·Taha Razzaq, Murtaza Taj, Asim Iqbal
published similarity 0.69
Abstract · 20 Pages, 3 figures, 3 Tables

The emergence of foundation models has marked a transformative shift in AI, enabling robust generalization across diverse downstream tasks through putative zero-shot learning. Large Language Models and Vision Language Models have demonstrated strong capabilities in tasks such as image interpretation, report generation, and question answering by effectively learning from multimodal data-images paired with associated text often with minimal supervision. In the healthcare domain, this ability to align visual and textual information reduces the reliance on extensive manual annotations, as models can leverage existing clinical reports and imaging data to learn meaningful representations. This integration holds promise for improving diagnostic support, treatment planning, and overall patient care, even in data constrained settings. In this review, we provide a definitive taxonomy of the medical VLM landscape, tracing the evolution from early Contrastive Alignment and Generative MLLMs to the cutting edge frontiers of Dense Pixel Grounding, Sparse Mixture-of-Experts (MoE), and Reasoning Incentivized (RL) architectures. We critically examine the "medical bottleneck" identifying the persistent challenges of data scarcity, the "hallucination" risks in generative diagnostics, the computational strain of 3D volumetric processing, and the lack of standardized, clinically grounded evaluation metrics.

0.44

MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering

cs.CL ·2026-09-03 ·Erfan Nourbakhsh, Ke Yang, Anthony Rios
top venue similarity 0.68
Abstract · Accepted to EMNLP Findings 2026

Medical visual question answering (Med-VQA) is often assumed to require medical fine-tuning, large models, or complex multi-agent pipelines. We revisit this assumption with \textbf{MedProb}, a lightweight probing framework that predicts multiple-choice Med-VQA answers from frozen VLM representations without free-text generation. Across PATH-VQA, SLAKE, and VQA-RAD, MedProb recovers substantially more answer-relevant signal than prompting and performs stronger than medical VLMs and agentic systems. Probing also reduces the apparent gap between small and large models compared to prompting, suggesting that smaller VLMs contain more recoverable Med-VQA signal than generation-based evaluation reveals. Across 14 matched general-purpose and medical VLM pairs, medical adaptation does not consistently improve this linear decodability. Finally, free-text generation exhibits an answer-position bias of up to 10 percentage points, whereas MedProb also has positional bias, however, it is impacted differently than prompting. Our main results target the multiple-choice/multiclass Med-VQA setting; we additionally show the probe can be extended to open-ended generation via a rejection-sampling scoring procedure.

0.33

Representation-guided in-context learning for medical image interpretation with multimodal large language models

cs.AI cs.CLcs.CV ·2026-09-21 ·Minda Zhao, Fangyu Hu, Yan Luo, Yutong Yang, Jiahui Cai +6 more
similarity 0.68
Abstract

Medical image interpretation is central to diagnosis and care, yet adapting general-purpose multimodal large language models (MLLMs) often requires resource-intensive domain-specific fine-tuning. Here we introduce representation-guided in-context learning (RG-ICL), a training-free inference framework that retrieves query-aligned demonstrations using frozen encoders, without task-specific parameter updates. Across eight datasets spanning histopathology, radiology and retinal fundoscopy, RG-ICL improved classification (mean gain 20 percentage points) and visual question answering (VQA) (mean gain 13 percentage points) over no-context and conventional ICL, approaching or exceeding training-based comparators. Which cases were retrieved mattered more than how many: 6 query-aligned cases outperformed up to 32 randomly selected ones, whereas fixed or random cases often reduced accuracy below baseline. For VQA, aligning reference cases with both image content and question intent produced further gains. These findings indicate that for medical image interpretation, curating which reference cases an MLLM sees is a practical alternative to retraining it.

0.16