CS Paper Radar

Calibration and Conformal Prediction for Medical Question Answering in Arabic and English: Evidence from Two Non-Parallel Benchmarks

Mariya

cs.LG ·submitted 2026-11-12 13:57 UTC · v1
code/data

Abstract

Large language models (LLMs) are increasingly tested on medical exam questions, but accuracy alone does not show when an answer can be trusted. I study the calibration and uncertainty of two open-weight models, Qwen2.5-7B-Instruct and Jais-adapted-7B-chat (both run in 4-bit precision), on 2,816 English multiple-choice questions from MedMCQA and 4,927 Arabic multiple-choice questions from MedAraBench. Accuracy was 0.561 (English) and 0.430 (Arabic) for Qwen, and 0.369 and 0.308 for Jais. Raw option probabilities were overconfident in all four settings (expected calibration error 0.242 to 0.486), and a single temperature parameter removed 82% to 93% of that error. Split conformal prediction reached its nominal coverage in all 24 settings I tested (within 0.006), but the sets were large when accuracy was low: at 90% coverage, temperature-scaled sets contained on average 63% (Qwen, English), 78% (Qwen, Arabic), 86% (Jais, English) and 89% (Jais, Arabic) of the options. The margin between the top two option probabilities separated correct from incorrect answers best for Qwen on English (AUROC 0.734) and worst for Jais on Arabic (0.579). Answering only the 20% most confident questions lowered Qwen's English error from 0.439 to 0.098. Jais showed a strong prior for option C (49% of English and 60% of Arabic predictions). Scoring Latin letters only cut the Arabic share to 29% but left accuracy, raw ECE and AUROC nearly unchanged. Because the two benchmarks differ in subject, level and number of options, I do not attribute differences between languages to language itself. Code is available at https://github.com/Mariyaaleem/arabic-english-medqa-uncertainty.

Comment
15 pages, 1 figure, 12 tables. Code: https://github.com/Mariyaaleem/arabic-english-medqa-uncertainty

Closest papers

Nearest neighbours by meaning across the whole index.

Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation

cs.CL ·2026-10-23 ·Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee +1 more
benchmarkhot direction similarity 0.65
Abstract

When a language model is asked whether its own answer is correct, three languages are involved: that of the question, that requested for the answer, and that of the verifier instruction asking for the judgment. Multilingual confidence evaluations seldom separate the three. We vary them independently in a six-language factorial design with ten open-weight models (3B-14B) on LSQA, MMMLU and TriviaQA, holding each answer fixed across verifier languages and measuring confidence with a first-token P(True) probe. The three languages move different parts of the evaluation: which of them governs accuracy depends on the benchmark, matching the response language to the question or verifier language sharpens discrimination but not calibration, and the verifier language alone shifts calibration error by up to 0.04 without changing accuracy. The scorer matters as well: an unvalidated judge would have shrunk the MMMLU English-Hindi accuracy gap from 0.198 to 0.038. Multilingual confidence results, model rankings included, are therefore comparable only when all three languages are reported and the scorer is validated.

0.29

Medical question answering collapses in Hausa: a pre-registered pilot on three small open models

cs.CL ·2026-11-16 ·Abbas Sani
benchmark similarity 0.64
Abstract · 10 pages, 5 figures

A system that reads nothing and always answers the most frequent letter scores \BaselineP{} on our Hausa medical multiple-choice set. \NBelowBaseline{} of the \NModels{} small open models we evaluate do not beat it. We lead with that because the usual summary of a multilingual result -- a percentage-point drop from English -- does not say whether what survives the drop is performance at all. On a paired English--Hausa set of \NEval{} MedMCQA items, three instruction-tuned models of 3--4B parameters lose between \GapMedGemma{} and \GapQwen{} points of accuracy, every gap significant after Holm correction. Back-translating the Hausa items into English recovers between half and two-thirds of the gap for the two general-purpose models but not demonstrably for the medically adapted one, so the deficit is substantially, though not wholly, a matter of the language of presentation. Answers are also far less stable: agreement across three option orderings falls from \ConsEnQwen{} to \ConsHaQwen{} in the worst case. An exploratory error analysis of \ENRows{} discordant items attributes most of the damage to medical vocabulary lost in translation (\ETerm{}, 95\% CI \ETermLo{}--\ETermHi{}), usually in the question stem. A fourth registered hypothesis proposed closing the gap with self-generated preference pairs. Its pilot gate found nothing to train on: asked to reason in Hausa rather than to pick a letter, the model echoes the prompt back, switches into Swahili, or simply stops without answering; only \GateParse{} of chains are parseable. We report that intervention as not run, which is itself the sharper result -- a forced-choice benchmark cannot see a model that has no generative command of the language at all. All hypotheses, thresholds and decision rules were registered before any model was run, and all deviations are logged.

0.23

Benchmarking decision models against generative large language models on medical multiple-choice questions

cs.CL cs.AI ·2026-10-28 ·Oscar Freyer, Daniel Glauert, Moritz Hofmann, Max Ostermann, Stephen Gilbert +1 more
similarity 0.64
Abstract

Medical workflows often require selecting among predefined options. We compared 13 decision models with 15 generative large language model configurations, including different reasoning settings, using 1,273 four-option MedQA questions under text-only conditions. Accuracy ranged from 67.71% to 89.08% for decision models and from 78.32% to 96.86% for generative configurations. Among the tested configurations, Mercury Decide had the lowest mean final-attempt cost (USD 0.0069 per 1,000 questions), while Clef Omni had the shortest median request duration (0.291 seconds). Decision-model probabilities distinguished correct from incorrect answers, but error rates among answers assigned probabilities of at least 0.90 ranged from 1.71% to 12.77%. Answer agreement across three runs ranged from 92.30% to 100%, while probability repeatability varied. Decision models offered resource advantages alongside lower peak accuracy; selecting models for medical workflows requires considering probability quality and repeatability together with accuracy and the resources needed to detect and correct errors.

0.22

One Confidence Threshold, Unequal Coverage: Language Gaps in Typed Decision Interfaces and How Translating Inputs Reduces Them

cs.CL cs.AIcs.LG ·2026-10-17 ·Junwei Yao, Yongxi Zhou, Zike Yang, Zhijun Wang, Jianan Liu +2 more
similarity 0.63
Abstract

Deployed classifiers often act on a prediction only when its confidence clears a threshold and pass the rest to a person. We ask whether one threshold serves all languages equally on typed decision interfaces, APIs that return a probability for each option of a fixed multiple-choice question. In a confirmatory study with a written analysis plan, three vendors' interfaces classified 1,000 MASSIVE items, each in 51 languages, with a threshold of 0.9. Even when a request is answered correctly in both English and another language, the English version clears the threshold more often, by 8.6-11.7 percentage points on average. Observed coverage, the share of requests handled automatically, is lower than English's in every other language, while the requests the gate handles are rarely wrong. The gap remains when every language is reweighted to the same accuracy, and a standard calibration metric (ECE) does not detect it on one interface. In exploratory analyses, English's coverage stays higher when each language gets its own threshold fitted to the same 5% error target. Translating inputs into English nearly closes the gap at 0.9 but leaves English ahead at that error target. Requests written natively in African languages show a gap in the same direction when English is their human translation. We recommend reporting per-language coverage at the operating threshold.

0.32

Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination

cs.CL cs.AIcs.CYcs.LG ·2026-10-22 ·Miguel Fernandez-de-Retana, Unai Zulaika, Ruben Sanchez-Corcuera, Aritz Bilbao-Jayo, Aitor Almeida
hot direction similarity 0.63
Abstract · 13 pages, 8 figures

Accuracy on medical examinations is common evidence offered for the clinical use of language models, yet a health system that intends to run an open model on its own hardware needs to know considerably more: whether the model recognizes when it is likely to be wrong, whether medical fine-tuning improves this ability, whether its correct answers rest on sound clinical reasoning, or whether it actually uses the images it is given. We examine these questions for four open models of 8 to 27 billion parameters on the Spanish MIR residency examination, scoring 1,552 questions from eight editions under the exam's own rules. Answering every question, the three larger models would rank among roughly the first 600 to 3,000 candidates, with 41 to 43 specialties still open to them. Abstention, however, never pays, because even their least confident answers are right more often than the 25% break-even. Medical fine-tuning reduced overconfidence at the cost of accuracy, without improving how well confidence separates right from wrong answers. Moreover, 13-38% of correct answers rejected a distractor for a clinically unsound reason that confidence did not reflect, and removing or swapping the provided contextual images left accuracy unchanged.

0.34

From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models

cs.AI cs.CL ·2026-10-30 ·Navapat Nananukul, Mayank Kejriwal
benchmark similarity 0.61
Abstract

Large language models (LLMs) now answer medical exam questions at or above expert level, but it is unclear whether such scores reflect clinical competence, or are instead affected by the format of the questions being asked (e.g., multiple-choice versus open-ended), prior exposure to the benchmarks, and the difficulty of the questions. These benchmarks are also public and widely used online, which may be resulting in LLM memorization. Models may perform better on questions that they have seen before. It is also not evident whether smaller models, which are more locally deployable in HIPAA-compliant hospital settings, have achieved human performance. In this study, we evaluate fourteen current models from six leading providers, one smaller and one larger model from each provider with two additional open-weight models, on 3,076 questions from four medical benchmarks (MedQA-USMLE, MedMCQA, PubMedQA, and MedBullets) under standard multiple choice, the same questions with the answer choices withheld, and reconstruction of a hidden question from its choices alone. In the standard multiple-choice setting, the strongest models exceed the human-expert baseline on MedQA-USMLE and PubMedQA, but the smaller models trail their larger models by 8.7 percentage points and only two of the six pass the human-expert bar on the licensing examination. Withholding the answer choices, however, makes every model struggle: mean accuracy falls from 82.3% to 50.9%, and the two largest drops belong to the medically specialised models. On memorization, the strongest models reconstruct one question in four from the answer choices alone, yet accuracy holds on the questions they cannot reconstruct. We further weigh accuracy against inference cost and test whether ensembling several models improves medical question answering.

0.15

When Confidence Fails: Overconfidence in LLMs under Uncertainty and Missing Clinical Information

cs.CL cs.AIcs.HCcs.LG ·2026-08-10 ·Maryam Tahermazandarani, Adnan Mahmood, Fahmida Islam, Quan Z. Sheng
published similarity 0.61
Abstract

Large Language Models (LLMs) have achieved strong performance in medical question answering and clinical reasoning tasks. However, their reliability under uncertainty remains poorly understood which raises critical concerns for deployment in high-stakes clinical settings. In such environments, incorrect predictions are inherently risky, but confident incorrect predictions can be particularly harmful as they may mislead clinical decision-making. In this paper, we conduct a systematic behavioral analysis of LLMs under clinical information uncertainty. We propose an evaluation framework based on the MedMCQA dataset consisting of two complementary uncertainty settings. First, we introduce linguistic uncertainty cues through prompt modifications to simulate ambiguous clinical contexts. Second, we construct an answer removal setting, wherein the correct option is deliberately excluded mandating the model to recognize insufficient information and abstain. We analyze both model accuracy and confidence behavior using multiple calibration metrics including calibration gap, Expected Calibration Error (ECE), and Unsafe Confident Error Rate (UCER) across 500 medical questions. Our results reveal a consistent failure mode, i.e., although accuracy degrades under increasing uncertainty, model confidence remains misaligned with accuracy. This leads to a substantial increase in unsafe confident errors, indicating that model confidence remains largely insensitive to clinically meaningful information loss. Furthermore, we observe significant variation across models in their ability to abstain when the correct answer is unavailable, with some models persistently producing high confidence hallucinated answers. These findings expose critical limitations in the epistemic reliability of current LLMs and highlight the need for uncertainty aware evaluation methods prior to their deployment in clinical workflows.

0.22

An Empirical Study of Development-Tuned Representation Sensitivity for Medical QA Data Selection

cs.CL cs.AI ·2026-09-09 ·Yuexin Wu, Dayou Yu, Vasile Rus
published similarity 0.61
Abstract · 16 pages. Substantially revised GenAI4Health @ NeurIPS 2026 version: updated title, development-tuned question-only selection protocol, expe…

We study medical question-answering (QA) data selection when a developer already has questions, labels, and auxiliary explanatory texts but can retain only $B$ examples for a final LoRA update. Representation-Sensitivity Prioritization (RSP) first restricts the candidate bank to its top-$4B$ predictive-entropy examples, then combines the frequency of answer flips with non-flipping contraction of the top-two answer margin under hidden-state perturbations. We evaluate this operational score on five medical QA datasets with google/medgemma-4b-it, $B=128$, and three seed branches. After removing four MedExpQA test questions that also occur in its acquisition pool, dataset-specific development-tuned RSP attains the highest observed five-dataset mean accuracy (53.44%) and Macro-F1 (48.21%) among the evaluated selectors. Relative to random selection within the same entropy-restricted pool, the descriptive differences are +1.49 and +2.10 percentage points, respectively; paired 95% bootstrap intervals include zero. The audit mixes benchmarks with different relationships to documented backbone post-training sources, and the same test cohorts had been used in earlier analyses before the expanded tuning grid was designed. Results are therefore exploratory, modest, and dataset-dependent rather than evidence of statistical superiority. RSP point estimates also exceed an archived no-update reference, but unpinned model revisions and incomplete historical input identifiers limit attribution of that difference to fine-tuning. Because every selected training example includes auxiliary text and no matched answer-only control is evaluated, the experiments also do not identify a causal benefit from auxiliary-text conditioning itself.

0.58