CS Paper Radar

Supervision-Aware Evaluation of MedGemma for Reference-Assisted Radiotherapy Contour Ranking

Jaden Moon, Caiwen Jiang, Songlin Zhao, Wei Liu

eess.IV cs.CVcs.LG ·submitted 2026-11-11 11:50 UTC · v1

Abstract

Radiotherapy contour ranking can support quality assurance, but vision-language model (VLM) gains may reflect supervision, score construction, available inputs, or representation access. We evaluated frozen MedGemma 1.5 in a single-institution study, a 12-patient held-out HaN-Seg test, five-fold cross-fitting on 30 non-test HaN-Seg patients, and transfer without refitting to 120 SegRap2023 patients. On held-out HaN-Seg, a supervised readout of the final prompt state exceeded a supervised readout of the model's yes/no output probabilities by +0.3047 pooled average precision (AP; 95% patient-bootstrap interval [+0.2822, +0.3269]); the advantage persisted against a parameter-count-matched nonlinear control. A readout of projected visual tokens exceeded the final-state readout in cross-fitting and transfer (+0.0706 AP averaged equally across patients on SegRap; 95% interval [+0.0607, +0.0804]), while a structured panel-feature readout remained competitive with visual tokens. The anatomy-conditioned readout did not demonstrate an advantage over the native-feature comparator. These results motivate visual-token and structured-feature controls in medical-imaging VLM evaluations; outcomes were computational or synthetic, not clinical acceptability.

Comment
5 pages, 1 figure, 3 tables. Submitted to IEEE ISBI 2027

Closest papers

Nearest neighbours by meaning across the whole index.

Authority-Preserving Evaluation of Medical Vision-Language Assistants

cs.CV cs.AI ·2026-09-14 ·Flint Xiaofeng Fan, Cheston Tan, Yew-Soon Ong, Roger Wattenhofer
publishedbenchmark similarity 0.62
Abstract · Accepted at 2nd Emerging LLM/LMM Applications in Medical Imaging (ELAMI) 2026, held in conjunction with MICCAI 2026. To appear in the Spring…

Medical vision-language models can propose how urgently a skin lesion should be reviewed, but the local service retains authority to accept or replace that proposal under referral policy, capacity, and locally held patient context. Proposal quality and selected-action quality are therefore distinct evaluation targets, and benchmark evidence transfers between them only when local review preserves the expected action score. We introduce AuthEval, a logging and evaluation framework that records both actions, scores the selected action under declared local criteria, and, where feasible, scores the declined proposal under the same rule. It reports the resulting authority gap only when the record supports it. Because the gap is the product of the proposal-change rate and the mean score change on changed cases, that rate alone determines neither its magnitude nor its sign. On ISIC 2019, with MedGemma and simulated local review, two constraint regimes with similar change rates produced an optimistic image-equal gap under capacity ($+0.744$ simulator units) but no detectable gap under safety. The declared evaluation unit also mattered: under mixed constraints the gap reversed from $+0.374$ to $-0.206$ when weighting shifted from image to lesion-aware cluster. AuthEval thus clarifies whether a study's records support claims about the model, the workflow, or both.

0.44

RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching

cs.CV ·2026-09-01 ·Charles Corbière, Léo Machado, Aubin Charley, Baptiste Callard, Pierre Manceron +1 more
publishedcode/data similarity 0.60
Abstract · Accepted to ECCV 2026 Workshop on Medical Foundation Models and Benchmarks

As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. Large language model (LLM)-based metrics are now the best-correlated with radiologist judgment, yet they output a single opaque score that neither a clinician nor a model builder can easily interpret or audit. We introduce RadMatch, a multi-stage, LLM-based metric that decomposes report comparison into a structured finding-level matching with significance-aware scoring and error characterization across seven clinical attribute dimensions (status, location, severity, morphology, certainty, longitudinal comparison, and measurement). The main score is the actionable-error count, both interpretable and auditable. Candidate findings are graded correct, partial, or incorrect, and unmatched findings are counted as missed or hallucinated. Triage and actionable safety recall/precision and per-subset views add complementary, deployment-oriented lenses. Across two expert benchmarks, RadMatch is the most clinically aligned metric, matching inter-radiologist agreement on ReXVal and more than doubling the best prior metric on the harder RadEvalExpert. Relying only on few-shot prompting, it is designed to extend to other modalities and anatomies. We will release RadMatch as open-source code with an interactive dashboard for inspecting results.

0.53

OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents

cs.AI physics.med-ph ·2026-10-27 ·Humza Nusrat, Kundan Thind
benchmark similarity 0.59
Abstract · 25 pages, 4 figures, 1 table, and the supplementary material (19 tables, 5 figures). Ancillary files: the paper in Markdown, structured resu…

Background and purpose: Radiotherapy planning agents based on large language models (LLMs) decide when to submit a plan or request review but are evaluated mainly by dose. This study developed OpenGray, a benchmark of planning tasks with known correct decisions. Materials and methods: OpenGray adds tasks with known answers to the OpenKBP-Opt head-and-neck dataset, including two contradictions with dose goals proven incompatible and a feasible control with a stored plan meeting every hard goal. Four systems, from a rule-based planning script to a full LLM agent, were evaluated on 30 patients, the LLM-based ones with a commercial (Terra) and a local (Ministral 3 8B) LLM. Decisions, achieved dose, problem recognition, and technical failures were recorded and also read from logs of 39 stereotactic radiosurgery runs of SAGE, a clinical LLM agent. Results: With Terra, the script preceded by an LLM review and the full agent had decision accuracies of 95.4% and 92.1%, respectively, yet escalated only 50% (95% confidence interval, 25-75%) and 25% (10-40%) of overlap contradictions, which required arithmetic on the anatomy. All four systems submitted every feasible control, but 24% of these plans missed a hard goal that the stored plan had met. In four SAGE runs, one goal failed at every attempt; the agent identified it in every reply but had no action for stopping. Conclusions: Favorable aggregate scores concealed specific decision failures. Planning agents should be evaluated on their decisions as well as their dose and given an explicit action for stopping and requesting review.

0.49

Scaling large language model-assisted line-of-therapy adjudication with human-in-the-loop evidence-calibrated selection

cs.LG q-bio.QMstat.AP ·2026-11-15 ·Olatunji Oni, Sanhita Sengupta, Marc De Benedetti, Nathanial Eddy
similarity 0.58
Abstract · 12 pages, 4 figures

Large language models (LLMs) have enabled the ability to scale clinical adjudication tasks traditionally dependent on exhaustive manual review. Line-of-therapy (LoT) adjudication, a major bottleneck in oncology cohort construction, is a prime example. Line boundaries must be inferred from longitudinal treatment histories according to nuanced, study-specific definitions, at every regimen transition for every patient. LLMs can apply this adjudication at scale, but their reliability varies across patients. Treatment histories differ in complexity, and ambiguous or atypical patterns depend on expert clinical judgment. Reliable automation thus requires verified support for model assignments and the ability to identify cases that warrant manual review. Here, we present evidence-calibrated selection (ECS), a human-in-the-loop framework for scaling LLM-assisted LoT adjudication. ECS derives confidence from observed clinician adjudications, prioritizes patients with the least-supported assignments for follow-up review, and returns completed adjudications to the model as precedent. As the evidence base grows, ECS continuously calibrates confidence to estimate residual error, thus enabling more efficient use of clinician time. In simulations on 136 clinician adjudicated patients with multiple myeloma, ECS achieved a residual error of 5% after review of 44.6% of patients, compared to 62.2% when review was directed by LLM self-reported confidence and 81.9% under random sampling. These results support a scalable approach to study-specific LoT adjudication, using clinician evidence to inform automated predictions, minimize error, and reduce review burden.

0.21

What Language Adds, and at What Cost: Four Controlled Probes of Pathology Vision-Language Models

cs.CV q-bio.QM ·2026-10-19 ·Wenhao Zhang, Zhongliang Zhou, Yi Ping, Zirui Wang, Sheng Li
similarity 0.58
Abstract

Vision--language models (VLMs) promise to make computational pathology queryable and able to draft reports, but how much language adds beyond the vision backbone is unclear, because almost every system is evaluated by its developers on test sets of their own choosing. We grade published evidence by how it was produced and test four claims with controlled probes that reuse released checkpoints and public data, varying one input at a time. Report text adds prognostic signal: adding a frozen embedding of the stage-redacted report to a slide embedding raises the pooled C-index by 0.019 for progression-free interval and by 0.016 for overall survival across 1{,}633 TCGA patients, and restoring the stage statement adds nothing. Prompts add no measurable accuracy at inference: with features and pooling fixed, zero-shot classification is statistically indistinguishable from a supervised linear probe for lung and renal cancer subtyping. Slide-level question answering is barely grounded: SlideChat answers 74.82\% of SlideBench-VQA(TCGA) items correctly with the slide and 74.41\% without it, whereas its language backbone alone reaches 43.52\%. Paraphrasing leaves accuracy unchanged but flips 8--23\% of answers. Self-reported gains from agentic navigation and few-shot language priors cannot yet be attributed to language. Language thus adds real but narrow value, and we propose minimum controls that would expose these failure modes in future evaluations.

0.18

Anatomy-Change-Aware Bidirectional Selective State-Space Memory for Clinically Deployed Thoracic Radiotherapy Auto-Contouring

eess.IV cs.CV ·2026-09-11 ·Galib Ahmed, Istiak Ahmed, Aritra Islam Saswato, Asib Mostakim Fony, Kazi Shahriar Sanjid +13 more
similarity 0.58
Abstract

We developed DAMM-Net++, a 2.5D architecture for thoracic OAR and target volume segmentation that addresses three persistent challenges in radiotherapy auto-contouring: inter-slice surface incoherence, systematic failure on small low-contrast targets, and the absence of per-case reliability signals. The central component is an anatomy-change-aware bidirectional selective state-space memory that models through-plane anatomical change and selectively propagates context along the axial slice sequence. A boundary-aware decoder sharpens near-surface predictions, and an uncertainty head provides calibrated per-voxel confidence for clinical triage. We evaluated 2,146 patients across four centers, an independent external cohort of 112 patients, and a multicenter reader study involving 17 radiation oncologists on 305 cases. The model achieves a mean Dice of 0.955 and HD95 of 3.78 mm, with the largest gains on low-contrast organs-at-risk (OARs) and target volumes where through-plane context is most critical. The uncertainty head is well-calibrated and supports case-level triage. In the reader study, AI assistance reduced contouring time by 75-80% across experience levels and raised junior-reader IoU from 0.861 to 0.925, matching the unedited model. External validation showed a modest internal-to-external drop (< 5%) with calibrated uncertainty transferring without recalibration. The complete deployment pipeline from DICOM ingestion to TPS-compatible RTSTRUCT export has been integrated into the clinical workflow at a partner hospital, where it is used to assist with contouring. These results suggest that anatomically motivated inter-slice memory, paired with uncertainty-guided review, offers a clinically viable path for thoracic auto-contouring.

0.20

Impact of Pretraining Length, Model Size, and Annotation Budget on Downstream Clinical Tasks of Foundation Models in Radiotherapy

cs.CV ·2026-11-17 ·Alejandro Cortina Uribe, Sidsel Højklint Poulsen, Ivan Richter Vogelius, Jens Petersen
code/data similarity 0.58
Abstract · 9 pages, 5 figures, 2 tables. Supplementary Material included

Cone-beam CT (CBCT) scans acquired during radiotherapy capture changes such as tumor shrinkage and radiation-induced tissue changes, offering great potential for foundation models that help monitor individual treatment responses. However, to our knowledge, no large-scale pretrained models for radiotherapy CBCT exist, and evaluating modeling choices is difficult since downstream tasks are not known in advance. We examine how pretraining length, model size, and finetuning strategy relate to downstream clinical performance, and how metrics tracked during pretraining correlate with it. We pretrain vision transformer encoders with masked image modeling on about 150,000 raw 3D radiotherapy CBCT scans, tracking representation quality via validation loss, effective rank, and linear probing. Downstream tasks are classification of feeding tube presence ($n=51,135$ CBCTs, $1,562$ patients) and clinical characteristics (T-, N-stage, sex, age, tumor site; $n=1,278$ patients) in head and neck cancer. We compare low-rank adaptation (LoRA) and full finetuning against no pretraining, varying annotated dataset size. Pretext validation loss was the most consistent predictor of downstream performance; effective rank and linear probing were also associated, but not consistently across model sizes and finetuning methods. LoRA performed comparably to full finetuning while training only 0.5-2.5% of the parameters. Fully finetuned models consistently outperformed models without pretraining across all sizes, with AUROC 0.91-0.97 vs. 0.58-0.85 for feeding tube classification and 0.99-1.00 vs. 0.71-0.97 for sex classification. Model size had only a small, inconsistent effect, suggesting smaller models suffice at lower computational cost. Code and pretrained models (non-commercial license): https://github.com/alexCortinaU/RT-CBCT_SSL

0.48

Personalising Deformable Image Registration with Clinical Context and Deformation History for Adaptive Radiotherapy

eess.IV cs.AI ·2026-10-13 ·Caiwen Jiang, Songlin Zhao, Yuzhen Ding, Jia Mi, Samir H. Patel +21 more
benchmark similarity 0.57
Abstract · 27 pages, 7 figures and 6 tables. The submission includes Supplementary Information

Adaptive radiotherapy must account for anatomical changes during treatment, but doing so depends on reliably mapping targets, organs at risk and dose across serial images. Existing registration methods treat each image pair independently, without using radiotherapy-specific context or the patient's preceding anatomical trajectory. We developed Clinical-context-Aware and Patient-adaptive Deformable Image Registration (CAP-DIR), integrating source contours, dose, clinical text and prior-registration features. We evaluated 16,361 CT examinations from 3,473 patients across four institutional cohorts and five public benchmarks. CAP-DIR increased internal-test organ-at-risk-region normalized cross-correlation from 84.76% to 96.92% relative to the strongest learning-based comparator, with low deformation folding, and remained superior across seven external cohorts. In a blinded review of 20 longitudinal cases, clinically acceptable contours increased from 45.0% to 82.5% for clinical target volumes and from 22.5% to 62.5% for organs at risk, while estimated editing time decreased. In a controlled benchmark, CAP-DIR reduced global dose-mapping error by 34.5% relative to the strongest comparator. These findings show that clinically informed, longitudinal registration can improve contour usability and dose mapping for adaptive radiotherapy.

0.37