CS Paper Radar

Faithful Explanations under Missing Modalities in Multimodal Dementia Prognosis

Firuz Kamalov

cs.AI ·submitted 2026-10-26 13:34 UTC · v1

Abstract

Large language models are increasingly used to turn the feature attributions of multimodal prognostic models into narratives for clinicians, yet costly modalities such as amyloid PET and cerebrospinal fluid biomarkers are often missing, and whether these explanations stay faithful under missing evidence has not been tested. We separate two failure points: phantom attribution, in which the model credits modalities that were never observed, and unfaithful narration, in which the report omits, cites or misjudges missing modalities. Using 307 participants of the Alzheimer's Disease Neuroimaging Initiative with complete data, we masked every subset of four costly modalities, predicted 36-month conversion from mild cognitive impairment to Alzheimer's disease, and restored the masked data to obtain interventional ground truth. All five missing-data strategies assigned 12% to 23% of attribution to unobserved modalities, and up to 45% with four absent; missingness-aware strategies minimised it at no cost in accuracy. Across 51,200 reports from four language models and four prompting regimes, naming absent modalities in the prompt raised acknowledgement from 48% to 88% and cut phantom evidence twenty-fold, but advice on whether acquiring a missing test would change the prediction was correct in only 27% to 41% of cases until checked against executed restorations, which raised accuracy to 69% for open-weight narrators and 99.9% for a proprietary model. Higher phantom attribution was associated with more phantom evidence and fewer faithful direction claims, and direction fidelity, measured by a calibrated language-model judge, depended on narrator scale rather than prompting. Narrators told to name absent modalities sometimes declared observed ones absent, an error that a structured template eliminated. We release the masking protocol and evaluation code.

Closest papers

Nearest neighbours by meaning across the whole index.

CoPoE: Multimodal Fusion via Decomposable Disease-Coordinate Product-of-Experts for Missing-Modality Alzheimer's Diagnosis

cs.LG cs.CV ·2026-10-08 ·Chihun An, Ikbeom Jang
publishedbenchmarkcode/data similarity 0.63
Abstract · Accepted at IEEE BIBM 2026

Multimodal Alzheimer's disease (AD) diagnosis benefits from integrating heterogeneous clinical, imaging, genomic, and biomarker evidence, but clinical cohorts frequently suffer from irregular modality missingness. Existing fusion methods often synthesize absent inputs, risking the introduction of artificial surrogates, or pool available signals into uninterpretable latent spaces. We present CoPoE (Disease-Coordinate Product-of-Experts), a disease-coordinate framework that maps multimodal evidence into a structured latent space partitioned into four distinct biological and clinical axes: genetic Risk, molecular Pathology, Neurodegeneration, and clinical Stage (R/P/N/S). Each observed modality parameterizes a diagonal Gaussian expert over the full RPNS vector, and a masked Product-of-Experts architecture fuses only the available modalities. Consequently, absent modalities add no factor to the fusion path, allowing the network to preserve a robust, decomposable posterior for any non-empty modality subset without synthetic imputation in the RPNS path. Through extensive missing-modality experiments on the ADNI dataset, CoPoE achieves the best all-modality performance and the highest mean AUROC across all 15 observed-subset evaluations among standardized missing-modality fusion baselines under a shared non-PET ADNI embedding benchmark, while substantially improving raw-probability ECE, Brier score, and NLL. Furthermore, PET-supervised probing shows evidence enrichment within the pathology (P) block under full modalities, with tau-related signal retained even when direct fluid biospecimen inputs are withheld. Our code is available at https://github.com/labhai/CoPoE.

0.44

TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment

cs.CV cs.AI ·2026-08-31 ·Merna Bibars, Bolaji Omofojoye, Allan I. Levey, Rachel Hershenberg, Gari D. Clifford +1 more
similarity 0.62
Abstract · 19 pages, submitted to IEEE Transactions on Affective Computing

Depression and anxiety in older adults with Mild Cognitive Impairment (MCI) are frequently underdiagnosed due to limited access to care. Multimodal analysis of remote clinical interviews is a scalable screening approach, but existing methods have three limitations. First, they do not correct temporal misalignment across multimodal features extracted at different resolutions, inducing spurious cross-modal associations. Second, remote recordings exhibit uneven modality dropout, but missing values are often zero-filled, making them indistinguishable from valid near-zero measurements. Finally, they do not jointly attribute predictions to modalities, questions, and interview moments, limiting fine-grained clinical interpretation. We propose a Temporally-Aligned, Missingness-Aware, Interpretable (TAMI) multimodal fusion framework. TAMI aligns speech, language, facial, and physiological features within question-answer segments on a shared timeline, encodes modality-level missingness over time, and conditions fusion on question context. In interviews with 49 older adults with MCI, TAMI achieved area under the receiver operating characteristic curve (AUROC) scores of 0.68 (depression) and 0.69 (anxiety). Fine-grained temporal alignment of multimodal features produced the largest performance gain ($Δ{\geq}0.1$). Multi-level interpretability analysis revealed that depression classification relied on eyegaze and open-ended questions, while anxiety classification depended on eyegaze and head pose, with attribution uniformly distributed across questions. Using only responses to the open-ended questions (5.1min), the depression model achieved an AUROC score of 0.67, which was not significantly different from using the full interview (19min) ($p>0.05$). Our findings support designing interview protocols centered on open-ended questions for depression screening in older adults with MCI.

0.30

Mitigating Cross-Modal Hallucinations by Correcting Modality Misrouting in Omnimodal LLMs

cs.AI ·2026-10-13 ·Yu Mi, Licheng Pan, Jiayi Zhu, Qirui Liu, Yitong Qiao +5 more
similarity 0.62
Abstract · 21 pages, 10 figures

Omnimodal large language models (OLLMs) are susceptible to cross-modal hallucinations under audio-visual interference. We find that, for a substantial fraction of incorrect joint predictions, the question-relevant unimodal branch can still recover the correct answer, suggesting that task-relevant evidence can remain recoverable from the relevant modality even when it is not reflected in the joint prediction. We refer to this recoverable failure pattern as \emph{modality misrouting}. We propose CaRe (Conflict-aware Residual Intervention), a training-free inference-time method for identifying and correcting modality misrouting. CaRe uses a text-only modality probe to identify a candidate modality-specific counterfactual branch, and flags potential misrouting when its prediction conflicts with the original joint prediction. For such cases, CaRe computes a residual direction from the discrepancy between the counterfactual and fused representations and uses it to steer the original multimodal state toward the relevant modality evidence, while retaining both modalities in the original audio-visual inference path. The intervention strength is adapted based on the confidence of the counterfactual branch. Experiments on CMM and AVHBench across two OLLMs show that CaRe consistently outperforms the evaluated inference-time hallucination-mitigation baselines.

0.19

AdaptAD: Foundation Model Adaptation with Missing Modality Robustness for Alzheimer's Disease Staging Across the Disease Continuum

cs.CV cs.LG ·2026-10-19 ·Dawa Chyophel Lepcha, Sophie A. Martin, Pierrick Coupé, Shabbir Syed Abdul
similarity 0.61
Abstract · 44 pages, 11 figures, 6 supplementary figures

Staging Alzheimer's disease (AD) across its continuum, from cognitively normal (CN) through mild cognitive impairment (MCI) to dementia from a single structural MRI remains difficult because 3-D networks overfit small cohorts, and FreeSurfer morphometry is often unavailable. We present AdaptAD, which reads a registered T1-weighted scan as 16 anatomically placed 2.5-D slices through the medial temporal lobe and adapts a frozen ImageNet-pretrained ViT-B/16 with low-rank adaptation (LoRA), training 3.5 M of 89 M parameters. Morphometry enters as tokens when present, or as a learned [MISSING] token, and a missing-aware stacker fuses both experts. Under a pre-registered protocol, ten methods were compared for three-class CN/MCI/AD staging on identical 5-fold splits of 597 ADNI participants, 29% without FreeSurfer, with Holm-corrected paired bootstrap tests. The locked model was tested once on 993 OASIS-3 participants (988 scored by every method). On ADNI, AdaptAD achieved a three-class macro-AUC of 0.745 (95% CI 0.709 to 0.781), ranking first and significantly exceeding full fine-tuning, ShaSpec and a 3-D CNN. On OASIS-3 it reached 0.780 (0.748 to 0.814), and on the CDR-based label (0.804) it significantly outperformed every 3-D and image-tabular method. LoRA matched full fine-tuning with 25 times fewer trainable parameters, and removing morphometry changed external macro-AUC by less than 0.003. The disease score tracked MMSE and CDR-SB within diagnostic groups and stratified MCI-to-dementia progression (log-rank p < 0.001), while saliency concentrated in medial temporal and periventricular regions. A parameter-efficient 2-D foundation model can therefore stage the full AD continuum from a routine T1 scan without morphometry, offering a scalable route to memory-clinic deployment.

0.20

ITU-DESC: a missingness-aware machine-learning dementia risk score in the UK Biobank cohort

cs.LG ·2026-10-19 ·Abdullah Kavaklı, Faik Boray Tek
similarity 0.61
Abstract

Background: Existing dementia risk scores rely on fixed, pre-selected features and linear models, or ignore bias from informative missingness in cohort data. Here we introduce the Istanbul Technical University Dementia Risk Score (ITU-DESC), an explainable machine-learning score for incident all-cause dementia that combines broad feature selection across roughly 8,000 candidate features with explicit control of missingness bias. Results: We studied approximately 500,000 UK Biobank participants with a median follow-up of 16.6 years. We withheld missingness patterns from the model using unconditional random hot-deck imputation. We stabilized feature selection by repeating BorutaSHAP-based feature selection with ten random-imputation seeds and selecting predictors chosen in at least eight of the ten seeds, which yielded 38 predictors. We trained a CatBoost gradient-boosted decision-tree model and evaluated it on a holdout test set of 50,041 participants. For all-cause incident dementia, ITU-DESC achieved an average precision of 0.138 (95% confidence interval 0.123-0.156) and an area under the receiver operating characteristic curve (ROC AUC) of 0.853 (0.844-0.863), significantly outperforming other risk scores, including the strongest comparator, the UK Biobank Dementia Risk Prediction model (0.118 and 0.841). We identified dementia risk predictors absent from established scores, including leukocyte telomere length and liver enzymes. Conclusions: ITU-DESC is the first dementia risk score to combine broad feature selection with explicit control of missingness bias.

0.23

I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models

cs.LG cs.CLcs.CV ·2026-09-07 ·Aydin Javadov, Daniel Schoess, Florian von Wangenheim
top venue similarity 0.60
Abstract · Accepted at VLM4RWD at NeurIPS 2026

Vision-language models (VLMs) are increasingly used in settings where some input modalities may be unavailable, yet we know little about whether they can faithfully explain how such missing information affects their own predictions. We introduce an interventional protocol for evaluating self-explanations of modality dynamics: models state what each modality alone would support, whether restoring a missing modality would change their answer, and whether the available evidence is sufficient; we then execute the corresponding intervention and compare these claims with realized behavior. We evaluate ten VLMs spanning open-weight and proprietary models across four tasks covering mixed, redundant, and unique modality regimes. We find a systematic tendency to overstate the sufficiency of available evidence. Models substantially underestimate the effect of restoring missing modalities: executed change exceeds predicted change in 78 of 80 model-task-condition settings, with task-level median executed change rates reaching 70.1\% while median predicted rates remain at most 9.6\%. Insufficiency claims have low recall, leaving many cases in which behavior changes despite a stated claim of sufficiency. Retrospective self-explanations show the same tendency, over-crediting single-input sufficiency in mixed regimes and interchangeability in redundant ones. Together, these results show that VLMs systematically mischaracterize how their predictions depend on available and missing evidence, motivating executable interventions as a behavioral test of multimodal self-explanations.

0.49

Being judged through the record: A Rosenhan-inspired test of psychiatric label sensitivity in large language models

cs.CY ·2026-10-26 ·Shaked Rosenblum, Teddy Lazebnik, Ziv Ben-Zion
similarity 0.60
Abstract

The integration of large language models (LLMs) into clinical decision-making raises unresolved questions regarding diagnostic bias. We investigate whether a documented psychiatric history alters LLM recommendations when the current clinical evidence is identical. Using a counterfactual experiment inspired by Rosenhan's 1973 field study, we factorially varied diagnostic-history condition (no prior diagnosis stated vs. prior schizophrenia diagnosis) and current-evidence condition (voices ceased vs. voices persisted) within a clinical vignette. Across five LLMs (GPT-5.6 Sol, Claude Sonnet 5, Gemini 3.6 Flash, Grok 4.6, and Mistral Large 2512), we generated 20,000 model responses across independent queries. When evidence supported discharge, adding a prior schizophrenia diagnosis increased recommendations for continued hospitalization nearly fivefold, from 4.7% to 22.3% (risk difference, 17.5 percentage points; 95% CI, 16.2-18.8). This shift occurred across all five models (risk-difference range, 10.4-34.4 percentage points). In the same condition, reference concordance decreased from 95.3% to 77.7%, while stated decision confidence increased. Across all model-by-evidence contrasts, confidence increases corresponded to Cohen's d values of 0.35-1.09. Signal-detection analyses showed that the prior diagnosis shifted the decision criterion toward hospitalization across all five models and reduced discrimination between the two current-evidence conditions in three of them. When current evidence supported continued hospitalization, the label increased the recommendation for hospitalization from 92.6% to 97.0%.

0.17

Behind the [MASK]: Disentangling Representation and Faithfulness in DAPF-Based Dementia Detection

cs.CL cs.LG ·2026-08-25 ·Pardis Ranjbar-Noiey, Natalie Parde
similarity 0.60
Abstract · 16 pages, 1 figure, 19 tables. Under review at ACL Rolling Review

Spoken-language analysis via prompt-based domain-adaptive models is a promising direction for low-resource, non-invasive dementia screening, but such models remain internally opaque. We study the interpretability of the Domain-Adapted models via Prompt-based Fine-tuning (DAPF) framework, which casts dementia detection as diagnosis-related masked-token prediction. We interpret DAPF and strong baselines using a variety of probing and analysis techniques, finding that DAPF achieved the best overall performance (accuracy=0.83 and macro-F1=0.83) with diagnosis most recoverable from its [MASK] representation. However, this representational advantage did not extend to token-level explanation faithfulness. DAPF attributions primarily reflected language task vocabulary, discourse markers, and transcription artifacts, with perturbation tests showing weak or negative effects. This suggests that its masked-token interface determines diagnosis information without producing faithful token-level explanations.

0.43