CS Paper Radar

MemoCare: An Interactive Multimodal Mobile System for Automated Cognitive Screening

Duy-Cat Can, Mau Minh Phuc Le, Tuan-Khoa Hoang, Hai-Dang Nguyen, Trung-Hieu Do, Dang Minh Ly, Minh-Duc Nguyen, Nghia TT Hoang, Nguyen Linh Trung, Huy-Hieu Pham, Huong Ha, Binh T. Nguyen, Oliver Y. Chén

cs.MM cs.CVcs.HC ·submitted 2026-10-07 17:19 UTC · v2
published

Abstract

MemoCare is a React Native mobile system for automated multimodal cognitive screening in English and Vietnamese. It integrates speech, temporal and spatial orientation, touchscreen interaction, and visuoconstruction. Google Speech-to-Text transcripts are scored with deterministic, task-specific rules. GPS-grounded current and historical administrative names support location scoring, touch events verify actions, and lightweight convolutional neural networks evaluate drawings. A separate model provides visual interpretation. Software verification passed 151/151 predefined language, spatial-answer, and touch-scoring regression cases and 48/48 fixed-coordinate spatial tests across four countries. Transcript-based tests exclude speech recognition accuracy. On a locked 71-image test set, validation-selected ShuffleNetV2 x1.5 achieved 91.33% mean balanced accuracy and 78.87% exact three-criterion accuracy. Four clinician co-authors rated the workflow (pooled median 4/5 across eight criteria; item medians 3-4.5), highlighting improvements needed in speech handling, response acceptance, and result transparency. The workflow automatically produces item-level and total scores.

Comment
8 pages, 1 figure, 1 table. Accepted at the demonstration track at MMM 2027

Closest papers

Nearest neighbours by meaning across the whole index.

A Visual Information Extraction System Using Multimodal Large Language Models for Scanned Cognitive and Behavioral Assessments

cs.AI ·2026-11-17 ·Enshuo Hsu, Linh Nguyen, Sunyang Fu, Heath Goodrum, Sandra Stansberry +3 more
similarity 0.67
Abstract

Objective. Paper-based assessments are common in patient care, but handwriting and scanning artifacts make these forms difficult to process at scale. This study proposes a modular, multimodal LLM-based system with human oversight for extracting fields and scores. We conducted a qualitative analysis of the use of LLMs for hand-drawing analysis. Materials and Methods. The system processes cognitive assessments (i.e., MMSE, MoCA, Mini-Cog) and the NICHQ Vanderbilt Assessment using two modules: 1) a form classifier that assigns a scanned page to a form type, followed by a corresponding 2) score extractor that extracts fields and scores into a schema. A human-in-the-loop design utilizing dual-model validation engages human annotators when two LLMs disagree, limiting manual review to challenging pages only. Results. End-to-end accuracy for cognitive assessments was 88%, with most form types exceeding 90%. For the NICHQ Vanderbilt, overall accuracy reached 97.5%, with all form types above 96%. The human-in-the-loop design further improved cognitive assessment accuracy to 97.5% while reducing manual work by 63.8%. In the Mini-Cog Clock Drawing task, qualitative analysis showed that 62.3% of all extraction errors stemmed from ambiguous drawing, in which cases the LLM provided reasonable readings. Discussion. Compared to current manual data extraction, our LLM-based system reduces processing time by 10 to 20 times and costs by over 12 times. The modular design allows seamless expansion to additional form types. Conclusion. Our system provides an accurate, scalable, and cost-efficient solution for structuring data trapped in handwritten paper forms for clinical decision support.

0.27

NarraSpace: Multimodal Scene-Grounded Evidence for Picture-Description Dementia Classification

cs.CL ·2026-10-20 ·Qingming Li, Ruiyan Sun, Satoshi Nakamura
benchmark similarity 0.63
Abstract · 30 pages, 3 figures

Picture-description dementia classification uses a shared visual scene, but a transcript-level prediction alone does not identify the scene evidence underlying its inputs. NarraSpace links words and speech times to named picture elements and retrieves utterances for predefined scene events. Narrative-spatial descriptors summarize coverage and temporal organization. A frozen language model contrasts each event's text under examples from Alzheimer's disease and healthy-control training participants, alongside transcript surprisal and coverage measurements. Retained source records link event scores to their source utterances, speech intervals, and lexical matches. On the official 48-participant test set of the Alzheimer's Dementia Recognition through Spontaneous Speech benchmark, a fixed support vector classifier achieves $80.88\%\pm2.56\%$ mean accuracy over fifty seeds; averaged probabilities yield $81.25\%$ accuracy and $91.32\%$ area under the receiver operating characteristic curve. NarraSpace makes the construction of scene-linked classification inputs explicit and inspectable by connecting event-indexed language-model measurements to their source utterances, speech timing, and extraction records. Such source-linked measurements could support clinician review of picture descriptions by presenting quantitative summaries alongside the original speech evidence.

0.21

Why Alzheimer's Speech Screening Fails to Generalize: Bridging the Deployment Gap via Cross-Corpus Evidence Anchoring

eess.AS cs.CLcs.SD ·2026-09-25 ·Zijian Lu, Sizhe Liu, Yin Zhang, Jixuan Deng, Xinrong Lin +4 more
published similarity 0.61
Abstract · Accepted to NCMMSC 2026

Speech-based screening is a promising, non-invasive approach for detecting Alzheimer's disease and related cognitive risks. However, models trained on a single domain often generalize poorly to unseen languages, tasks, or recording protocols. This paper investigates this deployment gap using a leave-one-corpus-out evaluation across four distinct datasets. Among 70 interpretable speech and language features, 59 exhibit direction conflicts between healthy control and cognitive risk groups across corpora, with pause, silence, and speech rate showing high protocol sensitivity. Furthermore, while the XLM-R text baseline achieves strong average performance, its Area Under the ROC Curve (AUC) drops to 0.520 on the weakest held-out domain. A standard GroupDRO baseline reaches a 0.766 mean speaker AUC and a 0.504 worst-domain AUC under the same protocol. To address this, we propose a fusion method that integrates XLM-R text baseline scores with evidence anchors selected during training. Balanced fusion achieves a 0.785 mean speaker AUC, while anchor-heavy fusion raises the worst-case speaker AUC to 0.615. This work highlights the need to audit feature transferability and report worst-case domain robustness in cognitive speech screening.

0.28

Improved automatic speech-based cognitive assessment using neuropsychologically calibrated diagnostic labels

cs.SD ·2026-10-20 ·Madhurananda Pahar, Caitlin Illingworth, Adam Hampshire, Peter J Hellyer, Agata J Hampshire +3 more
similarity 0.58
Abstract

Although automatic detection of cognitive impairment from conversational speech is a promising way, it becomes less accurate, as large real-world datasets often contain unreliable diagnostic labels due to self-referral and limited clinical verification, and here we investigated whether neuropsychologically calibrated diagnostic relabeling improves it. We analysed 138 hours of conversational speech from 720 participants, recalibrating unreliable diagnostic labels using attention, executive function, and memory performance via external clinical thresholds, Gaussian, and normative z-score calibration. Transformer-based classifiers were trained using both original and recalibrated labels and evaluated internally on 150 clinically verified participants and externally on Pitt, ADReSS and PROCESS-2 datasets. Memory- and MoCA-based z-score calibration produced the greatest improvements. Using our internal dataset, $F_1$ increased from $0.70$ to $0.83$ ($+13\%$) and from $0.54$ to $0.66$ ($+12\%$) for binary and multiclass classification, respectively. For external validation, the increases are $+4\%$, $+5\%$ and $+9\%$ on Pitt, ADReSS and PROCESS-2 corpus. Neuropsychologically calibrated relabeling substantially improves speech-based cognitive assessment, highlighting label quality, rather than model architecture, as a critical bottleneck in clinical AI for dementia screening.

0.25

Task-Centric Multimodal Learning for Cognitive Impairment Detection

cs.LG cs.CL ·2026-10-14 ·Sankalp Vaish, Rishank Singh, Sumukh Veeramalla, Caroline Summerour, John A. Batsis +1 more
published similarity 0.58
Abstract · Accepted for publication at the 2026 IEEE Global Communications Conference (GLOBECOM)

Dementia affects millions of older adults worldwide, yet early detection remains difficult outside clinical settings. Traditional models primarily rely on text or voice-based responses, capturing only a limited view of human cognition despite the inherently multimodal nature of human expression. This work proposes a contactless, multimodal screening framework that leverages video, audio, and text signals collected during structured voice-based interactions with Amazon Alexa. By incorporating these modalities, our approach captures a more holistic representation of human behavior encompassing speech patterns, vocal characteristics, facial expressions, and linguistic structure, which may enable more accurate detection of cognitive decline. Different participants completed three conversational tasks designed to probe spontaneous speech, comprehension, and recall. We pursued three objectives: (1) evaluate which of the three Alexa tasks provides the most informative cognitive signals, (2) compare the effectiveness of unimodal models, and (3) assess the benefits of multimodal fusion models. Task-level analysis shows that the video model achieves its best performance on Command-Reading, with an accuracy of 0.65 (AUC 0.69), while the audio model performs best on Question-Answering, reaching an accuracy of 0.67 (AUC 0.73). In contrast, the multimodal model demonstrates consistently stable performance across tasks, achieving an accuracy of 0.76 (AUC 0.75) on Command-Reading and 0.76 (AUC 0.73) on Question-Answering. These findings demonstrate that multimodal behavioral cues captured through naturalistic interactions can reliably distinguish MCI from healthy aging, highlighting the limitations of traditional diagnostic paradigms and supporting the shift toward multimodal, real-world cognitive assessment frameworks.

0.35

KIRA: A Full-Stack Reading Screening System with Dual-Pathway Analysis for Children

cs.CL cs.LG ·2026-10-20 ·Maggie Dong, Rohit Mekkoth, Zixuan Wang, Cheol Jun Cho, Jiachen Lian +1 more
published similarity 0.58
Abstract · Accepted to IEEE Spoken Language Technology 2026

Early-grade reading screening is critical for identifying children at risk of reading difficulty. Existing AI-based screeners are often limited in the scope of their analyses and evaluation metrics, or are proprietary systems that lack transparent model specifications and evaluation. We present KIRA, a dual-pathway screening system that pairs word-level fluency scoring with detailed phoneme-level error detection. We adapt pretrained word and phoneme recognizers to children's speech, reducing word error rate from 13.88% to 8.84% and phoneme error rate from 41.66% to 23.00%. Powered by these child-adapted models, we implement Word Reading (WRE) and Oral Reading Fluency (ORF), two foundational tasks in K-2 screening. The system integrates these recognizers with a web-based teacher-facing interface presenting fluency scores alongside phoneme feedback for potential targeted intervention. KIRA brings modern child-speech recognition into a full-stack, deployable, classroom-ready reading screener.

0.30

Fine-Tuning a Transducer for Impaired Speech: Where the Gains Appear and What They Mean for Evaluation

cs.CL ·2026-11-05 ·Kien Nguyen, Tuan Dinh, Bao Phan
published similarity 0.58
Abstract · 6 pages, 4 tables. Accepted at the SAPC2 Competition Workshop at NeurIPS 2026

Speech recognition still works poorly for many people whose speech is affected by conditions such as ALS, Parkinson's disease, cerebral palsy, Down syndrome or stroke. We took part in the Speech Accessibility Project Challenge 2 (SAPC2) with a public speech recognizer that we fine-tuned on the challenge data. Our best entry scored 7.81% character error rate on Test1, the first of two hidden test sets (official baseline 15.26%). Our live entries ran on CPUs with a combined median latency of about 0.9 seconds at 7.96-8.07% error. On new speakers, fine-tuning reduced errors most for the speakers that an off-the-shelf recognizer finds hardest to understand. Sentences already seen in training had far lower error rates than new ones (2.58% vs 6.42%), a larger difference than being a familiar speaker made. We close with lessons for evaluating such systems.

0.28

Cross-Linguistic Acoustic Biomarkers of Cognitive Impairment Across Interactive and Single-Speaker Speech Tasks

cs.AI eess.AS ·2026-11-13 ·Kenyu Ikeuchi, Tomoaki Ohtsuki, Michelle Cohn, Alyssa Weakley, Yui Ishihara +5 more
published similarity 0.57
Abstract · 7 pages, 1 figure, 5 tables. Accepted for publication in Proc. 48th Annual International Conference of the IEEE Engineering in Medicine and …

Multilingual dementia detection is essential for scalable and equitable screening. In recent years, automated speech analysis has emerged as a promising approach for supporting dementia detection by enabling non-invasive and scalable assessment of cognitive function. However, existing studies that use automated speech analysis often rely on one type of task, such as picture description, and use black-box models. These studies often neglect the role of task and language. To address these gaps, we performed a cross-linguistic and cross-task acoustic analysis that integrates a conversational dataset with structured speech tasks across four languages (Spanish, Mandarin, English, Japanese). We applied statistical, linguist-/clinician-guided, and machine learning model-based feature selection to derive compact acoustic feature sets while maintaining classification performance. This enabled systematic comparison of cognitive impairment-relevant features across datasets and tasks. Our results revealed that acoustic features did not generalize uniformly across tasks and languages. Temporal features remained robust across all datasets, consistently capturing timing disruption regardless of language or task. In contrast, speech rate showed strong task dependence -- emerging as informative mainly in single-speaker tasks with externally anchored content and pacing, but becoming less reliable when speech content was highly individualized. Pitch-related features further exhibited language-specific patterns, favoring chroma-related features in tonal languages (e.g., Mandarin) and F0-related features in non-tonal languages. These findings caution against indiscriminate pooling of heterogeneous speech data and underscore the need for interpretable, context-aware approaches to multilingual dementia detection using automated speech analysis.

0.17