CS Paper Radar

EviDial: When Correct Depression Predictions Lack Evidence

Junjie Xu, Jin Zhu, Yuzhe Yang, Luhan Wang, Xingjiao Wu, Daoguo Dong, Jing Yang, Wenxin Wu, Liang He

cs.CL ·submitted 2026-10-20 02:56 UTC · v1
benchmarkhot direction

Abstract

Depression dialogue benchmarks commonly assess models by comparing predicted symptom labels with clinical scale scores. However, a prediction may agree with a benchmark label even when the dialogue lacks enough evidence to justify it. We introduce EviDial, a benchmark that distinguishes label agreement from evidence-grounded assessment. EviDial comprises 382 depression dialogues from three sources and 2,744 participant-symptom cells, each classified as having absent, partial, conflicting, or sufficient evidence. Because evidence sufficiency lacks a directly observable gold standard, we derive calibrated evidence references from multiple judgments using generalizability theory, many-facet Rasch measurement, and random-effects latent class analysis. Our audit reveals that 82.8% of participant-symptom cells lack sufficient textual evidence for the corresponding symptom assessment, with substantial variation across sources. Across model families and prompting strategies, 69.2%-96.9% of predictions judged correct by benchmark labels are nonetheless unsupported by sufficient dialogue evidence. Further analyses show that strong agreement among LLM evaluators need not reflect independent evidence judgments, underscoring the need to calibrate automated reference panels. These findings demonstrate that label-based evaluation can overestimate evidence-grounded performance by conflating agreement with clinical labels and support in the observed dialogue. EviDial provides a measurement layer for separating these constructs and auditing LLM-based depression assessment more faithfully.

Closest papers

Nearest neighbours by meaning across the whole index.

EDA: Observe, Verify, and Assess for Evidence-Grounded Multimodal Depression Analysis

cs.AI ·2026-11-17 ·Changzeng Fu, Ming Shang, Xinyi Zhang, Jiadong Wang, Runhao Zeng +1 more
similarity 0.71
Abstract

Existing multimodal depression assessment methods often summarize long recordings into global behavioral descriptions. This leaves the source and decision relevance of individual observations difficult to inspect. We propose EDA, an evidence-grounded framework organized around Observe, Verify, and Assess. In Observe, literature-informed acoustic and visual cues are localized into source-linked Evidence Records with temporal intervals and behavioral measurements. In Verify, frozen multimodal agents examine whether the cited media support each observation. The simplified Support admission policy retains positively verified records and their review grounds. In Assess, statistical descriptors and temporal events support reference-based behavioral comparison, while PHQ-9-guided text analysis provides complementary symptom evidence. Cross-modal disagreement triggers review of the original text before final selection. Experiments on D-Vlog and LMVD evaluate human-referenced cue agreement and downstream assessment utility separately. AI cue annotations achieve 91.02% accuracy against human consensus labels on shared source segments. With a frozen 7B backbone, EDA Support improves F1 over direct prediction by 7.61 and 9.88 percentage points on D-Vlog Leave-Query-Out and LMVD, respectively. These findings support explicit, source-linked behavioral evidence for inspectable multimodal depression assessment.

0.24

Candidate Generation and Definition-Guided Verification for Sentence-Level Depression Symptom Recognition

cs.CL ·2026-09-01 ·Weiming Li, Catarina Barata, Miguel Constante, Joao Sanches
publishedhot direction similarity 0.69
Abstract

Sentence-level recognition of depression symptoms is challenging because similar expressions can differ in symptom relevance, and language-model inference is insufficiently grounded in diagnostic definitions. This study proposes a two-stage framework separating symptom-candidate generation from definition-grounded verification. A contrastively fine-tuned sentence encoder generates a symptom candidate per sentence, and a fine-tuned language model verifies whether the candidate is present or absent using the sentence, its context, and a candidate-specific diagnostic definition, checking its judgment against that definition before answering. Evaluated against encoder, inference-based, medical, and general LLM baselines and a matched single-stage supervised classifier, the proposed pipeline attains the best accuracy and F1 scores of all methods, with rationales matching expert-authored annotations. A preliminary clinical audit indicates moderate alignment with diagnostic definitions, with explanation quality strongly dependent on prediction correctness. The results support decomposing symptom recognition into candidate generation and definition-grounded verification, though performance remains limited for rare categories.

0.45

DepressionAgent: Reading, Listening, Seeing, and Deliberating Multimodal Evidence for Depression Risk Assessment

cs.HC ·2026-08-14 ·Fangjie Zhu, Haifeng Lu, Sicheng Zhao, Runhao Zeng, Xiping Hu
similarity 0.67
Abstract

Multimodal depression risk assessment requires jointly interpreting textual, acoustic, and visual cues that are often subtle, non-specific, context-dependent, and potentially inconsistent across modalities. Existing multimodal approaches predominantly learn latent representations through feature fusion, leaving the evidence underlying a prediction and the treatment of cross-modal disagreement largely implicit. We propose DepressionAgent, an evidence-centric agentic framework that transforms multimodal depression assessment from implicit feature fusion into explicit evidence deliberation. DepressionAgent first converts textual, acoustic, and visual inputs into modality-specific evidence, and then organizes self-report and behavioral evidence into parallel support--challenge deliberation branches. Cross-modal arbitration explicitly examines agreement and disagreement between the two branches, with conflict reflection revisiting inconsistent assessments before decision making. A subsequent risk reflection mechanism provides an independent textual second opinion for initially low-risk cases to reduce potentially missed risk signals. Without depression-specific supervised training or parameter fine-tuning, DepressionAgent achieves competitive performance on multiple public benchmarks. Extensive ablations, cross-model evaluations, qualitative analyses, and clinician assessments further demonstrate the effectiveness and inspectability of the proposed framework.

0.04

SIGMA: speaker-isolated, symptom-structured multimodal estimation of depression severity from clinical interviews

cs.SD cs.AIcs.LG ·2026-11-17 ·Zhuoyuan Tang, Tao Wang, Kai Yang, Li Yuan, Yuezhou Zhang +6 more
similarity 0.67
Abstract

Clinical interviews are central to depression assessment but time-consuming and rater-dependent. Automated severity estimation offers scalable screening, yet existing methods often retain interviewer speech (confounding participant cues), fuse modalities late (losing fine-grained cross-modal dynamics), or score symptoms independently (ignoring inter-symptom correlations). We present SIGMA, a speaker-aware, symptom-structured framework for depression-severity estimation from semi-clinical interviews. SIGMA uses speaker diarization to restrict input to participant speech, cross-attention so that text, audio and video condition one another during encoding, and a graded-response head that predicts the eight questionnaire items jointly through a shared latent severity trait with item-specific ordered thresholds. On the Extended Distress Analysis Interview Corpus, SIGMA achieves state-of-the-art performance (CCC 0.762, RMSE 4.13, MAE 3.30 over five seeds). Ablations confirm that isolating participant speech and joint, structured symptom prediction each substantially improve concordance and estimation accuracy.

0.12

Interpretable Symptom Vectors for Depression in a Large Language Model

cs.CL cs.AIcs.LGq-bio.NC ·2026-09-01 ·Fangyi Zhu, Ajay Subramanian, Allison Constant, Camille Wang, Ravish Gupta +1 more
similarity 0.66
Abstract · 26 pages, 6 figures

Patients with depression present with diverse symptom profiles, yet clinical practice routinely reduces this variation to a single severity score. Large language models (LLMs) can potentially capture various symptoms and their severity from patient speech. However, how depressive symptoms are represented inside LLMs remains poorly understood, limiting clinical trust. To examine whether internal model activations match clinician judgment, we analyzed the residual stream of Gemma-3-27B-PT using mechanistic interpretability techniques. Recording activations across symptom descriptions drawn from validated clinical instruments, we found that symptom groups geometrically separated the most at layer 21 across multiple distance metrics. Using Semantic Projection, we then projected held-out naturalistic text onto Symptom Vectors constructed from these instruments. The resulting per-symptom coefficients preserved clinician-annotated rank ordering across mood, somatic, and suicidality axes. Furthermore, a single depression vector in Layer 21 separates held-out depressive from non-depressive text (AUC = 0.789), which can be used as an emotional valence gate that restricts symptom projection to depressive speech. These results reveal a decorrelated, clinician-aligned symptom signal readable directly from internal activations, offering a mechanistic foundation for interpretable depression-assessment tools.

0.58

From Relevant Posts to Attributable Spans: Explicit Symptom Evidence for Depression Questionnaire Inference

cs.CL ·2026-10-14 ·Sumanth Manduru, Carlotta Domeniconi
similarity 0.65
Abstract

Longitudinal social-media histories offer an opportunity to assess depression, potentially supporting the inference of standardized depression questionnaire responses. However, this task requires evidence that is semantically relevant to a symptom and attributable to the user, while being sufficiently specific to support an individual questionnaire item. Existing approaches rely on semantic similarity to retrieve posts, which can introduce misleading or non-attributable evidence, particularly in multi-symptom posts and discussions of others' experiences. We propose SREE (Symptom Retrieval and Evidence-Grounded InfErence), an evidence-grounded pipeline that extracts first-person and symptom-specific text spans, selects a representative subset using a Facility Location-inspired objective, and performs Evidence-Gated Inference over the selected evidence. We evaluate evidence retrieval on BDI-Sen and evidence-conditioned questionnaire inference on eRisk 2019-2021. Compared with state-of-the-art retrieval approaches, SREE achieves relative improvements of 70% in post-level precision and 59% in F1, while retaining at least 30% fewer posts. On multi-symptom sentences, it also outperforms dense retrieval in precision and F1 even with twice the retrieval budget. Across all three eRisk collections, SREE achieves the highest exact item-level agreement while remaining competitive on ordinal and aggregate metrics. These results highlight the importance of localizing attributable evidence and controlling its composition before inferring depression questionnaire responses.

0.19

LongCounsel-8: A Benchmark Suite for Longitudinal Depression Tracking from Multi-Session Counseling Dialogues

cs.LG cs.AI ·2026-09-03 ·Jiayi Li, Bingsheng He, Zhaomin Wu
benchmarkcode/data similarity 0.64
Abstract · 25 pages. Dataset: https://huggingface.co/datasets/hiddensev/LongCounsel-8

Large language models (LLMs) are increasingly used to assess depression in counseling conversations, where psychology evidence suggests that tracking change across sessions matters more than any single score. However, existing dialogue benchmarks score each session on its own, and an accurate score for every session does not guarantee that a method follows the changes between sessions. We introduce LongCounsel-8, a benchmark for tracking depression across counseling sessions. Its systematic pipeline takes client profiles from real counseling cases, depression scores from real longitudinal data, and turn length and form from real counseling dialogues, yielding 7,749 five-session trajectories from three LLMs. Every session is generated to express a preset score on the PHQ-8, a standard eight-symptom depression questionnaire. Validation shows that the labels are self-consistent and the dialogues close to real counseling. Following clinical research on treatment monitoring, we propose a trend-based evaluation protocol reflecting real-world tracking needs: changes between sessions, crossings of the screening cutoff, trajectory types, and symptom-level changes. Extensive experiments show all five existing methods tested fall short in tracking trends: none estimates changes better than a no-change baseline, at most 40% of cutoff crossings are detected, and adding earlier sessions lowers change error but causes more missed improvements.

0.64

From Dialogue to Diagnostic Hypotheses: Evaluating LLM Agents for Psychiatric Assessment

cs.AI ·2026-10-05 ·Milad Mohammadi, Fatemeh Akrami Shamsabadi, Zahra Mohseni, Amirhossein Safdarian, Malekfarhad Malek +2 more
similarity 0.64
Abstract · 48 pages, 29 figures, 21 tables; includes appendices

Although LLMs show promising diagnostic reasoning on supplied clinical evidence, their ability to elicit such evidence through free-form psychiatric interviews remains less understood. We evaluate interview quality and diagnostic hypotheses using simulated patients and human participants. For controlled simulation, we introduce PsyCPG, transforming clinical cases into structured patient profiles. On 30 paired cases, experts rate PsyCPG simulations higher in quality than raw-case role-play (4.50 versus 3.83/5). We introduce PsyCIDRA, a dual-agent framework: its interviewer agent uses a notepad tool, expert-written skills, and ICD-11 reference tools to guide interviews; its diagnostic agent analyzes the transcript and produces a report for expert review, ranking differential diagnoses with supporting, conflicting, and missing evidence. Across four models and 53 development cases, PsyCIDRA achieves higher observed diagnostic agreement than direct prompting. On 81 held-out simulated cases, exact-code rank-1 accuracy reaches 60.5% versus 51.9%. We conduct 101 interviews with human participants; two psychologists, blinded to interview condition and model reports, independently assess interview quality and transcript-supported hypotheses. In separate arms, PsyCIDRA agrees with experts on whether to issue a hypothesis in 79.6% of cases versus 65.4% for direct prompting. This study provides an empirical basis for understanding and evaluating LLM agents in free-form psychiatric interviewing.

0.16