CS Paper Radar

Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Posts

Andrew Aquilina, Xiang Lorraine Li, Yu-Ru Li

cs.CL ·submitted 2026-08-29 21:23 UTC · v1

Abstract

Judgments about psychological distress are socially situated: what counts as concerning hinges on community norms around emotional expression, vulnerability, and help-seeking. Yet large language models (LLMs) used for distress detection are typically aligned to a single, undifferentiated standard. How well do these models capture the perspectives of the communities whose language they assess? We address this question through a perspectivist annotation study in which 321 participants provided 9,587 judgments on 1,198 Reddit posts spanning six identity-based communities, yielding community-specific labels. Raters in the contextualized in-group condition show a modest tendency to agree more with their community than uncontextualized out-group raters (OR = 1.18), an effect varying significantly across communities. We then evaluate nine open-weight LLM configurations and four frontier configurations against these labels. Open-weight LLMs systematically over-estimate distress: when communities perceive none-to-mild distress, these models achieve only 31-44% accuracy, predominantly producing false positives. GPT-5 and Gemini 2.5 Pro show the same none-to-mild inflation even when their full-sample over/under rates are mixed, while Claude Opus 4 is more conservative. This pattern does not simply mirror an outsider reading position: uncontextualized out-group human aggregates were nearly symmetric, with 18% over-estimation versus 19% under-estimation. Instead, the models that inflate none-to-mild cases exhibit a distress prior that exceeds both contextualized in-group and uncontextualized out-group human judgments. These findings have implications for equitable AI deployment in mental health contexts, where miscalibrated distress detection may unevenly affect the communities being assessed.

Closest papers

Nearest neighbours by meaning across the whole index.

Veterans' Mental Health Crisis Severity Annotation: Do LLMs Agree More Than Human Experts?

cs.AI ·2026-11-05 ·K M Sajjadul Islam, Pranav Yaramalla, Nadiyah Johnson, Mark Flower, Praveen Madiraju +1 more
publishedcode/data similarity 0.64
Abstract · Accepted in IEEE Big Data 2026

Large language models (LLMs) offer a scalable approach for annotating mental health and crisis-related social media content, but their reliability in safety-critical tasks remains unclear. This study evaluates LLM-based crisis-severity annotation of veteran-related social media posts against annotations from eight Veteran Subject Matter Experts (SMEs). Using 149 tweets and five severity categories, we evaluate 15 open-source and 10 proprietary LLMs across eight prompt variants, examining annotation performance, inter-model agreement, prompt sensitivity, and high-severity crisis detection. Human reviewers showed higher ordinal than nominal agreement (α = 0.494), indicating that disagreements often occurred between neighboring severity levels. In contrast, selected LLMs demonstrated substantially higher inter-model consistency, with ordinal agreement (α) reaching 0.8336 under the baseline prompt. Proprietary models generally showed better agreement with expert annotations and greater stability than open-source models. GPT-5.6-Terra achieved the best overall performance, while the open-source Qwen2.5-7B showed the strongest performance in identifying Acute Crisis with Suicidal Ideation cases. However, higher model consistency did not necessarily correspond to agreement with human judgments or reliable detection of rare high-severity cases. Label-flipped demonstration experiments also revealed model-specific sensitivity to misleading in-context examples, while explanatory demonstrations did not consistently improve performance. Overall, LLMs show promise as scalable complements to expert annotation, but require careful evaluation before use in high-stakes mental health applications.

0.42

Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity

cs.CL cs.AI ·2026-09-30 ·Xinkai Chen
top venuecode/data similarity 0.62
Abstract · 16 pages, 3 figures. Extended version of a paper accepted at MHSM 2026 (IEEE ICDM 2026 Workshops); includes BDI-II extraction for the 9B mod…

A large language model (LLM) can rate depression severity directly from a social media post, or mark which clinical criteria the post shows and let a simple rule turn the count into a label. The latter is easier to audit because a clinician can check each marked criterion. We compare these approaches on two Reddit corpora using three LLMs (from 9B to frontier scale) and two questionnaires (PHQ-9, BDI-II), and measure agreement with quadratic weighted kappa. For the two frontier models, criteria extraction scores above chain-of-thought on one corpus only when its decision thresholds are fitted on labeled data. Neither model's gain is significant, with or without recalibrating chain-of-thought on the same labels. With thresholds fixed a priori from PHQ-9's criteria, extraction shows no gain on either corpus, even where models mark over two criteria per post. The 9B model behaves differently on a corpus from depression communities. It labels most posts as severe, whether prompted directly or with chain-of-thought, while the a priori rule, which uses no labels, beats both. After chain-of-thought is recalibrated on the same labels, no significant gap remains, consistent with a calibration effect. Yet higher ordinal agreement does not ensure better detection of severe cases. PHQ-9 criteria extraction misses most severe posts, and moving from direct prompting to chain-of-thought and then to extraction increases misses in nearly all comparisons. On one corpus, a relabeled stress dataset, a model using that dataset's own features, some computed from the post text, is not significantly different from frontier criteria extraction under the a priori rule.

0.56

Veracity labels flip the emotional tone of language models but leave racial disparities intact

cs.CY cs.CL ·2026-10-08 ·Nouar AlDahoul, Myles Joshua Toledo Tan, Yasir Zaki
code/data similarity 0.62
Abstract · 14 pages, 7 figures; Supplementary Information (42 pages, 6 figures, 27 tables) included. Data and code: https://github.com/Nouar-NYUAD/llm-…

Large language models (LLMs) increasingly deliver information through personas, but it is unclear whether correcting the facts also makes their framing consistent across identity labels. We audited four LLMs (GPT-5.4 mini, NVIDIA Nemotron 3 Nano, Llama 4 Scout, and Claude Haiku 4.5), which rated simulated affect on six dimensions (mood, anger, fear, confidence, perceived control, and mental state) in response to fact-checked false and true headlines presented without and with a veracity label. Experiment 1 evaluated 1,000 false and 1,000 true headlines without personas; Experiment 2 evaluated 300 of each under 12 personas crossing race, gender, and age. The label strongly changed the emotional tone: ordinal logistic regressions with headline-cluster bootstrap inference showed that refuting false headlines shifted ratings from mostly negative to mostly positive for GPT, Llama, and Claude (e.g., Claude's "confident" ratings rose from 22% to 99.6%), whereas Nemotron showed smaller, mixed shifts, and confirming true headlines produced only small shifts. The label did not, however, remove persona-associated disparities. The models assigned more negative simulated affect to Black than to White personas in 82 of 84 statistically significant comparisons across models and conditions, both without and with the label, and the disparity persisted on headlines unrelated to race; Asian and female personas were also predominantly assigned more negative affect, whereas age effects varied by model and condition. Factual correction and demographic consistency are therefore distinct properties of LLM behavior, and persona-conditioned systems should be audited for both before deployment in sensitive contexts.

0.39

Mixed Feelings, Single Polarity

cs.CL ·2026-10-11 ·Emily Öhman, Saif Mohammad
similarity 0.62
Abstract

Emotion datasets are routinely mapped onto positive and negative valence, by assigning each emotion category one polarity. We test where this works and where it fails, using SemEval-2018 Affect in Tweets (English, Arabic, Spanish), Affective Text aligned with EmoBank, raw GoEmotions annotations, naturalistic hashtag corpora, and a controlled annotation study. Fixed projection works well when a text carries labels of one valence. It fails on a subset identifiable from the labels alone: tweets whose emotion labels point in opposite directions. These are 9.0% of polar SemEval test tweets but cause 54.8% of the polarity errors of a learned projection, are more than twice as likely to receive a neutral-or-mixed gold label, and are not rescued by modeling the full label set. Annotators produce such configurations routinely: 16.3% of multilabel GoEmotions judgments combine emotions from the dataset's own positive and negative parents. In a controlled study holding word and reader constant, context changes the coarse valence judgment in 56.8% of cases, against 19.9% for controls. We recommend projecting label configurations rather than labels, abstaining or keeping a mixed class where labels conflict, and reporting coverage alongside accuracy.

0.48

AI-inferred expressed well-being and collective-action discourse in climate-change campaigns on X

cs.CL cs.CY ·2026-08-11 ·Wentao Xu
similarity 0.61
Abstract

Climate campaigns are often evaluated through attention and mobilization, but less is known about the well-being language that accompanies them. Whether campaign periods alter positive affect and hope, and whether happiness aligns with action language, remains unresolved. We analysed 364,118 public Twitter/X posts from Earth Day, Earth Hour, Global Climate Action Day and World Environment Day in 19 occurrence-years, using 30-day pre-event, event and post-event windows. A versioned weighted lexical model estimated happiness, future-oriented hope, collective capability, distress and action language. Event-period happiness prevalence was 9.02 percentage points higher than the pre-event baseline , whereas paired occurrence contrasts showed a 10.75-point decline in action language, indicating a happiness--action divergence. The happiness estimate remained positive across composition and text-deduplication checks, but was less precise under a 19-cluster wild bootstrap. Happier source posts had lower odds of an observed matched retweet cascade.

0.04

DistressBench: Evaluating Multi-Dimensional Crisis Support in Large Language Models

cs.CL cs.CY ·2026-10-08 ·Drew Rein, Patrick Oathout, Vishal Kumar, Udari Madhushani Sehwag
similarity 0.61
Abstract · 39 pages, 9 figures, 13 tables

Language models have been adopted faster than evaluation practices that reliably assess their risks, and a growing share of use is affective rather than informational. Within that traffic users disclose suicide and self-harm (SSH) crises. Existing safety benchmarks score models primarily on refusal, but refusal is not the desired behavior here; the conversation requires a substantive supportive response. We introduce DistressBench, 718 English conversations with clinician-authored user turns spanning 23 SSH subcategories, each paired with a task-specific rubric of weighted atomic criteria across seven dimensions of crisis support and a clinician reference response. A model's score is the weighted average over the criteria its reply satisfies, reading directly as the fraction of required care delivered. The best of 25 frontier models satisfies 88.4% of weighted criteria and the median model 71.4%, against 98.7% for clinician references scored on the same criteria. Pooled across all 25 models, in 35.3% of conversations where a model satisfied every criterion for recognizing the crisis, it failed to route the user toward human help. The shortfall concentrates in the dimensions requiring the model to act rather than withhold. Contrasting each dimension's median model against its best separates capability limits from choices: de-escalation reaches 74.6% in the strongest model against a median of 46.0%, and safety disclaimers 84.6% against 8.3%. Multi-turn conversations score below single-turn ones (68.5% against 74.2%). We release the corpus, rubrics, judge calibration labels, and a companion annex of 20 long-horizon red-team transcripts. These results argue for evaluating crisis safety on the care a model delivers rather than on what it declines to say, and for treating de-escalation and resource routing as deployment requirements.

0.21

Secondhand Stress: How Stressful Disclosures Affect Psychosocial Wellbeing in Online Graduate Admissions Communities

cs.CL ·2026-10-15 ·Veda Duddu, Agam Goyal, Sabelle Huang, Ayush Umashankar Prasad, Mingquan Yi +1 more
similarity 0.61
Abstract · Preprint: 22 pages, 15 tables

Online communities provide spaces for individuals navigating stressful life transitions to disclose difficulties and seek peer support. Yet encountering others' distress may also have psychosocial implications for community members. Drawing on stress crossover theory, we examine how engagement with peer distress during graduate admissions is associated with subsequent psychosocial expression and participation, and whether similar associations extend through downstream interactions. We analyze 171,981 posts and 1,130,413 comments from two Reddit communities across three admissions cycles, using propensity-score stratification and baseline-adjusted comparisons. Users who engaged with distress disclosures during application preparation subsequently exhibited higher classifier-estimated anxiety, depression, and stress expression, alongside greater commenting activity and lexical redundancy. These associations recurred across admissions cycles, while no significant psychosocial associations were detected among downstream respondents interacting with previously exposed users. Our findings highlight how greater distress-related expression can coexist with continued community participation, while distinguishing direct engagement with distress disclosures from subsequent social contact. We discuss implications for designing online communities that preserve opportunities for self-disclosure and peer support while remaining attentive to the experiences of those encountering distress-related content.

0.25

Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generation

cs.CL cs.AIcs.CY ·2026-09-17 ·Mohit Chandra, Nabin Kim, Eli Min, Aamogh Sawant, Tanmay Sutar +1 more
similarity 0.61
Abstract · 25 pages, 6 figures, 17 tables

As access to professional mental healthcare remains limited, many individuals turn to online platforms such as Reddit to seek peer support situated within human lived experience. However, a significant portion of such queries go unanswered, presenting an opportunity for using Large Language Models (LLMs) to fill this gap. While LLMs have demonstrated strong performance on clinical benchmarks, their ability to generate lived-experience informed and community-aligned peer support is underexplored. Addressing this gap, we introduce the COmmunity-centered Peer Engaged Support (COPES) dataset and a three-axis evaluation framework to assess LLM alignment with community perspectives to mental health support seeking queries. Evaluating zero-shot and post-trained (SFT and DPO) models, we show that post-training on COPES significantly improves Strategy Alignment (>50% for general-purpose models) and alignment in Emotion & Tone. However, we also observe that such improvements are heterogeneous and alignment improvements vary significantly across subreddits and requested coping strategies. Furthermore, post-training induces distributional shifts, heavily favoring problem-focused recommendations while suppressing emotion-focused strategies. Together, this work shows that while curating community-driven data improves the alignment of LLM responses, model performance remains disparate across distinct sub-communities and specific mental health needs.

0.43