CS Paper Radar

Adaptive Ecological Momentary Assessment with a Hybrid Language Model: Formative Expert Review and Retrospective Evaluation

Arash Ahmadi, Dingjing Shi, Yaser M. Banad

cs.HC ·submitted 2026-09-05 21:30 UTC · v1

Abstract

Ecological momentary assessment (EMA) measures experience in daily life, but fixed questionnaires and schedules collect information of uneven value and can interrupt participants. We present and retrospectively evaluate EMA-E4B, a hybrid framework for question selection and prompt timing. Separate ridge models propose an item set and delay; a supervised Gemma 4 E4B language layer produces the final structured response and explanation. Evaluation distinguishes proxy action performance, output conformity, and formative judgments of response quality. The data contain 4,372 records from 79 participants and yield 3,516 sequential cases under a participant separated split. One involved domain expert preferred the complete hybrid response in 15 of 20 decisive comparisons, with two ties among 22 reviews. On 75 reused development cases, hybrid question utility and timing similarity were 0.832 and 0.818; the head alone reached 0.852 and 0.818. A separate 60 case comparison with untouched E4B under the same head gave action differences of -0.0031 and -0.0105. Thus, the language layer produced structured responses with action scores comparable to or slightly below the reference configurations, while the expert feedback favored the complete hybrid response. These observations establish a concrete, inspectable framework and clarify the distinct roles of action scoring and response review. Repeated adaptive administration and practical effects on measurement and participant burden remain future research.

Closest papers

Nearest neighbours by meaning across the whole index.

Localized Ecological Momentary Assessment for Mental Health Research in China: An Implementation-Oriented Framework and Preliminary Case Application

cs.HC ·2026-08-20 ·Xinying Zhao, Yue Li, Jiafeng Wang, Yunfan Fu, Ruilin Guo +3 more
similarity 0.62
Abstract

Background: Ecological momentary assessment (EMA) is increasingly used in mental health research, but research-grade deployment requires platforms supporting protocol configuration, automated delivery, participant management, and data export. In China, these requirements are not consistently supported. Objective: We aimed to identify workflow gaps affecting localized EMA deployment, develop an implementation-oriented framework for platform assessment, and assess Huixin EMAI. Methods: We reviewed EMA platforms reported in Chinese mental health studies in CNKI and Wanfang. A multidisciplinary panel of 6 experts developed the Multi-dimensional EMA Platform Evaluation Framework (MEPEF) and benchmarked 7 platforms across 43 indicators in 6 domains. MEPEF was then applied to Huixin EMAI using deployment logs from 48 participants, questionnaires from 44 participants, and semistructured interviews with 6 researchers. Results: We identified 66 empirical studies. Most relied on instant-messaging-based workflows (36/66, 54.5%), whereas specialized EMA platforms were less common (14/66, 21.2%). MEPEF provided a 6-domain framework for cross-platform benchmarking and highlighted a trade-off between localized deployability and advanced research functions. In a Huixin EMAI deployment, 1893 of 2472 expected prompts were completed (76.6%), with a median response latency of 4.0 minutes (Q1-Q3 0.0-13.0). Participant feedback indicated favorable acceptability; researchers reported support for core workflows but gaps in control, delivery monitoring, and data readiness. Conclusions: The main challenge for EMA in Chinese mental health research appears to lie less in feasibility than in recurring workflow gaps affecting localized deployment. This study translates these gaps into structured evaluation and design targets, providing an implementation-oriented pathway for advancing localized EMA platforms.

0.25

PsycheMME: A Psychology-Informed Benchmark for Social Understanding and Interaction in Videos

cs.CV ·2026-10-20 ·Heran Hu, Yuhang Dong, Zhucun Xue, Kaiwei Chen, Yong Liu +1 more
benchmark similarity 0.60
Abstract

Understanding human needs and selecting responses require distinct judgments that aggregate accuracy can obscure. We introduce PsycheMME, a video benchmark informed by psychology, with eight capabilities spanning grounding, states, needs, interaction decisions, memory, and risk. Drawing on Social Information Processing and the Perception-Action Cycle, this organization separates human understanding from response selection. We construct 1,000 questions from 630 episodes through independent answering with proposed references withheld and expert review. Our Reasoning-Verified protocol weights answer credit by the support and logical consistency of explicit rationales against annotated cues. Across 17 models, Timing has the lowest Verified scores and credit retention, with weaker support judgments for correct Timing answers. Models with similar Need scores still differ in answer completeness on open questions. Input controls and SFT/GRPO examine context dependence and adaptation. Evaluating answers alongside rationales locates where models fail across understanding, response selection, and justification.

0.26

Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark

cs.CY stat.AP ·2026-09-10 ·Eunjeong Song, Sehee Hong
benchmark similarity 0.60
Abstract · 49 pages, 3 figures, 12 tables. Data, materials, and code: https://doi.org/10.7910/DVN/S7QUCA

Large language models (LLMs) are proposed as synthetic respondents for pilot testing, but their usefulness depends on whether they supply the evidence assessment development requires. We calibrated rating scale models on 14 digital-use skill items from 6,245 adults and used the human item parameters to evaluate responses generated for 1,300 demographically matched personas. LLM responses had high internal consistency ($α\approx .94$) but used the lowest category in 0.2-0.3% of responses versus 13.7-21.5% for humans, and no persona chose it on every item. These gaps changed the response-category judgment on both subscales and the PC targeting judgment; on the human-calibrated scales, 12 of 14 items had infit below 0.70, indicating responses more predictable than the Rasch model expects. Preregistered changes to the prompt, category order, and persona information did not restore the human lower range. High internal consistency is insufficient evidence that LLM responses can replace human pilot data.

0.31

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness

cs.CL cs.AIcs.LG ·2026-10-22 ·Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro
benchmark similarity 0.59
Abstract · 23 pages,

LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.

0.25

LLMs as Judges and Examinees for Medical Item Difficulty Prediction

cs.CL cs.AI ·2026-10-28 ·Xiaomeng Xiong, Marcus Walker, Yanlin Jiang
similarity 0.59
Abstract

Predicting medical item difficulty from item text alone remains challenging. Prior work has largely relied on linguistic features or text representations derived from general-purpose and domain-specific language models, but predictive performance has remained modest. This study takes a different approach by using difficulty-related signals that emerge when open-weight large language models (LLMs) interact with test items. Using USMLE and PANCE medical licensure examination items, we examined LLMs in two roles. As judges, the models provided absolute difficulty ratings and pairwise difficulty comparisons. As simulated examinees, they generated response-based signals including correctness, keyed-option probability, response entropy, and the probability margin between the top two options. We then combined the strongest judgment and response signals with baseline item features in downstream prediction models. Pairwise difficulty judgment was the most consistent individual LLM signal across both datasets and outperformed absolute ratings, and adding simulated-examinee signals yielded only small further gains. The best configurations reached a Spearman correlation of $ρ$ = .447 on USMLE, exceeding the best system in the 2024 BEA shared task ($ρ$ = .25), and $ρ$ = .494 on PANCE. These results were comparable to direct absolute judgments from the proprietary frontier model Claude Opus 5 ($ρ$ = .443 and .488, respectively). Overall, judgment and response signals from open-weight LLMs captured difficulty-related information beyond text representations. Because these models can be deployed locally, the approach offers a practical option for predicting the difficulty of unreleased or high-security test items.

0.28

Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis

cs.HC cs.AIcs.CL ·2026-08-16 ·Alona Strugatski, Licol Zeinfeld, Giora Alexandron
similarity 0.59
Abstract

The rapid development and growing deployment of large language models (LLMs) have made it increasingly important to understand their capabilities. A common approach is to evaluate LLMs using assessment instruments originally designed to measure skills and competencies in humans, such as standardized exams, and to use performance on these instruments as evidence for generalizable claims about LLMs' underlying abilities on the same skills the assessments are intended to measure in humans. However, from a validity perspective, such inferences require that the relationship between observed performance and underlying constructs established for humans also holds for LLMs. In particular, a necessary condition for transferring score interpretations is similarity in the latent structure of responses to the assessment. In this study, we examine whether this condition holds in two educational contexts: high-school chemistry and a quantitative reasoning section of a university entrance exam. Using a case study design, we compare human response data with responses generated by six multimodal LLMs. Our analytical approach combines exploratory factor analysis, factor congruence, and resampling to assess latent structure similarity across human learners and LLMs. Across both instruments, we find systematic differences between human and LLM factor structures, showing evidence that the analyzed assessments may not measure the same constructs for humans and LLMs. These findings call into question the validity of evaluation practices that use educational assessments to make claims about AI capabilities.

0.38

Identifying Effective Strategies in Prompt Evolution for Stress and Emotion Prediction

cs.CL ·2026-10-15 ·SeungHyun Yi, Seungjun Yi, Sriram Sattiraju, Haeun Park, Joakim Nguyen +10 more
similarity 0.57
Abstract · 4 figures, 2 tables. Submitted to the 2027 IEEE International Conference on Healthcare Informatics (ICHI 2027)

Early stress detection and risk prediction are crucial for timely mental health assessment and support. While large language models have recently shown promise for social media stress prediction, relying on accuracy can obscure false positives. Also, prompt revisions yield inconsistent gains across models and tasks, with unclear links to strategy combinations and instruction changes. By comparing six methods on social media stress and emotion data across two model sizes, we found that GEPA and its extension combining instructions from two prompts (GEPA+merge) improved prediction over initial prompts in several settings. We also identified five prompting strategies associated with development gains. These strategies included directing attention to emotional evidence, distinguishing current from past distress, illustrating indirect cues, requiring emotional context, and guiding label selection. Moreover, we documented four observations across tasks and evaluation settings. In a separate set of searches, binary stress prediction improved early, whereas multilabel emotion prediction initially declined and partly recovered. The other three concerned incorrect or conflicting instructions, gains on small comparison samples that did not hold on full development data, and changes in selected prompts' development and reported test rankings. These findings identify effective prompt evolution methods, strategy combinations associated with gains, and task and evaluation limits, providing guidance for applying prompt evolution methods to social media stress detection.

0.25

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

cs.CL cs.AI ·2026-10-13 ·Kawai Chung, Chunkit Chan, Yuxuan Liu, Yauwai Yim, Xi Yang +7 more
benchmarkcode/datahot directionhighly cited similarity 0.57
Abstract · Code and evaluation materials: https://github.com/HKUST-KnowComp/Jev-ToM

Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative to generative large language models for bounded Theory of Mind (ToM) selection. We present Jev-ToM, comparing five typed decision models and seven generative baselines on 11,117 English text questions from nine ToM benchmarks across six capability groups. TypeSafe Jev reaches 73.2% source-balanced accuracy including native controls and ranks first or second across all six capabilities, leading beliefs, goals/desires/intentions, and emotions. Other typed models show distinct strengths across tasks. Separate resource comparisons suggest that Jev could reduce the cost and waiting time of repeated ToM selection. Confidence-based selection remains benchmark-dependent, and supplied reasons can repair errors or overturn correct answers depending on their attachment. Candidate expansion reveals distinct selection and output-validity problems. Related-question cues account for most of Jev's downstream gains, with task-dependent effects from predicted states. The findings motivate testing Jev as a ToM selection component in assistants, with task-specific confidence calibration, candidate design, and output validation as integration priorities.

0.67