CS Paper Radar

Scales, Reflections, and Conversations: A Multi-Modal Approach to Emotion Annotation

Pragya Singh, Prashasti Gupta, Hitesh Bhandari, Kanishk Goel, Mohan Kumar, Pushpendra Singh

cs.HC ·submitted 2026-09-04 12:10 UTC · v1
published

Abstract

Mental health concerns are increasing worldwide, highlighting the need for interventions that support everyday emotional well being. Prior work has demonstrated the potential of wearable and mobile technologies to deliver data driven interventions. However, developing effective data-driven systems requires access to emotion data that captures individuals' emotional variability and change in everyday contexts. Existing approaches to data collection largely rely on frequent, prescheduled prompts and predefined scales or questionnaires. These methods often fail to account for participants' availability, agency, or the complexity of their emotional experiences, resulting in shallow, context poor data. In this paper, we present a feasibility study of a participant centric, multimodal emotion-annotation application designed around users' emotional intensity and availability. Our findings show how multimodal emotion logging can shape participants' experiences and data logging behaviors, and demonstrate its potential to support the collection of richer, more nuanced emotion data.

Comment
Accepted at MobileHCI 2026
DOI
10.1145/3821661

Closest papers

Nearest neighbours by meaning across the whole index.

Emotion Experience, Expression, and Perception: Emotion Analysis on Multimodal Social Media Posts

cs.CL ·2026-09-16 ·Christopher Bagdon, Carina Silberer, Roman Klinger
top venuedataset similarity 0.64
Abstract · Accepted for publication at EMNLP 2026 main conference

Emotions are an essential aspect of human communication, particularly on social media, where authors frequently combine text and images to convey their emotions. Yet prior work on emotion analysis of social media posts has overlooked two important aspects in regard to measuring how well readers can reconstruct the authors' intent: (1)~the image modality, with most work focusing solely on text, and (2)~the real-world events that trigger the expressed emotions, and their relationship to the post content. We therefore study the relation between (a) the author's experience of the event that caused them to write a social media post and (b) the content of the post, with a focus on readers' capability to reconstruct that emotion expression. To do that, we introduce the Multimodal Multi-Emotion-Model dataset Mult2EMo, created by collecting annotations from both authors and readers on the posts and their triggering events. We find that reconstruction is possible but challenging for both human readers and computational models. We show that understanding the triggering event is crucial for accurate reconstruction, and that reconstruction is particularly challenging when posts rely heavily on the image to express emotion.

0.34

Scoping the Landscape of Large Language Model-based Affective Personal Informatics: Review, Reflection, Future Direction

cs.HC ·2026-10-14 ·Minjie Liu, Lihuan Li, Xuran Li, Flora Salim, Benjamin Tag
survey similarity 0.63
Abstract

Affective personal informatics (PI) involves capturing, tracking, analyzing, and reflecting on personal data to support emotional self-awareness and mental well-being. Recent advances in large language models (LLMs) are reshaping affective PI, yet a comprehensive survey of LLM-based affective PI systems is lacking. To bridge this gap, we systematically analyzed 45 papers published between January 2020 and June 2026 across four major computing and health databases, examining their macro trends, system taxonomies, and research methodologies. Our synthesis reveals an overreliance on multi-stage, text-centric pipelines, a predominance of active, screen-bound interactions, and a severe demographic skew toward young, university-campus-based cohorts in empirical evaluations. Based on these findings, we critically reflect on prevailing architectural and methodological shortcomings and outline actionable future directions for building end-to-end multimodal, low-burden, and inclusive affective PI systems.

0.26

EmotionDialogCN: A Spontaneous Multimodal Dataset for Mandarin Emotional Dialogue

cs.CV ·2026-08-21 ·Yi Zheng, Yifan Xu, Yan Zhou, Hejia Chen, Chunyu Qiang +6 more
similarity 0.63
Abstract

Face-to-face audiovisual interaction is central to human communication, conveying rich emotional and social cues. However, existing multimodal dialogue datasets remain limited by inadequate emotion annotations, poor emotional diversity, and small scale. We introduce EmotionDialogCN, a large-scale audiovisual-emotional dataset designed to capture authentic face-to-face communication. It contains 21,880 dialogue sessions performed by 119 professional actors across 20 everyday scenarios, covering 18 emotion categories with over 400 hours of recordings, the largest and most comprehensive dataset of its kind. A novel data collection framework minimizes equipment interference, enabling natural and nuanced emotional expressions. EmotionDialogCN achieves an emotion distribution deviation of 0.64 from real human emotion statistics (versus 5.65 for prior datasets) and consistent subject framing (52-59% frame occupancy). Together, these properties translate into stable unimodal and multimodal performance across acoustic, lexical, and visual modalities, with fusion results further underscoring strong multimodal alignment and cross-modal complementarity.

0.12

Layer Roles Matter: Mitigating Modality Imbalance in Multimodal Emotion Reasoning via Adaptive Layer Profiling

cs.AI ·2026-11-09 ·Pengyu Huang, Peipei Song, Beier Zhu, Zhiyuan Han, Jinpeng Hu +1 more
top venue similarity 0.62
Abstract

Multimodal Emotion Reasoning (MER) requires perceiving and integrating emotion cues from multiple modalities. Recent advances increasingly leverage Multimodal Large Language Models (MLLMs) to enhance reasoning capacity. However, due to the inherent heterogeneity in conveying emotions across modalities, these models often suffer from modality imbalance and confront flawed emotion understanding. In this paper, we conduct an in-depth analysis on MER models and uncover two key empirical insights. (i) In MER, modality imbalance manifests two distinct failure modes: unimodal over-reliance and insufficient cross-modal utilization, which call for different intervention strategies. (ii) MLLM layers exhibit implicit functional hierarchies in multimodal emotion processing, playing separable roles in stabilizing unimodal representations and integrating cross-modal information. Motivated by these observations, we propose a Layer-Aware Modality Rebalancing (LAMR) framework by explicitly aligning intervention strategies with layer roles. Specifically, we first introduce two quantized metrics for adaptive layer role profiling. In each layer, Modality Entropy (ME) measures the dispersion of modality contributions, and Cross-modal Intensity (CMI) captures the strength of cross-modal interactions. Then, we respectively enhance modality equilibrium in stabilization layers and cross-modal propagation in integration layers. Extensive experiments on two comprehensive MER benchmarks demonstrate that LAMR effectively mitigates modality imbalance and improves emotion reasoning across diverse MLLM backbones, without additional training.

0.30

Reconstruction and Reflection of Positive Experiences through Resurfacing Laughter-indexed Everyday Moments

cs.HC ·2026-09-11 ·Jun Fang, Jiajin Li, Yuntao Wang, Kexin Miao, Susu Wang +3 more
similarity 0.62
Abstract

Positive everyday moments often escape deliberate recording, while continuous self-tracking can generate extensive records that are difficult to revisit. We explore laughter as a naturally occurring, sparse index for constructing contextualized personal records to support later reconstruction and reflection. A formative study with 12 participants characterized laughter as an affective but semantically incomplete index and informed \textit{LaughAnchor}, a mobile and wearable self-tracking system. During participant-initiated recording, the system assembles detected laughter and aligned context into candidate moments for later reconstruction and reflection, with layered context disclosure, user-controlled curation, and near-term and long-term resurfacing. In a three-week field deployment with 12 participants, passive indexing preserved moments they considered unlikely to record deliberately but valued retrospectively. During resurfacing, participants attributed affective re-experiencing to laughter and used additional context both to reconstruct episodes and to explore already-recalled experiences. Across moments and reviews, resurfacing supported rediscovery and broader awareness of relationships, routines, and emotional states. These findings inform self-tracking designs that use sparse affective indices to organize contextual records for reconstruction and reflection, while keeping interpretation and retention under user control.

0.25

Cross-Modal Emotion Understanding: A Transformer-GAT Approach for Dialogue Emotion Recognition

cs.AI ·2026-09-02 ·Jiaqi Qiao, Yifan Lyu, Xiujuan Xu
similarity 0.61
Abstract · 12 pages, 9 figures

Multimodal emotion recognition is a key research area in affective computing, with applications in sentiment analysis, intelligent customer service, and human-computer interaction. However, existing methods often rely on single-modal features or simple multimodal fusion, failing to capture the synergy between global and local contexts, which limits model performance and emotion understanding. To address this challenge, we propose Transformer-GAT, a hybrid framework that combines Transformer and the Graph Attention Network to enable cross-modal emotion understanding. The Transformer is used to capture global semantic information, while the Graph Attention Network is employed to model fine-grained relationships between modalities, thereby enhancing the representation of emotional features. Experiments on the IEMOCAP and MELD datasets show that our model achieves weighted F1 scores of 72.45% and 77.37%, outperforming state-of-the-art methods. These results demonstrate that Transformer-GAT effectively integrates multimodal features, balances global and local contexts, and provides deeper emotional insights, offering new directions for multimodal emotion computing.

0.27

User Experience in Human-Machine Interaction: Insights from Field Studies in Autonomous Mobility

cs.HC ·2026-08-31 ·Helen Schneider, Svetlana Pavlitska, J. Marius Zöllner
published similarity 0.61
Abstract · Accepted for publication at ABAW Workshop at ECCV2026

Autonomous vehicles (AVs) promise safer, cleaner, and more inclusive mobility, yet large-scale adoption is hindered by user acceptance rather than by technical challenges. Prior studies on acceptance and user experience largely rely on surveys, simulators or Wizard-of-Oz setups, often over-representing technologically enthusiastic participants and focusing on drivers instead of passengers. We address this gap with real-world field studies with AVs in real traffic, totaling 144 participants. Using multi-modal sensing, we evaluated EGG, heartbeat, breathing, camera and voice signals for affect inference in combination with vehicle data. Our results show that breathing, camera and voice measurements are reliable and pratical in naturalistic passenger contexts. We further contribute a validated study protocol, a self-assessment app for real-time assessment during human-machine interaction, and a tailored questionnaire to capture participant attitudes towards AVs. By grounding UX evaluation in real-world contexts, this work lays a foundation for user-centered design of autonomous mobility systems and robotics in general. Our work bridges the gap between affective computing and technical implementation of autonomous vehicles.

0.14

Beyond Multimodal Emotion Conflict: Temporal Context-Aware Hidden Emotion Detection with Multimodal Large Language Models

cs.AI ·2026-11-01 ·Yonghao Wang, Jiaxin Zhou
benchmark similarity 0.61
Abstract · 6 pages, 4 figures

Multimodal emotion recognition commonly combines textual, acoustic, and visual signals to infer affective states. Recent work has begun to study conflicts among modalities, but a conflict does not necessarily indicate that a person is concealing an emotion. It may also result from mixed emotions, individual expressive styles, or contextual factors. This proposal aims to distinguish ordinary emotional inconsistency from emotion concealment and to infer the underlying emotion when supported by evidence. We plan to construct a multimodal benchmark with explicit annotations of expressed emotion, internal emotion, and concealment conditions, and to develop an MLLM-based framework that integrates fine-grained facial Action Units (AUs), cross-modal features, and temporal context.

0.07