CS Paper Radar

From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models

Hezhao Zhang, Thomas Hain

cs.SD cs.AIcs.CLeess.AS ·submitted 2026-09-30 10:26 UTC · v1

Abstract

Speech emotion recognition (SER) is the task of assigning emotion labels to utterances. Early systems relied on acoustic features, whereas recent approaches combine multiple modalities, most commonly speech and text. Still, performance remains poor on many datasets. Large language models (LLMs) have therefore attracted interest for SER, as they can process diverse inputs jointly with instructions. However, direct audio input raises questions of explainability. To address similar questions in image classification, concept bottleneck models were introduced. This work adapts concept bottlenecks to SER to examine how individual predictions depend on transcripts, acoustic descriptions and speaker attributes. Experiments test three LLMs on CREMA-D, IEMOCAP and MELD, with concepts extracted by separate tools. On scripted corpora, LLMs are strongly biased towards the transcript in the zero-shot setting, which lowers Macro-F1 from 27.8 to 5.8 on CREMA-D. Fine-tuning removes this bias, and the transcript raises Macro-F1 from 41.8 to 45.1. Removing speech rate changes 48% of Neutral predictions to Disgust on CREMA-D; removing intensity level on MELD changes predictions despite little change in Macro-F1. These findings show that aggregate performance changes alone do not capture the effects of concept removal on individual predictions.

Comment
5 pages, 2 figures. Submitted to ICASSP 2027

Closest papers

Nearest neighbours by meaning across the whole index.

Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models

cs.AI cs.CLcs.MMcs.SD+1 ·2026-10-29 ·Fengji Jing, Guy Laban
code/data similarity 0.76
Abstract

Voice-interactive large language models process both what a speaker says and how they say it, yet the two channels do not always agree. We ask whether that disagreement degrades speech emotion recognition. Mapping text and audio into a shared valence-arousal-dominance space, we convert the distance between them into a continuous, threshold-free incongruence score with a two-component Gaussian mixture, and validate the score against human judgement. We evaluate three audiocapable models on acted, conversational and spontaneous speech (eNTERFACE'05, MELD and CMU-MOSEI). With the model held constant across modalities, incongruence raises the odds of error whenever audio is present, in every corpus, after controlling for the ground-truth emotion and for distance in the feature space. Supervised classifiers trained on the same features behave in the opposite way, which suggests that the difficulty lies in how these models combine raw evidence rather than in the signal itself. Text-only effects are unreliable: swapping only the model that reads the transcript reverses their sign on one corpus. Rebuilding the score without the emotion lexicon reproduces the central result, and among six simpler alternatives it is the only label-free score that predicts error in all three corpora. Code, data tables and prompts are released.

0.13

From Generative to Discriminative: Lightweight Adaptation of SpeechLLMs for Multimodal Emotion Recognition

cs.CL cs.AIcs.SDeess.AS ·2026-09-17 ·Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello, Manjunath K E, Kadri Hacioglu +2 more
similarity 0.75
Abstract · Under review at ACL Rolling Review (ARR)

SpeechLLMs can recognise emotion by generating a label token, but this approach relies on the LLM's existing output representations. We compare this generative approach with a discriminative alternative that uses a linear classification head on top of the LLM's hidden states. Because most multimodal emotion recognition studies evaluate with ground-truth transcripts, whereas deployed systems receive ASR output, we run every comparison with both, using a compact speechLLM (frozen Whisper-medium encoder, 3B LLM) on IEMOCAP and MELD. The classification head improves Macro F1 in 10 of 14 configurations. On IEMOCAP, it improves every configuration using ASR transcripts, reaching 76.97\% Macro F1, with a significant gain when using speech alone. On MELD, where ASR transcripts have a 32.38\% word error rate, it helps with speech alone but not with ASR transcripts. The benefit thus depends on both the dataset and the transcript source: in 4 of the 6 configurations that use text, the head's effect changes sign between ground-truth and ASR transcripts. Conclusions drawn with ground-truth transcripts therefore need not hold with ASR transcripts. Finally, comparing the learned class vectors with the LLM's vocabulary shows that, on IEMOCAP and MELD, their nearest tokens are mostly emotion-related words rather than explicit emotion labels.

0.20

Beyond Accuracy: What Are Speech Emotion Recognition Models Learning?

eess.AS ·2026-10-24 ·Prince Baah-Mensah, Dennis Owusu Asamoah
similarity 0.74
Abstract · 6 pages, 4 tables

Speech Emotion Recognition (SER) is important in human-machine interactions, enabling machines to detect and classify human emotions from solely human speech. Speech emotion recognition (SER) benchmarks routinely report accuracies above 90 percent, yet these models fail when deployed or tested on diverse speakers. The papers overlook a fundamental issue: models may be learning speaker identity and tone rather than generalized emotional cues. We introduce a two-condition audit protocol that measures, for any published SER model, how much of its reported performance survives when speakers are held out. Each model is evaluated twice under identical architecture, features, and training recipe: a control condition that reproduces the common random-split practice, and a strict speaker-independent condition with disjoint speakers. We apply the protocol to seven model configurations from five published papers, spanning convolutional, recurrent, transformer, vision-transformer, and self-supervised wav2vec2 architectures, on RAVDESS, EMO-DB, and SAVEE, with a fourth corpus (EARS) as an external check. Across 21 model-dataset pairs, published headline numbers reproduce close to the control but drop by 6.9 to 63.1 percentage points under strict speaker independence. Our results quantify speaker memorization as a dominant, previously unmeasured component of published SER performance, and we recommend two-condition reporting as a community standard.

0.09

VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale

cs.SD cs.AIeess.AS ·2026-09-25 ·Christoph Schuhmann, Robert Kaczmarczyk, Gollam Rabby, Felix Friedrich, Maurice Kraus +5 more
benchmarkcode/data similarity 0.74
Abstract · 33 pages, 6 figures, 8 tables. Christoph Schuhmann and Robert Kaczmarczyk contributed equally. Code and benchmark: https://github.com/LAION-…

Expressive speech synthesis has outpaced expressive speech perception: systems now render fine-grained vocal performances that no public benchmark can score. Most benchmarks for this inverse problem stop at six to nine basic emotion categories, largely on acted speech. This paper introduces VoiceNet, a human-annotated representation-level benchmark for voice performance understanding on permissively-licensed in-the-wild speech. VoiceNet has two subsets: VoiceNet-Emo applies a 40-emotion taxonomy with three expert ratings per item, and VoiceNet-Ext, a preliminary subset, scores 57 talking-style attributes including speaking rate, vocal tension, breathiness, and register. The paper also releases Emolia, an emotion-annotated version of the Emilia corpus, with a curated rebalanced subset enriched by dense MOSS-Audio Thinking annotations. Two voice-text contrastive baselines train on this data: a 110M-parameter VoiceCLAP-Small for fast large-scale data filtering and a 7B VoiceCLAP-Large for state-of-the-art performance. Both outperform existing CLAP baselines, which sit near chance on VoiceNet-Emo. On VoiceNet-Emo, VoiceCLAP-Large aligns more closely with the aggregate expert consensus than individual experts agree with one another: a comparison against the majority label rather than evidence of surpassing human emotion perception. All systems evaluated here are voice-text embedding models: VoiceNet scores representation-level attribute recognition and retrieval, not end-to-end spoken-dialogue behaviour. Clustering and filtering uncurated speech corpora into subsets that span diverse talking styles and emotions remains an open challenge; VoiceCLAP embeddings offer a promising tool for this task. VoiceNet, Emolia, and VoiceCLAP are publicly available for research use.

0.31

Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue

cs.CL cs.AIcs.LGcs.MM ·2026-09-30 ·Yutong Hu, Jinho Choi
similarity 0.71
Abstract · 15 pages, 6 figures, 11 tables

Emotion recognition in conversation has been widely studied, but applying Large Language Models (LLMs) to continuous dimensional emotion evaluation in multimodal dialogue remains largely unexplored. We propose an LLM-based framework that performs discrete emotion recognition and Valence-Arousal-Dominance (VAD) dimensional evaluation on IEMOCAP, incorporating acoustic cues as natural language descriptions following the SpeechCueLLM approach. We evaluate six models spanning the LLaMA, GPT, and Qwen families under zero-shot prompting, few-shot prompting, and LoRA fine-tuning. LoRA fine-tuned LLaMA models substantially outperform prompt-engineered GPT models on both tasks despite GPT's larger scale, a gap we attribute to domain adaptation rather than model capacity. Our best model achieves a Valence CCC of 0.7822, a new state-of-the-art on IEMOCAP. Ablation studies confirm that textual audio descriptions meaningfully improve smaller models (+3.5 to 3.6 weighted F1) while contributing little for the largest model, suggesting audio cues are most valuable when linguistic capacity is limited. The performance asymmetry across VAD dimensions closely mirrors the annotator agreement hierarchy in IEMOCAP's own annotations.

0.16

VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models

cs.CL cs.SD ·2026-08-28 ·Luc Debaupte, Tyler Baumgartner, Brandon Tai, Candice Fan, Bill Wang +1 more
benchmarkcode/data similarity 0.71
Abstract · 6 pages, 6 tables. Benchmark, baseline predictions, and aggregate results are publicly available

Voice products increasingly need affective cues that are present in speech but absent from transcripts. We introduce VocalAffectBench, a public, test-only benchmark for evaluating whether AI audio models can identify expressed vocal emotion from raw audio. The benchmark contains 273 human-recorded English WAV clips from 51 speaker accounts totaling 1.95 hours across seven labels: angry, disgusted, fearful, happy, neutral, sad, and surprised, with 39 clips per class. All baselines are evaluated from audio alone, without transcripts or contextual metadata. Across six released baselines, average accuracy is 35.5%. The strongest baseline, gemini_3_5_flash, reaches 46.5% on the seven-way task, above the 14.3% random baseline but far from robust emotion recognition. A secondary valence-bucket analysis maps labels into positive, neutral, and negative classes, excluding surprised because its valence is ambiguous. Aggregate accuracy under this coarser view is 50.9%. Performance is highly uneven across classes. By recall, neutral is identified most reliably at 75.6% averaged across baselines, while surprised and fearful reach only 10.7% and 15.4%, respectively. These results show that the evaluated baselines can extract some affective signal from speech, but discrete expressed-emotion recognition remains fragile, especially for non-neutral emotions that are often most important in voice agent workflows.

0.59

EmoHarness: Boosting Multimodal Emotion Understanding with Meta-Harness Evolving

cs.CL cs.AIcs.CV ·2026-11-09 ·Fenghai Li, Bingxuan Li, Jiawen Zhang, Yifan Shen, Jiateng Liu +2 more
code/data similarity 0.69
Abstract · 18 pages, 11 figures, 6 tables. Project page: https://emo-harness.github.io. Code: https://github.com/Emo-Harness/EmoHarness

Small multimodal models can misjudge emotion when words, voice, and visual cues conflict. We present EmoHarness, which runs such a model, the frozen student, inside a harness: the prompts, settings, and verification code around its calls. A human-written starting harness, the scaffold, numbers the student's visual observations, transcript, and prosody descriptions as citable evidence and has the student weigh candidate emotions against it. In Meta-Harness Evolving (MHE), a stronger LLM, the Builder, reads the student's errors and execution traces offline and edits the harness; static checks and validation gates decide which edits are kept, and the Builder never sees evaluation items. On 5,507 MME-Emotion items, EmoHarness raises task-macro Recognition by 7.6-13.4 points over a single video-only call for all four students and both Builders. Transcripts and prosody descriptions supply most of this boost (+5.38 to +10.71); the scaffold changes the audio-augmented call by -0.79 to +2.54, partly through extra multi-label predictions, and evolved edits add at most +0.95, although no evolved increment survives Holm correction. The harness's emotion context also transfers to IntentQA, with small gains for three of four students, none of which survives correction. EmoHarness thus boosts small multimodal students chiefly by supplying and organizing audio-derived evidence.

0.51

Exposing Weaknesses in Emotion Recognition in Conversations

cs.AI ·2026-09-05 ·Amir Ben Khalifa, Fanny Bezancon, Bessam Abdulrazak, Amine Trabelsi
similarity 0.69
Abstract

Emotion Recognition in Conversations (ERC) aims to identify speakers' emotions in multi-turn dialogue. Accurate emotion recognition can support a wide range of applications, including empathetic conversational agents, mental health support, and educational technologies. While many recent approaches rely on task-specific fine-tuning, such models may exploit dataset-specific cues. A central yet rarely questioned assumption in ERC is that each utterance can be assigned a single unambiguous emotion label. To investigate this assumption, we study ERC using Large Language Models (LLMs) in a zero-shot setting while incorporating preceding conversational turns as context. We show that aggregate metrics mask systematic failures. Errors concentrate around utterances containing negations, exclamations, and interjections. This pattern is consistent across all evaluated models, suggesting limitations in the benchmarks rather than model-specific weaknesses. A controlled re-annotation study involving four human annotators supports this finding: strong agreement is observed in only 35 percent of cases, with neutral utterances dominating high-agreement instances, while many emotional categories fall into low-agreement regimes. These findings suggest that many apparent model errors reflect genuine annotation ambiguity rather than poor emotion understanding. Standard single-label evaluation is therefore insufficient. To address this limitation, we introduce an LLM-as-Judge framework that evaluates each emotion independently according to its plausibility in the conversational context rather than enforcing a single-label decision.

0.23