CS Paper Radar

ACR-Net: Mitigating Semantic Dominance via Contrastive Acoustic-Semantic Decoupling

Mengke Zhang, Yanda Shao, Tianhe Wu, Kai Feng

cs.SD cs.AI ·submitted 2026-10-14 12:42 UTC · v1
top venuebenchmark

Abstract

Speech Emotion Recognition (SER) models and Audio LLMs fail when vocal tone contradicts textual semantics. Extensive training on congruent data engenders a rigid semantic prior, causing "Semantic Dominance", a severe text bias during cross-modal conflicts. We introduce the ASPIRE benchmark of adversarial audio-text pairs across four conflict types, alongside two metrics: Semantic Overconfidence Penalty (SOP) and Latent Decoupling Degree (LDD). Unlike fusion methods that merely re-weight collapsed features, our Acoustic Conflict Resolution Network (ACR-Net) uses Cross-Modal Attention and Contrastive Decoupling Loss to disentangle contradictory representations into orthogonal latent spaces. Experiments show existing models fail under polarity conflicts due to extreme semantic overconfidence. Conversely, ACR-Net preserves acoustic fidelity, maintaining high LDD to drive SOP near zero, achieving superior Acoustic Accuracy (ACC) in both adversarial and standard scenarios.

Comment
5 pages, 1 figure, 4 tables. Published in Interspeech 2026
Journal ref
Proc. Interspeech 2026, pp. 1207-1211 (2026)

Closest papers

Nearest neighbours by meaning across the whole index.

When Vocal Tone and Literal Meaning Diverge: An Acoustic-Semantic Incongruity Study for Large Audio-Language Models

eess.AS ·2026-08-29 ·Yu-Wen Chen, William Ho, Maxim Topaz, Zoran Kostic, Julia Hirschberg
top venuedataset similarity 0.69
Abstract · EMNLP 2026 Findings

Affective cues across modalities may be incongruous (e.g., sarcasm or mocking praise), potentially leading to misinterpretation when relying on a single modality. Large Audio-Language Models (LALMs) have recently gained popularity and been applied to multimodal emotion recognition, but their ability to disentangle acoustic and semantic cues, especially in incongruent cases, remains underexplored. To address this gap, we introduce CREMA-ASIS, a dataset specifically created to investigate incongruence between acoustic emotion and semantic sentiment cues. It pairs acoustic emotion labels with semantic sentiment polarities. Using this dataset, we evaluate LALM biases within a multitask framework and conduct a layer-wise analysis to identify modality dominance across layers. Our findings reveal that LALMs struggle with semantic-acoustic incongruent cases, rarely predicting incongruity, and that LALMs are predominantly influenced by semantic information. However, supervised fine-tuning significantly improves LALM performance on our CREMA-ASIS test set while preserving transcription accuracy and joint emotion recognition. Results demonstrate potential for enhancing both acoustic and semantic understanding on out-of-domain data.

0.41

When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue

cs.CL cs.AIcs.LGeess.AS ·2026-08-27 ·Yen-Ju Lu, Yuzhe Wang, Yaohan Guan, Xiluo He, Jiarui Hai +6 more
benchmark similarity 0.67
Abstract · 24 pages, 4 figures

Understanding spoken dialogue requires joint reasoning over lexical content and paralinguistic acoustic signals such as emotion and conversational intent. However, existing evaluations often allow shortcuts based on transcripts or single-modality solutions, obscuring whether models genuinely ground predictions in speech. We formalize this failure mode as cross-modal disagreement, where transcripts suggest plausible but incorrect surface interpretations while acoustic cues such as prosody or speaking style support different answers. We develop a scalable framework that identifies text-biased surface interpretations and converts disagreement regions into conflict QA examples. We also include consistent cases where transcript-based and speech-grounded interpretations agree, enabling evaluation beyond adversarial audio dependence. This results in ContraTalk, a controlled benchmark containing 501 questions across five discourse dimensions: interaction behavior, emotion state, dialogue act, social stance, and conversational intent. We further develop an agentic-style reasoning framework that converts speech into an Audio Twin, a text-readable representation of localized acoustic cues that exposes acoustic evidence to the reasoning model. Experiments show that strong text-only LLMs exceed 90% accuracy in consistent cases but drop to 33-48% in conflict cases. Direct AudioLLMs provide only partial grounding, still selecting the transcript-biased trap in roughly 30-40% of conflict cases. Our Audio Twin framework improves conflict-case accuracy while reducing trap selection, but its consistent-case behavior remains backbone-dependent. These results identify transcript-based shortcuts as an important failure mode in spoken dialogue understanding and show that explicit acoustic evidence aggregation provides a more controllable interface for diagnosing and improving speech-grounded reasoning.

0.49

Discriminator-free adversarial domain adaptation for robust speech classification

eess.AS cs.SD ·2026-11-09 ·Yacouba Kaloga, Ina Kodrasi
similarity 0.66
Abstract · Submitted to ICASSP 2027

The performance of speech classification systems such as speech emotion recognition or pathological speech detection can degrade substantially when there is a domain shift between training and test data. This shift can arise due to e.g., differences in recording setup, acoustic environment, speaker, language, or speaking style. Adversarial domain adaptation addresses this problem by encouraging representations that are less sensitive to domain-specific differences. However, many adversarial approaches rely on training a separate domain discriminator, which can be unstable and difficult to train. In this paper, we propose to use a discriminator-free adversarial learning network (DALN) for unsupervised domain adaptation in speech classification tasks. DALN reuses the task classifier as an implicit discriminator and aligns domains through the nuclear-norm Wasserstein discrepancy on classifier outputs. We extensively evaluate the performance of DALN on intra-lingual, cross-lingual, and noisy adaptations for speech emotion recognition and pathological speech detection. Extensive experimental results show that DALN generally achieves the best performance across the considered tasks and domain shifts, outperforming discriminator-based domain adaptation approaches.

0.12

CAD: Conflict-Aware Decoding to Mitigate Cross-Modal Hallucinations in Omnimodal Large Language Models

eess.AS cs.SD ·2026-08-26 ·Yuchen Deng, Chang Sun, Hai-Tao Zheng, Feidiao Yang, Yuxing Han
similarity 0.66
Abstract

Omnimodal large language models (Omni-LLMs) integrate audio, video, and text, yet remain vulnerable to cross-modal hallucinations, where one modality improperly influences predictions about another. Existing training-free decoders modulate modality influence through perturbation or relevance weighting, but do not assess predictive compatibility within the joint audio-visual branch. Because joint-branch discrepancies may indicate either harmful interference or useful complementarity, reliable intervention requires assessing both discrepancy magnitude and actionability. To this end, we propose Conflict-Aware Decoding (CAD), a training-free framework comprising Potential Conflict Magnitude Estimation (PCME) and Conflict Actionability Assessment (CAA). PCME quantifies potential conflict using audio-video disagreement and the deviation of the joint prediction from a relevance-weighted unimodal reference. CAA then applies Dempster-Shafer reliability discounting to task-space answer relations, using query relevance and answer decisiveness to determine whether intervention is warranted. When an actionable conflict is identified, CAD selectively reallocates decoding weight from the joint branch to the unimodal branches. Experiments on CMM, AVHBench, WorldSense, and VideoMME show that CAD consistently outperforms the base decoder and competitive training-free methods across multiple audio-visual backbones. On Qwen2.5-Omni-7B, CAD improves overall accuracy by 14.1 and 8.0 percentage points on CMM and AVHBench, respectively, without model retraining.

0.34

EMODE: Dynamic Para-Semantic Experts for Emotion-Aware Speech Language Modeling

cs.CL cs.MMcs.SD ·2026-10-03 ·Jianan Pan, Yiwen Gu, Xinze Li, Rui Wang, Kejie Huang
benchmark similarity 0.64
Abstract

Large speech language models have demonstrated strong capabilities in unified cross-modal understanding and generation, yet paralinguistic cues, especially emotion, remain difficult to preserve. Existing systems typically rely on entangled acoustic representations, which allow the underlying language model to depend excessively on recovered lexical content instead of grounding its behavior in acoustic-prosodic evidence. We address this limitation with EMODE, an emotion-aware speech language model built around \textbf{Dynamic Para-Semantic Experts (DPSE)}. DPSE decomposes continuous speech features into semantic and paralinguistic pathways, routes them dynamically, and fuses them before integration into the language model. To turn this structural decomposition into functional specialization, EMODE is trained with a three-stage curriculum consisting of semantic warm-up, paralinguistic activation, and joint refinement, guided by Orthogonal Expert Guidance (OEG), Semantic-to-Acoustic Alignment (SAA), and Gating Diversity Regularization (GDR). Experiments on SER test, empathetic response evaluation, and the newly constructed bilingual MEPA benchmark show that EMODE improves the balance between lexical fidelity and emotional sensitivity, strengthens affect-grounded response generation, and exposes the value of explicit para-semantic factorization for robust cross-corpus emotion understanding.

0.14

ReH-FUSE: Reliability-Aware Hierarchical Fusion of Experts for Multimodal Emotion Recognition in Conversation

cs.LG ·2026-09-12 ·Guan-Hua Wen, Hou-Chiang Tseng, Kuan-Yu Chen
similarity 0.63
Abstract · 5 pages, 2 figures, 6 tables. Submitted to ICASSP 2027

Multimodal emotion recognition in conversation (ERC) requires adapting to the instance-dependent reliability of different evidence sources. Lexical content may be decisive, vocal expression may provide complementary cues, or accurate recognition may require cross-modal interaction; fixed fusion does not explicitly account for this variation. We propose ReH-FUSE, a reliability-aware framework with dialogue-aware text, audio, and cross-modal experts. Its decision-level router first models the relative preference between text and audio and then balances the resulting unimodal mixture against the cross-modal expert. This factorization separates unimodal competition from cross-modal selection. Across three independent runs on IEMOCAP, ReH-FUSE achieves 74.34% weighted F1 and 73.11% macro F1; on MELD, it achieves 68.03% weighted F1. Controlled ablations show that learned routing outperforms uniform expert averaging and benefits from cross-modal interaction.

0.24

Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Grounding

cs.SD cs.AIeess.AS ·2026-09-26 ·Abdelrahman Mohamed, Lars Kai Hansen, Zheng-Hua Tan
similarity 0.63
Abstract

Audio emotion recognition (AER) typically assigns a single label to an entire recording, leaving the temporal scope of that label ambiguous when multiple speakers and affective events are present. We address this limitation by reformulating AER as a Temporal Affective Grounding (TAG) task that associates emotions with temporally bounded speech spans and vocal tone descriptions. To support this formulation, we curate temporally annotated versions of existing emotion recognition datasets and construct recordings containing two to four affective speech spans, including overlapping speech. Training in this longer format with a standard language-modeling objective can degrade both emotion recognition and temporal grounding performance, while tone descriptions can provide shortcuts for emotion prediction. To address these challenges, we introduce Masked Temporal Affective Grounding (M-TAG), a supervised training objective that combines full-sequence language modeling with emotion and timestamp cross-entropy losses under attention masking. The masking varies the context visible to emotion-label tokens to reduce reliance on shortcuts and improve generalization, while the timestamp loss incorporates a distance-aware weight to penalize larger temporal errors. We evaluate EMO-TAG, a model fine-tuned using our dataset and objective, on emotion recognition and affective temporal-grounding against three AER and audio-language baselines: Flamingo-Next, Audio-Reasoner, and AffectGPT. Our results show that existing models achieve limited affective temporal-grounding despite competitive emotion recognition performance.

0.11

SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training

cs.SD cs.CL ·2026-08-31 ·Eunseo Choi, Hyunku Kang, Chanwoo Kim
top venue similarity 0.63
Abstract · Accepted to INTERSPEECH 2026

Speech emotion recognition (SER) faces two fundamental challenges: scarcity of labeled data and inter-speaker variability, both of which hinder generalization of emotion recognition systems. While prior adversarial approaches address speaker variability, they fall short in leveraging powerful pre-trained representations. We propose SISER (Speaker-Invariant Speech Emotion Recognition), integrating wav2vec 2.0 as a feature encoder and ECAPA-TDNN as a speaker discriminator within an entropy-based adversarial training scheme. wav2vec 2.0 provides rich self-supervised representations that alleviate dependency on large labeled datasets, while ECAPA-TDNN enables suppression of speaker identity via a stronger adversarial signal than shallow classifiers. Evaluated on IEMOCAP, SISER achieves a UA of 60.63%, outperforming the baseline (51.15%) and wav2vec 2.0 without speaker suppression (56.46%), with ablation emphasizing that the choice of speaker classifier architecture is a key factor.

0.45