CS Paper Radar

ParaGeo: Decomposing Paralinguistic Variation into a Shared Latent Geometry

Yuhan Liu, Yuxuan Ou, Ruoxi Su, Mohamed Ahmed Zaki, Yunbo Long

cs.SD cs.LG ·submitted 2026-10-02 10:46 UTC · v1
benchmarkcode/data

Abstract

Speech delivery varies with both the requested paralinguistic attribute and the linguistic content. We introduce ParaGeo, a matched-content decomposition of paralinguistic variation in a frozen speech language model. Synthesized audio tokens are replayed with a fixed listening prompt; pooled key/value (K/V) representations are centered and projected into a shared low-dimensional space. Our GLM-4-Voice probe spans 80 requested controls from 12 benchmark families across eight sentences. With a globally fitted calibration basis, content-held-out centroid accuracy using this basis is 9.49% versus a 1.25% permutation baseline; same-label cross-content cosine similarity is 0.285 versus 0.017, and both conditional permutation tests yield p = 0.001. A separate ten-scenario, six-style probe reveals reproducible contrast directions across scenarios. Static, additive, and temporal interventions produce attribute-, layer-, and schedule-dependent response profiles. These results provide a shared coordinate representation for measuring paralinguistic structure and an empirical starting point for latent speech control. Code is available at https://github.com/yuhanlydia/ParaGeo.

Closest papers

Nearest neighbours by meaning across the whole index.

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

cs.SD cs.AI ·2026-09-01 ·Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh, Bhiksha Raj
code/datahighly cited similarity 0.72
Abstract

Audio language models are designed to understand speech, yet it remains unclear whether they capture how something is said beyond what is said. We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5-Omni-7B, and Chroma-4B, using the Expresso dataset with controlled speaking styles. We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output. All models strongly encode speaking style in the late encoder, that is, the top third of the audio encoder's layers, but this information is consistently degraded before reaching the output. The projector reshapes representation geometry without removing information, while decoders differ in how much style they preserve depending on architecture and training objective. At the output level, models fall into two behaviors. Some are content driven, where predictions depend mainly on text. Others are acoustic driven, where predictions vary with speaking style. The leakage metric quantifies this difference, and qualitative results confirm it. Overall, we identify a gap between what models encode and what they use, highlighting a key limitation in current audio language models.

0.58

ParaCalib: Semantically Calibrated Paralinguistic Modeling for Depression Detection

eess.AS ·2026-10-01 ·Yuxin Li, Yifei Li, Yi-Wen Chao, Xiangyu Zhang, Eng Siong Chng +1 more
top venue similarity 0.69
Abstract

Vocal behavior provides important signals for speech-based depression detection, but its interpretation often depends on what is being said and how it functions in context. However, existing methods typically treat acoustic cues as context-independent markers, making it difficult to distinguish vocal form from its context-dependent communicative function. We propose ParaCalib, a framework that semantically calibrates paralinguistic behavior by interpreting vocal patterns relative to utterance-level semantic context and inferred communicative function and representing them in a comparable state space. Concretely, ParaCalib uses an Audio-Language Model (ALM) to generate contextualized vocal descriptions and an LLM-based Paralinguistic State Extractor (PSE) to map these descriptions into a structured Semantically Calibrated Paralinguistic (SC-Para) representation. ParaCalib achieves the highest mean Macro-F1 among the evaluated methods, reaching 71.9\% on DAIC-WOZ and 90.5\% on MODMA. Our controlled analysis provides direct evidence of semantic calibration at the PSE stage: under a fixed caption-level acoustic description, varying the accompanying semantic context changes the inferred depression-related paralinguistic evidence. Our exploratory analysis further identifies recurring configurations of paralinguistic states associated with depression labels rather than a single uniformly dominant state.

0.39

Layer-wise Linear Probing of an Encoder-free Audio-capable Language Model

eess.AS ·2026-11-16 ·Laishram Rahul, Shruthi B. S., Parismita Gogoi, Sishir Kalita, Deepak KT +1 more
published similarity 0.68
Abstract · Accepted in The 29th International Conference of Oriental COCOSDA

Encoder-free audio-capable multimodal language models pro- vide a unique setting for studying how speech representations emerge inside a language-model decoder. In Gemma-4-12B, short waveform frames are projected directly into the shared embedding space, and all subsequent contextual processing is performed by the decoder layers. We use this architecture to examine where acoustic, phonetic, segmental, and paralinguis- tic information becomes linearly accessible across depth. Us- ing the TIMIT corpus, we extract all 49 representations from the frozen model, including the embedding output and the out- puts of its 48 decoder layers. We train linear probes for log- F0, log-energy, voicing, phone identity, word-onset structure, speaker identity, recorded speaker gender, and dialect-region labels. The results show a broad progression in linear acces- sibility. Acoustic properties are most strongly decodable in the early layers, paralinguistic attributes are most accessible in the early-to-middle layers, and phone identity and boundary- sensitive information reach their highest performance around layer 11. Several properties remain strongly decodable across broad near-peak regions rather than at a single uniquely optimal layer. In deeper layers, phone identity and closed-set speaker identity decline sharply, while voicing, log-F0, and gender re- main more accessible. These findings provide an initial map of how speech information develops when the decoder is the first contextual stage to process audio.

0.16

PromptSep-Speaker: Generative Target Speaker Extraction via Multimodal Prompting and Semantic Alignment

cs.SD eess.AS ·2026-10-16 ·Yutong Wen, Ke Chen, Prem Seetharaman, Haonan Chen, Oriol Nieto +3 more
similarity 0.68
Abstract · Submitted to ICASSP 2027

Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.

0.11

PAID: Phoneme-Aligned Acoustic Units for Cross-Corpus L1-Background Classification

eess.AS ·2026-09-09 ·Yangyang Qu, Michele Panariello, Massimiliano Todisco, Nicholas Evans
datasethot direction similarity 0.68
Abstract

First-language (L1) background classification from second-language English speech aims to identify pronunciation-related acoustic cues that remain informative across speakers and corpora. Global pooling is commonly used to aggregate frame-level speech representations, but averaging features across an utterance may obscure local acoustic differences in how individual phonemes are pronounced. To model these pronunciation-related cues before utterance-level aggregation, we introduce Phoneme-Aligned Input Design (PAID), which constructs acoustic units from frozen Whisper representations using transcript-assisted forced alignment. PAID applies a shared nonlinear mapping to each acoustic unit before utterance-level aggregation and incorporates auxiliary phoneme-label embeddings. We construct a six-L1 dataset with speaker-disjoint partitions and evaluate cross-corpus performance on the Speech Accent Archive (SAA). PAID achieves 89.72% Macro-F1 on the source Test set and 80.14% on SAA. Matched comparisons show that phoneme-aligned acoustic-unit modeling improves SAA Macro-F1 by 13.44 percentage points over global pooling. To address uneven classification performance across L1 backgrounds, we additionally evaluate class-sensitive training, which improves observed weakest-class performance.

0.47

CycleSpeech: Reciprocal Alignment for Instruction-Controlled Speech Synthesis and Paralinguistic Understanding

cs.SD ·2026-09-21 ·Huan Liao, Haonan Han, Xingwen Han, Dekun Chen, Yuancheng Wang +1 more
dataset similarity 0.67
Abstract

Instruction-controlled speech synthesis and paralinguistic understanding are often trained independently, leaving reciprocal feedback between the two tasks underexplored. We introduce CycleSpeech, a framework that connects generation and understanding through a shared, structured voice profile that serves as a common target for supervision and reciprocal feedback. The forward cycle assesses whether synthesized speech expresses the intended attributes by comparing recovered and target profiles. The backward cycle evaluates whether profiles inferred from real speech can guide reconstruction of the source speaking style. To support both directions, we construct a bilingual dataset of 20,046 examples pairing instructions, target speech, speaker references, and structured profiles. Building on joint supervised fine-tuning, CycleGRPO alternates policy updates using reciprocal rewards grounded in profile consistency and speaking-style reconstruction. Fixed target profiles anchor feedback from the evolving counterpart. This procedure requires neither human preference annotations nor an additional preference-trained reward model. Evaluations on Chinese and English benchmarks show improved instruction adherence and profile recovery while maintaining competitive synthesis quality. Compared with Step-Audio-2-mini, CycleSpeech improves instruction-match accuracy by 4.50 and 10.06 percentage points in Chinese and English, respectively. Controlled ablations further support the contribution of cycle feedback to generation control. These results support structured voice profiles as an interface for reciprocal training between speech generation and paralinguistic understanding. An online demo is available at https://cyclespeech.github.io.

0.32

CASA: Content-Acoustic Speaking Assessment with Speech Encoder and Large Language Model

cs.CL eess.AS ·2026-08-13 ·Nhan Phan, Ilona Lähteenmäki, Anna von Zansen, Olli-Pekka Pauna, Yaroslav Getman +2 more
code/data similarity 0.67
Abstract · To be submitted to ICASSP 2027. Code is available at https://github.com/aalto-speech/casa

Research on automatic speaking assessment (ASA) has increasingly adopted multimodal speech large language models to assess learners' speaking performance. However, existing studies provide limited analysis of how acoustic and content information contribute to predictions and how stable the resulting performance is. We propose CASA, a simpler architecture combining Whisper-medium and Qwen3.5-2B that achieves state-of-the-art performance while providing a more interpretable separation between speech delivery and content. On the Speak & Improve Corpus 2025, CASA achieves a root mean square error (RMSE) of 0.358, improving on the previous best RMSE while using approximately half the estimated inference parameters. The general-purpose architecture is designed for adaptation to other ASA corpora without structural changes and relies on three handcrafted fluency features. Through ablations and repeated runs, we analyze the individual and complementary contributions of acoustic and content information, examine performance variability, and demonstrate the potential of large language model reasoning for training-free content validation.

0.26

Training a Compact, Strong, and Realistic Text-to-Speech Model

cs.SD cs.AI ·2026-10-10 ·Runyuan Cai, Yu Lin, Yiming Wang, Xiaodong Zeng
code/data similarity 0.66
Abstract

Text-to-speech (TTS) systems are conventionally optimized for intelligibility and speaker similarity. However, neither objective captures the prosodic detail that distinguishes human performances from synthetic speech, leaving a realism gap that larger models and higher-quality data can narrow but not close. In the absence of an objective that directly targets this gap, realism remains a persistent challenge in TTS. We present Audio8-TTS, a compact 0.6B-parameter model for zero-shot voice cloning that achieves state-of-the-art performance in intelligibility and speaker similarity, with particularly strong results in perceptual naturalness. To align the training signal with human perception rather than signal-level fidelity, we construct a quality ladder from widely used open-source TTS systems and train a realism critic to assess generated speech against matched human recordings of the same sentences. An identity-erasure objective discourages the critic from relying on generator fingerprints. We train the model in three stages: multilingual supervised fine-tuning; group relative policy optimization (GRPO) with word error rate (WER) and speaker similarity (SIM) rewards; and GRPO guided by the realism critic. The final stage improves perceived prosodic naturalness and speaker similarity while preserving WER performance.

0.21