CS Paper Radar

A Robust, Explainable, and Efficient Deep Metric for Speech and General Audio Quality

Konstantin Schmidt, Pablo M. Delgado, Andreas Brendel

eess.AS ·submitted 2026-10-26 17:41 UTC · v1

Abstract

We present a robust Speech and Audio Quality Metric (SAQM) that merges features of traditional quality metrics with a CNN trained on a large amount of data using contrastive learning methods. To overcome the problem of limited amounts of human-labeled training data, we propose two pretraining strategies relying on pseudo-labels. One uses an established objective metric PEAQ-2f (based on ITU-R BS.1387) as surrogate loss, the other is based on contrasting a wide range of coding modes by a rule-of-thumb ordering based on expected perceived quality. As a key novelty, our approach lets us generate saliency maps by propagating gradients to identify which feature has the greatest impact on the final metric outcome and enables our system to provide insights regarding the type and extent of distortions present in the tested audio signal. The resulting (SAQM) operates at low computational complexity, performs comparably to (SOTA) metrics on speech while still performing well on general audio content.

Closest papers

Nearest neighbours by meaning across the whole index.

XSQ-AST: An Explainable Audio Spectrogram Transformer Framework for Localising Synthetic Speech Artifacts

eess.AS cs.LGcs.SDeess.SP ·2026-09-21 ·Ben Heritage, Luca Resti, Mónica Villanueva Aylagas, Timothy Mehlenbacher, Konrad Tollmar +1 more
similarity 0.71
Abstract · Submitted to IEEE ICASSP 2027

Localising artifacts in synthetic speech remains challenging, as most evaluation methods yield only global quality scores. This paper presents XSQ-AST, a framework that combines the SQ-AST speech quality model with WhisperX phoneme alignment and multiple saliency methods to produce temporally localised artifact diagnostics without model retraining. Saliency maps are projected onto continuous distributions via kernel density estimation and onto phoneme boundaries via phoneme-discretised saliency maps. A 40-participant listening test validated the framework across five perceptual dimensions. Attention Rollout, Attention Flow and an adapted GradCAM produced temporal distributions that correlated with listener highlights, with different methods best suited to different artifact types. An AUC-ROC analysis confirmed discrimination above chance.

0.27

HE-NAC: High-Efficiency Neural Audio Coding via Hierarchical and Residual Multi-Band Quantization

eess.AS cs.SD ·2026-10-11 ·Darius Pétermann, Wootaek Lim, Seungkwon Beack, Minje Kim
top venuecode/data similarity 0.69
Abstract · Published online as Early Access in IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP). Code: https://github.com/darius…

High-fidelity audio coding requires faithful reconstruction of perceptually important high-frequency components, which are often lost due to the aggressive temporal decimation and quantization of neural codecs. State-of-the-art neural audio coding methods rely on adversarial learning to hallucinate these missing components, yielding perceptually plausible but imprecise reconstructions; a limitation for high-fidelity music applications. As a remedy, we propose high-efficiency neural audio coding (HE-NAC), which provides a multi-band extension of the popular residual vector quantization (RVQ) scheme. HE-NAC introduces an effective training mechanism that harmonizes a bitrate-free blind upsampling process, residual learning to compensate it with minimal bitrate, and a multi-head decoder architecture for band-specific reconstruction. As a result, HE-NAC achieves a balanced trade-off, reliably reconstructing the core-band ($\leq$ 3 kHz) component while preserving overall sound fidelity. Subjective listening tests and objective evaluations demonstrate that HE-NAC outperforms both the DAC baseline and MPEG HE-AAC at equal or lower bitrates.

0.51

ESDCodec: Ultra-Low Bitrate Speech Codec with Enhanced Semantic Disentanglement and Subband Decoding

eess.AS ·2026-11-10 ·Nan Xu, Kecan Mao, Xin Gu, Yi Zhao, Zhi Gong
similarity 0.68
Abstract · 5 pages, 2 figures

Despite recent advances in neural speech codecs, achieving highfidelity speech reconstruction at low bitrates remains a formidable challenge. To address this limitation, we propose ESDCodec, a speech codec that integrates a thoroughly enhanced semantic quantizer and a conditioned decoder network. Specifically, we employ a randomly initialized and frozen codebook, followed by a lightweight projector, to encode semantic details entirely within a linear space while enhancing codebook utilization. To further improve perceptual quality, we design a condition network that explicitly injects prior subband knowledge into the upsampling decoder. Taking the de-quantized feature as input, this network predicts subband signals, thereby providing fine-grained guidance for waveform reconstruction. Extensive experimental results show that ESDCodec achieves superior reconstruction performance at an ultra-low bitrate of 0.85kbps. For LLM-based speech generation tasks, ESDCodec also consistently outperforms existing codec models.

0.15

Is Semantics Enough for Speech Mean Opinion Score Prediction?

cs.SD ·2026-09-03 ·Tianyu Lan, Yufei Shi, Yang Ai, Honghao Sun, Huipeng Du +1 more
publishedtheory similarity 0.67
Abstract · 5 pages, 2 figures, accepted by ISCSLP 2026

Mean Opinion Score (MOS) is the gold standard for evaluating synthesized speech naturalness. However, current automatic MOS predictors are dominated by self-supervised learning (SSL) models that prioritize high-level semantics, potentially compromising their ability to capture critical acoustic details. In this paper, we systematically investigate representations from three paradigms: SSLs, acoustic-only neural audio codecs (NACs), and unified NACs that integrate semantics into reconstruction-based architectures. Extensive benchmarking on the standard BVCC and multiple out-of-domain (OOD) datasets demonstrates that features synergizing semantic understanding with fine-grained acoustic modeling achieve a higher performance upper bound in speech quality assessment. Ultimately, our findings highlight that semantics alone are not enough; a dual focus on semantic content and acoustic fidelity is essential for robust MOS prediction.

0.31

Exploring the Design Space of Representation Learning for Audio Transformations

cs.SD eess.AS ·2026-08-28 ·Sungho Lee, Marco Martínez-Ramírez, Junghyun Koo, Wei-Hsiang Liao, Kyogu Lee +1 more
published similarity 0.67
Abstract · Accepted to ISMIR 2026

Neural audio representation learning has enabled a range of content-oriented applications, but the resulting features remain limited for tasks involving audio processing. Furthermore, it is not obvious what processing-aware representations should capture: the processing itself, abstracted away from source content, or the processed audio that retains it. Existing approaches implicitly commit to one or the other and also differ in their models, data, and evaluation, obscuring which design choices drive their behavior. We address both questions within a unified framework of three objectives: processing consistency, description alignment, and equivariance via forward prediction. We compare all combinations of the objectives under a controlled setup and reveal their relative strengths and interactions. Our framework produces both a transformation embedding and a processed-audio embedding, and we find that the two play complementary roles: distance-based tasks favor the former, while probe-based tasks favor the latter. Combined with improvements in network architecture and training pipeline, our representations outperform prior baselines across retrieval, probe-based evaluation, and style transfer.

0.44

Lightweight Neural Prediction for MDCT-Based Perceptual Audio Coding

eess.AS cs.SD ·2026-11-06 ·Niccolo Abate, Gabriel Soule, Marina Bosi
publishedcode/data similarity 0.67
Abstract · 10 pages, 2 figures, 9 tables. Presented at the 161st AES Convention, Nashville, TN, USA, October 30-November 1, 2026 (published version: ht…

Perceptual audio codecs often use predictive coding to further reduce signal redundancy. Rather than learning the entire codec pipeline end-to-end, we insert a small neural network as a drop-in, deterministic predictor in an otherwise conventional codec: the network estimates the next Modified Discrete Cosine Transform (MDCT) coefficient block from a short context of reconstructed blocks, and the encoder transmits the quantized residual. We compare Multilayer Perceptron (MLP) and Convolutional Neural Network (CNN) predictors against a non-predictive baseline and a per-line Normalized Least Mean Squares (NLMS) linear predictor (similar to the MPEG-2 AAC Main Profile predictor). The CNN consistently outperforms the MLP, the linear predictor, and the baseline in signal-to-noise ratio (SNR), noise-to-mask ratio (NMR), and MUSHRA evaluation scores. Using a 512-sample analysis window (w = 512), our neural predictors recover, and the CNN exceeds, the coded quality obtained by the non-predictive baseline after quadrupling the analysis-window length to w = 2048, at substantially lower algorithmic delay. The CNN uses only 0.13M parameters (roughly 20x fewer than the MLP) and has a decoder real-time factor (RTF) of ~0.10 on a consumer CPU.

0.36

LAION-Audio-XL: A Web-Scale Corpus and Scaling Studies for Contrastive Language-Audio Pretraining

cs.SD cs.CL ·2026-11-08 ·Gijs Wijngaard, Felix Förster, Ross Wightman, Marianna Nezhurina, Christoph Schuhmann +2 more
datasetcode/data similarity 0.67
Abstract · Project page: https://projects.laion.ai/clap2/

Recent progress on contrastive audio-language models can be attributed mainly to the scale and quality of their training data. Larger training datasets expose models to a broader range of sounds and corresponding audio captions. To exploit this, we collect 1.48 billion audio samples from academic datasets, internet datasets, web collections, Common Crawl, and YouTube, and process them with a multi-stage curation pipeline. Furthermore, we generate short and long captions for samples in our corpus that lack them. We use our dataset to study the scaling behavior of contrastive audio-language models based on the NaFlex architecture for the audio encoder. Our experiments identify trade-offs between model size and training data for a given compute budget. Moreover, our resulting models, CLAP2, outperform prior audio-text models of comparable size on the MAEB and HEAR benchmarks. We release our dataset, captions, model checkpoints, and code.

0.26

PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation

cs.MM cs.AIcs.SD ·2026-09-04 ·Yuchen Sun, Qian Yang, Jun Wang, Detai Xin, Guoqiao Yu +2 more
top venuebenchmarkcode/data similarity 0.67
Abstract · 19 pages, 10 figures, 4 tables. Accepted at ACM Multimedia 2026 (MM '26). This arXiv version includes supplementary appendices not included …

Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality. Existing benchmarks either treat audio as an auxiliary component of video quality or assess it in isolation from audiovisual grounding, making it difficult to diagnose where current systems truly succeed or fail in audio generation. We present PRISM-Bench, the first audio-centric diagnostic benchmark for T2AV generation. Built from a rigorously curated dataset of 900 human-verified samples, PRISM-Bench factorizes audio evaluation along two orthogonal axes: audio type (Speech, Music, and Sound) and sound-source visibility (On-screen vs. Off-screen). It evaluates generated content across four perceptual dimensions (Audio-Visual Coherence, Audio Quality, Audio Expressiveness, and Prompt Following) with 35 fine-grained criteria. To ensure reliable assessment, we adopt an enhanced MLLM-as-a-Judge protocol based on blind, side-by-side comparison against ground-truth references, demonstrating strong alignment (over 70% mean agreement) with human raters. Our evaluation of recent T2AV systems highlights a significant performance gap between frontier and open-source models. Furthermore, we demonstrate that current generation paradigms overfit to perceptual fidelity while struggling with complex grounding and control tasks, particularly in generating music and synchronized On-screen audio.

0.46