CS Paper Radar

DNN-Based Frequency-Dependent Estimation of Speech, Music, and Noise Power in Acoustic Mixtures for Hearing-Aid Scene Analysis

Mats Lang, Thomas Haubner, Nina Kiessling, Christoph Hoog Antink, Henning Puder

eess.AS ·submitted 2026-08-18 08:10 UTC · v1

Abstract

Acoustic scene analysis is essential for adapting hearing-aid signal processing algorithms to the current listening environment. However, state-of-the-art (SOTA) systems typically rely on multiple independent estimators for tasks such as scene classification, Voice Activity Detection (VAD), or Signal-to-Noise Ratio estimation, which increases computational complexity and fails to exploit dependencies between related tasks. To address this problem, we propose a unified and interpretable acoustic scene representation by decomposing the observed mixture spectrum into speech, music, and noise power components. This is motivated by the typical listening targets of hearing-aid users. In particular, we estimate time- and frequency-dependent power proportions by a causal low-complexity Deep Neural Network, from which multiple downstream acoustic scene analysis measures can in principle be derived by simple post-processing. In this work, we validate the proposed representation using VAD as a representative downstream task and show performance comparable to a SOTA estimator while providing a substantially richer scene description.

Comment
Accepted to International Workshop on Acoustic Signal Enhancement (IWAENC) 2026

Closest papers

Nearest neighbours by meaning across the whole index.

SAS: Scene-Annotated AudioSet Strong for Joint Scene-Event Analysis

cs.SD ·2026-11-06 ·Michel Olvera, Paraskevas Stamatiadis, Romain Serizel, Slim Essid, Gaël Richard
similarity 0.66
Abstract

Context is central to human auditory perception, as listeners interpret sounds within their acoustic environment to resolve ambiguity. However, large-scale datasets that jointly capture sound events and acoustic scenes remain limited. We address this by augmenting AudioSet Strong with clip-level scene annotations under two taxonomies for joint sceneevent modeling. A corpus-scale analysis shows that scene identity is better captured by events distinctive to a scene than by those that merely occur frequently. Reweighting event co-occurrences toward scene-specific associations produces representations that better align with human-derived taxonomies, while also exposing scenes where the acoustic and semantic views diverge. This geometric divergence predicts classification errors, while supervised validation confirms that the proposed scene annotations provide meaningful complementary information for developing scalable context-aware machine listening systems.

0.18

Hierarchical Interpretable Multitask Audio Representation Learning: An Analysis of Event-Scene Relationships through the Information Bottleneck

eess.AS cs.AI ·2026-10-15 ·Claudia Montero-Ramírez, Alba Martínez-Serrano, Carmen Peláez-Moreno
benchmark similarity 0.64
Abstract

Joint Acoustic Scene Classification (ASC) and Audio Tagging (AT) models traditionally rely on unconstrained multihead architectures. However, this design ignores the natural hierarchical dependency between sounds and acoustic scenes. In this work, we investigate a hierarchical (sequential) conditioning paradigm that predicts scene categories directly from intermediate event probabilities. Through a large-scale empirical benchmark with 66 full training runs across three state-of-the-art backbones (PaSST, PANNs, and BEATs) and different loss-weighting settings (beta in [0.0, 1.0]), we show that hierarchical conditioning achieves competitive classification performance while altering multitask optimization dynamics. Unlike multihead baselines, which sequentially solve the easier ASC task before optimizing AT, the hierarchical architecture enables synchronized dual-task learning for specific loss-weighting settings (beta in [0.6, 0.9]). Furthermore, we propose a multi-perspective interpretability framework that combines an analysis of the information bottleneck, manifold geometry, and Formal Concept Analysis (FCA) on a Term-Frequency Inverse-Document-Frequency (TF-IDF) event representation, providing complementary insights into event-scene relationships. Our findings reveal that hierarchical conditioning systematically yields lower conditional entropy, higher mutual information, and superior latent cluster separability (higher Silhouette scores), establishing a more structured and semantically faithful intermediate feature space.

0.14

Joint Language Modeling for In-the-Wild Speech-Scene Generation with Distinct Token Representations

eess.AS cs.AIcs.LG ·2026-11-19 ·Semin Kim, Ju Yeon Kang, Sunghwan Ahn, Seonuk Kim, Nam Soo Kim
similarity 0.64
Abstract

Speech-scene generation aims to synthesize intelligible target speech together with diverse environmental sounds as a coherent acoustic scene. The task is considered challenging in two respects. Speech and environmental audio differ substantially in their semantic and acoustic characteristics, yet existing approaches typically model the complete scene using a single mixed acoustic representation. At the same time, the two sounds are closely related in real-world speech scenes, whereas conventional approaches usually rely on simulated scenes mixed from independently recorded sources. We propose a large language model (LLM)-based framework for joint modeling of distinct speech and environment token streams. A source-aware neural audio codec extracts temporally aligned speech and environment tokens from mixed audio, and a pretrained LLM generates both token streams in an interleaved fashion conditioned on the natural-language scene description and speech content. This formulation preserves source-specific representations while modeling the token streams within a shared generative context. Since both streams are extracted directly from mixed recordings, the model can also be adapted to in-the-wild speech scenes without isolated sources. Our experimental results demonstrate that the proposed framework improves speech intelligibility, scene relevance, and audio quality over existing speech-scene generation techniques, with consistent gains from real-world adaptation. We further investigate the effects of distinct token representations and interleaved generation, and analyze whether the environmental sound remains stable while speech is active and how the two streams interact.

0.09

REDnet: Recursive Encoder and Decoder for Speech Separation under Unknown Number of Speakers and Variable Number of Microphones

eess.AS ·2026-08-25 ·Fulin Wu, Zhong-Qiu Wang
similarity 0.63
Abstract · in submission

We propose $\textit{recursive encoder and decoder}$ (RED) for building a single deep neural network (DNN) model that can separate multi-speaker mixtures containing unknown numbers of speakers and variable numbers of microphones arranged in an unknown geometry, a task that has not been studied yet. The decoder of RED recursively detects whether there are active speakers left and separates one speaker at a time. It is designed to be trained in an end-to-end fashion to improve separation performance. The encoder of RED recursively encodes each microphone channel of the input mixture, sequentially incorporating spatial cues. Combining both, the DNN can be trained to separate mixtures not only with unknown numbers of speakers but also with variable numbers of microphones, achieving state-of-the-art performance on multiple public datasets.

0.18

Hear Every Sound Object: Rescuing Overlooked Sources in Auditory Scene Analysis

eess.AS ·2026-10-22 ·Younghoo Kwon, Yongjoon Lee, Jung-Woo Choi
similarity 0.63
Abstract · 5 pages, 4 figures, Submitted to ICASSP 2027

Auditory scene analysis (ASA) models extract information about individual sound sources from a complex auditory mixture. Recent object-based ASA models process the mixture through a sequence of modules and split it into source-wise features (i.e., object features) only at the last layer. However, substantial disparities in performance across sources limit overall performance. Our analysis using centered kernel alignment (CKA) reveals that CKA scores correlate strongly with per-source performance, with their disparities widening from intermediate to deeper layers. These findings motivate the early split and exchange of object features to address source-wise performance disparities before they widen in deeper layers. To this end, we propose PARASITE, a plug-in refinement architecture that splits object features at earlier layers and progressively refines them through sequential layers. Each layer integrates object features separated at different depths using Object-wise Gated Fusion (OGF) and then refines them using Windowed Cross-Object Attention (WCOA). We introduce the DeepASA++ model, which combines PARASITE with an ASA backbone and delivers state-of-the-art results on both ASA2 and STARSS23. DeepASA++ achieves a SELD score of 0.240 on STARSS23 and surpasses prior multi-stage inference architectures, including chain-of-inference.

0.12

Evaluating Inference-time Algorithms for Semantic Sound Scene Segmentation

eess.AS cs.SD ·2026-10-02 ·Sripathi Sridhar, Gordon Wichern, Yoshiki Masuyama, Mark Cartwright, Jonathan Le Roux
published similarity 0.63
Abstract · Accepted to DCASE 2026

Spatial semantic segmentation of sound scenes (S5) involves the detection and extraction of target sound events in audio files. However, in typical detection-extraction pipelines, errors in the detection module propagate to the extraction module, degrading overall performance. In this work, we develop a multichannel detection-extraction model and evaluate inference-time algorithms to improve the class detection and target sound extraction in a multi-stage setup. We evaluate two fitness functions: binary cross entropy and mixture consistency; two relabeling strategies: naive and conditional confusion matrix relabeling; and three source estimate evaluation models: a multi-label classifier, a fine-tuned single-source classifier, and an off-the-shelf audio judge. We thoroughly evaluate the impact of these design choices on the DCASE 2025 Task 4 dataset, and validate entropy-based gating of inference-time updates on the held-out evaluation set, laying the groundwork for inference-time updates of S5 systems in real-world scenarios.

0.28

A Stem-Agnostic Approach to Hybrid AI Music Detection

cs.SD eess.AS ·2026-09-22 ·Richa Namballa, François Rigaud, Romain Hennequin
similarity 0.63
Abstract · 4 pages + references, 5 figures, submitted to the 2027 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

The inclusion of generative audio in the music production process has led to an increase in hybrid music tracks that blend authentic human performances with AI-generated stems, challenging traditional AI music detectors which operate in a binary setting. In this work, we propose a stem-agnostic framework for identifying synthetic audio sources within hybrid musical mixtures. We introduce the inspectrogram, a novel time-frequency representation that maps localized probabilities of synthetic content across the audio spectrum. By combining the inspectrogram with a Wiener filter estimating target stem energy dominance, a single CNN model evaluates whether the specific stem is generated. Trained on rendered hybrid mixtures and evaluated across various stem classes, our model achieves strong performance on high-frequency sources such as vocals, drums, and guitar, but struggles on the low-frequency, narrow-band bass. We conclude that the quality of separation impacts the detection accuracy and identify source separation as a primary bottleneck and a crucial direction for future research.

0.29

UltraM2M: Leveraging Text Transcripts and Mixture Constraints for Weakly-Supervised Speech Enhancement

eess.AS ·2026-10-04 ·Liu, Jiachen, Wu, Fulin, Wang +1 more
similarity 0.61
Abstract · in submission

We propose UltraM2M, a weakly-supervised speech enhancement algorithm building upon the recent unsupervised mixture-to-mixture (M2M) algorithm. M2M realizes unsupervised speech enhancement by training deep neural networks on a set of real-recorded noisy-reverberant multi-channel mixture signals to estimate target speech and non-target signals. The two estimated signals are penalized by a so-called mixture-constraint (MC) loss, which constrains them to reconstruct the observed mixture signals. Although shown to be effective, the mixture constraint may be too weak to enable sufficient noise reduction. To deal with this, UltraM2M extends M2M by further leveraging text transcripts of real-recorded mixtures to design an automatic speech recognition (ASR) loss to penalize the estimated speech signal. The ASR loss can be viewed as a form of weak supervision that could help unsupervised enhancement. Evaluation results on the CHiME-4 dataset show the effectiveness of UltraM2M.

0.18