CS Paper Radar

Lightweight Systems for Spatial Semantic Segmentation of Sound Scenes

Mayank Mishra, Paul Magron, Romain Serizel

cs.SD eess.AS ·submitted 2026-10-08 20:19 UTC · v1
hot direction

Abstract

Spatial semantic segmentation of sound scenes (S5) jointly performs audio source separation and tagging on a multichannel audio mixture. Therefore, an S5 system can be decomposed into an audio tagging model and a separation model. The baseline S5 system in task~4 of the DCASE 2025 challenge uses M2D and ResUNet models for tagging and separation, respectively. These are large in overall size and therefore not appropriate for use in embedded devices. The main aim of this paper is to assess the performance loss in an S5 system when using lightweight tagging and separation models instead. To this end, we first compare the tagging performance of lightweight audio tagging models, namely PANNs and DyMNs, in terms of label prediction accuracy against the baseline M2D model. Secondly, we assess separation performance in terms of the scale-invariant signal-to-distortion ratio of the baseline ResUNet separator relative to that of a convolutional-based separator using SuDoRMRF. Finally, we combine the tagging and separation models across various configurations and assess the overall performance of the S5 system using the class-aware signal-to-distortion ratio.

Closest papers

Nearest neighbours by meaning across the whole index.

Evaluating Inference-time Algorithms for Semantic Sound Scene Segmentation

eess.AS cs.SD ·2026-10-02 ·Sripathi Sridhar, Gordon Wichern, Yoshiki Masuyama, Mark Cartwright, Jonathan Le Roux
published similarity 0.72
Abstract · Accepted to DCASE 2026

Spatial semantic segmentation of sound scenes (S5) involves the detection and extraction of target sound events in audio files. However, in typical detection-extraction pipelines, errors in the detection module propagate to the extraction module, degrading overall performance. In this work, we develop a multichannel detection-extraction model and evaluate inference-time algorithms to improve the class detection and target sound extraction in a multi-stage setup. We evaluate two fitness functions: binary cross entropy and mixture consistency; two relabeling strategies: naive and conditional confusion matrix relabeling; and three source estimate evaluation models: a multi-label classifier, a fine-tuned single-source classifier, and an off-the-shelf audio judge. We thoroughly evaluate the impact of these design choices on the DCASE 2025 Task 4 dataset, and validate entropy-based gating of inference-time updates on the held-out evaluation set, laying the groundwork for inference-time updates of S5 systems in real-world scenarios.

0.28

Logbook: Extremely Long-form Audio Event Understanding

eess.AS cs.AIcs.CLcs.LG+1 ·2026-10-05 ·Kwanghee Choi, Suwon Shon, Dmitriy Serdyuk, Guitang Lan, Chao-Wei Huang +5 more
benchmarkcode/data similarity 0.66
Abstract · Submitted to ICASSP 2027. Source code available at https://github.com/facebookresearch/logbook

Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we introduce Logbook, a benchmark for hour-scale audio understanding, with recordings ranging from ten minutes to six days. Given a continuous audio recording and an event label vocabulary, a system must predict a gap-free segmentation with an event label and a description per segment. We compare 52 systems, end-to-end and cascaded, and ablate fine-tuning, context length, and reasoning budget. We find the task tractable, though the best systems remain below the human reference. Also, over-segmentation is pervasive, and fine-tuning partially mitigates it. Finally, end-to-end are often better than cascaded systems, but degrades with longer context.

0.25

Learning Where to Split: Dynamic Band Splitting for Audio Source Separation

cs.SD ·2026-10-10 ·Bingzhe Zhang, Kai Li, Xiaolin Hu
code/datacommunity pick similarity 0.63
Abstract · 25 pages, 11 figures, including appendices. Code and audio demos: https://cslikai.cn/Dynamic-Band-Splitting

Audio source separation models must preserve fine spectral structure while keeping the cost of the separator tractable. Band-split architectures reduce this cost by manually partitioning the spectrum into predefined frequency bands and encoding each band as a token, but the same boundaries are used for every mixture. Because informative spectral structure varies with source content, fixed boundaries can merge discriminative components or spend tokens where little resolution is needed. We introduce Dynamic Band Splitting (DBS), an end-to-end framework that adapts boundary locations and band widths to each mixture under a fixed band-token budget. DBS predicts content-conditioned boundary scores within frequency blocks, selects a prescribed number of boundaries using hard Top-$K$, and applies content-dependent pooling to obtain compact band tokens. After separation, it smooths and expands these tokens to the original STFT grid, refines them along frequency, and decodes complex masks with a frequency-shared head. DBS improved separation over the corresponding static models on DnR v2, MUSDB18-HQ, and EchoSet, with DBS-BSRNN achieving relative gain of 29.0% in average SI-SDR on DnR v2. These results and boundary analyses on DnR v2 show that content-adaptive frequency partitioning benefits multiple separator backbones under a fixed band-token budget. Code and audio demos are available at https://cslikai.cn/Dynamic-Band-Splitting

0.37

Self-Supervised Object-Centric Representation for Multi-Task Audio Analysis

cs.SD ·2026-10-30 ·Parthasaarathy Sudarsanam, Tuomas Virtanen
published similarity 0.63
Abstract · Accepted to IWAENC 2026

This paper studies the capability of an object-centric audio representation to support multiple downstream tasks within a unified framework. The object-centric model is trained in a self-supervised setup using slot attention by reconstructing Audio Spectrogram Transformer (AST) patch embeddings from a set of latent slots. It is trained on synthetic audio mixtures, which provides controlled multi-source audio scenes. These slot representations are evaluated for the classification, detection, and separation of sound events by training task specific output layers. On the classification task, the model achieves an F1-score of 0.8, comparable to clip-level AST embeddings which achieve 0.7. For sound event detection, it reaches a macro F1-score of 0.7, outperforming clip-level AST embeddings at 0.4 and matching AST patch embeddings at 0.7, which encode finer time-frequency information. In source separation, it achieves an SI-SDR improvement (SI-SDRi) of 18.4 dB. Furthermore, in a zero-shot source separation setting, the model achieves an SI-SDRi of 14.5 dB. These results demonstrate that slot-based representations capture transferable and compositional structure in audio scenes, enabling generalization across tasks and datasets.

0.21

SAS: Scene-Annotated AudioSet Strong for Joint Scene-Event Analysis

cs.SD ·2026-11-06 ·Michel Olvera, Paraskevas Stamatiadis, Romain Serizel, Slim Essid, Gaël Richard
similarity 0.62
Abstract

Context is central to human auditory perception, as listeners interpret sounds within their acoustic environment to resolve ambiguity. However, large-scale datasets that jointly capture sound events and acoustic scenes remain limited. We address this by augmenting AudioSet Strong with clip-level scene annotations under two taxonomies for joint sceneevent modeling. A corpus-scale analysis shows that scene identity is better captured by events distinctive to a scene than by those that merely occur frequently. Reweighting event co-occurrences toward scene-specific associations produces representations that better align with human-derived taxonomies, while also exposing scenes where the acoustic and semantic views diverge. This geometric divergence predicts classification errors, while supervised validation confirms that the proposed scene annotations provide meaningful complementary information for developing scalable context-aware machine listening systems.

0.18

Less is More: Encoder-only Audio-Visual Segmentation

cs.CV cs.AI ·2026-09-24 ·Ilpo Viertola, Vladimir Iashin, Sophie Tötterström, Esa Rahtu
code/data similarity 0.62
Abstract · Submitted to ICASSP 2027. Project page https://ease-avs.notion.site

Audio-Visual Semantic Segmentation (AVSS) aims to identify, segment, and classify sound-emitting objects in video frames. Previous Transformer-based AVSS approaches largely inherit design principles from image segmentation models. Recent studies show that these image segmentation models contain redundant components that contribute little to the segmentation performance. Following this insight, we propose Encoder-only Audio-Visual Segmentation (EASE). EASE runs at up to 365 FPS, 3x faster than prior State-of-the-Art (SotA) AVS models at comparable accuracy, and trains in under 11 GPU-hours. Furthermore, we achieve SotA AVSS performance across different backbones and input resolutions. Our results demonstrate that AVSS can be both simpler and faster, providing a scalable foundation for future research and real-time applications. Code, model weights, and samples are available at https://ease-avs.notion.site

0.32

Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation

cs.CV ·2026-08-29 ·Tianrui Hui, Shaofei Huang, Qisong Han, Yaxiong Wang, Lechao Cheng +4 more
top venuecode/data similarity 0.61
Abstract · Accepted by ACM MM 2026

Open-Vocabulary Audio-Visual Semantic Segmentation (OV-AVSS) aims to perform pixel-level segmentation of sound-emitting objects from an open set of categories. The previous method relies on a class-agnostic foreground definition, which groups semantically diverse objects into a heterogeneous positive set, causing the model to learn unstable sounding patterns and produce unreliable proposals. To address this, we reformulate the objective to be category-specific and propose a novel Acoustically Grounded Cost Learning (AGCL) framework to transform the static, audio-agnostic visual-text priors into dynamic, audio-grounded cost representations. For intra-category soundingness discovery, we devise Audio-Modulated Cost Generation (AMCG) and Audio-Guided Temporal Aggregation (AGTA) modules to enable both frame-level sounding region highlighting and video-level temporal refinement with a low-intrusive audio injection mechanism. For inter-category distractor discrimination, we introduce a Synergistic Distractor Mining (SDM) strategy, which selectively penalizes acoustically and semantically confusing negative categories to learn more discriminative decision boundaries. Extensive experiments on the AVSBench-OV dataset demonstrate that our method significantly outperforms previous state-of-the-art approaches, particularly on unseen categories. Code is available at https://github.com/spyflying/AGCL.

0.60

MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness

cs.SD eess.AS ·2026-10-08 ·Jinbo Hu, Hang Su, Lichun Fan, Heinrich Dinkel, Gang Li +8 more
code/data similarity 0.61
Abstract

Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.

0.24