CS Paper Radar

SAS: Scene-Annotated AudioSet Strong for Joint Scene-Event Analysis

Michel Olvera, Paraskevas Stamatiadis, Romain Serizel, Slim Essid, Gaël Richard

cs.SD ·submitted 2026-11-06 09:21 UTC · v1

Abstract

Context is central to human auditory perception, as listeners interpret sounds within their acoustic environment to resolve ambiguity. However, large-scale datasets that jointly capture sound events and acoustic scenes remain limited. We address this by augmenting AudioSet Strong with clip-level scene annotations under two taxonomies for joint sceneevent modeling. A corpus-scale analysis shows that scene identity is better captured by events distinctive to a scene than by those that merely occur frequently. Reweighting event co-occurrences toward scene-specific associations produces representations that better align with human-derived taxonomies, while also exposing scenes where the acoustic and semantic views diverge. This geometric divergence predicts classification errors, while supervised validation confirms that the proposed scene annotations provide meaningful complementary information for developing scalable context-aware machine listening systems.

Journal ref
Proceedings of the 11th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2026), Oct 2026, Boston, MA, United States, United States

Closest papers

Nearest neighbours by meaning across the whole index.

Logbook: Extremely Long-form Audio Event Understanding

eess.AS cs.AIcs.CLcs.LG+1 ·2026-10-05 ·Kwanghee Choi, Suwon Shon, Dmitriy Serdyuk, Guitang Lan, Chao-Wei Huang +5 more
benchmarkcode/data similarity 0.75
Abstract · Submitted to ICASSP 2027. Source code available at https://github.com/facebookresearch/logbook

Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we introduce Logbook, a benchmark for hour-scale audio understanding, with recordings ranging from ten minutes to six days. Given a continuous audio recording and an event label vocabulary, a system must predict a gap-free segmentation with an event label and a description per segment. We compare 52 systems, end-to-end and cascaded, and ablate fine-tuning, context length, and reasoning budget. We find the task tractable, though the best systems remain below the human reference. Also, over-segmentation is pervasive, and fine-tuning partially mitigates it. Finally, end-to-end are often better than cascaded systems, but degrades with longer context.

0.25

Hierarchical Interpretable Multitask Audio Representation Learning: An Analysis of Event-Scene Relationships through the Information Bottleneck

eess.AS cs.AI ·2026-10-15 ·Claudia Montero-Ramírez, Alba Martínez-Serrano, Carmen Peláez-Moreno
benchmark similarity 0.73
Abstract

Joint Acoustic Scene Classification (ASC) and Audio Tagging (AT) models traditionally rely on unconstrained multihead architectures. However, this design ignores the natural hierarchical dependency between sounds and acoustic scenes. In this work, we investigate a hierarchical (sequential) conditioning paradigm that predicts scene categories directly from intermediate event probabilities. Through a large-scale empirical benchmark with 66 full training runs across three state-of-the-art backbones (PaSST, PANNs, and BEATs) and different loss-weighting settings (beta in [0.0, 1.0]), we show that hierarchical conditioning achieves competitive classification performance while altering multitask optimization dynamics. Unlike multihead baselines, which sequentially solve the easier ASC task before optimizing AT, the hierarchical architecture enables synchronized dual-task learning for specific loss-weighting settings (beta in [0.6, 0.9]). Furthermore, we propose a multi-perspective interpretability framework that combines an analysis of the information bottleneck, manifold geometry, and Formal Concept Analysis (FCA) on a Term-Frequency Inverse-Document-Frequency (TF-IDF) event representation, providing complementary insights into event-scene relationships. Our findings reveal that hierarchical conditioning systematically yields lower conditional entropy, higher mutual information, and superior latent cluster separability (higher Silhouette scores), establishing a more structured and semantically faithful intermediate feature space.

0.14

Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception

cs.SD cs.AI ·2026-09-14 ·Yanfeng Shi, Yan Song, Junhui Li, Tinggan Huang, Wu Guo +2 more
similarity 0.71
Abstract · Submitted to ICASSP 2027

Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception, particularly in precise event localization. Existing approaches primarily post-train LALMs to predict event boundaries as timestamp tokens. However, this generative formulation lacks explicit correspondence between the timestamp predictions and fine-grained acoustic evidence, limiting the precision and reliability of temporal localization. To address this issue, we augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query. Specifically, the frozen LALM encodes the event query with audio as context, and the grounding model combines these query representations with fine-grained audio features to localize the target event at the frame level. Extensive experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods. Further evaluation shows that the grounding model can provide temporal evidence to support downstream reasoning.

0.17

Evaluating Inference-time Algorithms for Semantic Sound Scene Segmentation

eess.AS cs.SD ·2026-10-02 ·Sripathi Sridhar, Gordon Wichern, Yoshiki Masuyama, Mark Cartwright, Jonathan Le Roux
published similarity 0.70
Abstract · Accepted to DCASE 2026

Spatial semantic segmentation of sound scenes (S5) involves the detection and extraction of target sound events in audio files. However, in typical detection-extraction pipelines, errors in the detection module propagate to the extraction module, degrading overall performance. In this work, we develop a multichannel detection-extraction model and evaluate inference-time algorithms to improve the class detection and target sound extraction in a multi-stage setup. We evaluate two fitness functions: binary cross entropy and mixture consistency; two relabeling strategies: naive and conditional confusion matrix relabeling; and three source estimate evaluation models: a multi-label classifier, a fine-tuned single-source classifier, and an off-the-shelf audio judge. We thoroughly evaluate the impact of these design choices on the DCASE 2025 Task 4 dataset, and validate entropy-based gating of inference-time updates on the held-out evaluation set, laying the groundwork for inference-time updates of S5 systems in real-world scenarios.

0.28

Hear Every Sound Object: Rescuing Overlooked Sources in Auditory Scene Analysis

eess.AS ·2026-10-22 ·Younghoo Kwon, Yongjoon Lee, Jung-Woo Choi
similarity 0.70
Abstract · 5 pages, 4 figures, Submitted to ICASSP 2027

Auditory scene analysis (ASA) models extract information about individual sound sources from a complex auditory mixture. Recent object-based ASA models process the mixture through a sequence of modules and split it into source-wise features (i.e., object features) only at the last layer. However, substantial disparities in performance across sources limit overall performance. Our analysis using centered kernel alignment (CKA) reveals that CKA scores correlate strongly with per-source performance, with their disparities widening from intermediate to deeper layers. These findings motivate the early split and exchange of object features to address source-wise performance disparities before they widen in deeper layers. To this end, we propose PARASITE, a plug-in refinement architecture that splits object features at earlier layers and progressively refines them through sequential layers. Each layer integrates object features separated at different depths using Object-wise Gated Fusion (OGF) and then refines them using Windowed Cross-Object Attention (WCOA). We introduce the DeepASA++ model, which combines PARASITE with an ASA backbone and delivers state-of-the-art results on both ASA2 and STARSS23. DeepASA++ achieves a SELD score of 0.240 on STARSS23 and surpasses prior multi-stage inference architectures, including chain-of-inference.

0.12

CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

cs.SD cs.LG ·2026-10-07 ·Marcel Gibier, Thomas Thebaud, Olivier Boëffard, Jean-François Bonastre
benchmark similarity 0.69
Abstract · Submitted to ICASSP 2027

Automatic audio scene description turns a recording into a text account of a situation. One difficulty is deciding which elements of the audio should be kept, since a description cannot include them all. Annotators disagree about this, making a ground truth hard to obtain. In this work, we first define the ground truth, then generate the data. We focus on audio events and define sound salience with a simple rule: a sound is salient when a speaker audibly reacts to it. For scale and variety, a controlled set of scenarios fixes the ground truth, and a language model writes the dialogues. The resulting corpus, CARES, contains 10,000 two-speaker scenes. We then benchmark six audio-language models on three tasks: identifying the scene, tagging the sounds present, and classifying reactions. We show that these models hear the sounds but miss how the speakers react to them.

0.13

AVPS-360: A Spatial Audio-Visual Benchmark for Instance Segmentation and Tracking of Sound-Producing Persons

cs.CV ·2026-10-29 ·Yifu Sun, Parthasaarathy Sudarsanam, Tuomas Virtanen, Archontis Politis
benchmark similarity 0.69
Abstract

Audio-visual instance segmentation (AVIS) segments and tracks individual sound-producing objects in videos. Existing benchmarks predominantly use monaural audio, which lacks inter-channel cues for spatially distinguishing concurrent sound sources. We introduce AVPS-360 (Audio-Visual Person Segmentation 360), a spatial audio-visual benchmark bridging AVIS and sound event localisation and detection (SELD). Built upon STARSS23, it pairs first-order Ambisonics audio with synchronised 360° video across 230 recordings totalling 10.5 hours. Recordings average 164.0 seconds. The dataset captures intermittent sound production in real-world interactions dominated by speech, alongside laughter, clapping, and footsteps. It provides 31,678 manually reviewed instance masks linked to source identities and sound-event labels. We also present a baseline that integrates spatial audio with visual features for frame-level segmentation and identity association across frames. Experiments on AVPS-360 show that existing AVIS methods struggle with detection and identity association in long recordings with intermittent sound production, while our baseline substantially improves performance in both aspects.

0.13

Probing Large Audio-Language Models for Compositional Understanding of Sounding Actions

cs.SD ·2026-09-28 ·Michel Olvera, Paraskevas Stamatiadis, Changhong Wang, Ga{ë}l Richard
top venuebenchmarkcode/data similarity 0.69
Abstract

Large audio-language models (LALMs) excel at understanding and reasoning tasks over atomic sound events, yet their ability to infer higher-level human activities from such fine-grained events remains largely unexamined. Everyday human actions and activities, such as setting a table, cleaning the house, or preparing a breakfast emerge compositionally from temporally distributed sound events, requiring abstraction beyond the event-centric granularity that dominates current training and evaluation paradigms. Our benchmark evaluates a wide set of LALMs under a principled framework that tests how language-based reasoning, grounded in acoustic perception, structures sound abstractions into higher-level understanding. By systematically varying exemplar typicality and distractor similarity, our evaluation exposes \added{that current models do not reliably perform compositional inference from atomic acoustic events to higher-level human activities solely from audio.} All data, taxonomies, and evaluation scripts are publicly available on our companion website: https://alm-sounding-actions.onrender.com/

0.51