CS Paper Radar

Geometry-Informed Distributed Acoustic Scene Understanding

Yiyuan Yang, Shitong Xu, Niki Trigoni, Andrew Markham

cs.SD ·submitted 2026-09-07 22:18 UTC · v1
top venue

Abstract

Acoustic scene understanding in multi-room environments is a difficult task. Most existing systems use a single centralized microphone array, and they often fail because walls and doors block sound signals. To address this challenge, we propose a geometry-informed distributed acoustic scene understanding framework. Our system leverages distributed microphones and uses an audio spectrogram transformer and a topology-aware graph neural network to fuse spatio-temporal acoustic features. Then, these features are decoded into discrete semantic triplets. Finally, a frozen large language model combines these symbolic observations with the environmental geometry. This allows the system to perform spatial understanding, infer plausible missing transitions, and generate a physically consistent narrative of the scene. Experiments on a custom multi-room simulator demonstrate that our framework outperforms centralized baselines and improves spatial consistency under simulated occlusion.

Comment
Accepted by Interspeech 2026 Conference

Closest papers

Nearest neighbours by meaning across the whole index.

MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness

cs.SD eess.AS ·2026-10-08 ·Jinbo Hu, Hang Su, Lichun Fan, Heinrich Dinkel, Gang Li +8 more
code/data similarity 0.73
Abstract

Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.

0.24

AcousticAgent: A Floorplan-Centric Simulation Platform for Distributed Multi-Room Speech Processing

eess.AS ·2026-10-16 ·Zhiqi Ai, Han Cheng, Shiyi Mu, Yongjin Zhou, Shugong Xu
dataset similarity 0.72
Abstract

Existing acoustic simulation tools either generate room impulse responses (RIRs) efficiently in parameterized rooms or rely on preconstructed three-dimensional scenes, making it difficult to build large numbers of editable residential multi-room environments. We present AcousticAgent, a floorplan-centric acoustic simulation platform that converts structured floorplans into editable, acoustically connected multi-room scenes. AcousticAgent jointly models room geometry and connectivity, furniture, materials, sound sources, and microphone configurations. A scene-oriented Acoustic Materials Database maps semantic elements to acoustic parameters, while a floorplan-aware hybrid geometrical-acoustics solver generates multichannel RIRs and reverberant audio. The platform also supports directional and moving sources, distributed microphones, and VLM-assisted scene construction from floorplan images and text. Using AcousticAgent, we construct FP-RIR, a dataset covering 15,376 residential floorplans with diverse room topologies, propagation conditions, and source--receiver configurations. We validate its acoustic fidelity against real-world measurements and evaluate whole-home source localization, speech enhancement, and far-field speech processing under single-device and distributed microphone settings. The results demonstrate the effectiveness of floorplan-aware modeling for large-scale residential acoustic simulation and the benefits of distributed spatial processing in multi-room environments. AcousticAgent and FP-RIR provide scalable and configurable resources for developing and evaluating residential speech systems.

0.11

MESH: Multi-receiver Evaluation of Spatial Hearing across Egocentric and Allocentric Views

eess.AS cs.SD ·2026-11-12 ·Wen Wen, Bohan Li, KwanHo Tam, Yu Xi, Jing Peng +2 more
datasetcode/data similarity 0.68
Abstract · 20 pages. Wen Wen and Bohan Li contributed equally. Evaluation code: https://github.com/Wenanzhi/MESH-Benchmark . Dataset: https://huggingfa…

Spatial hearing in multi-receiver environments requires identifying who speaks, where they are, and how they move. Existing benchmarks evaluate spatial prediction, transcription, tracking, and distributed speech processing, but offer limited evaluation of whether speech signal remains associated with the correct physical source across receivers and motion. We introduce MESH, which evaluates this association through egocentric and allocentric views of shared acoustic scenes. The egocentric view uses local recordings and peer metadata, while the allocentric view combines synchronized recordings from multiple receivers in a shared 3D world frame. MESH contains nearly 40,000 questions across 13 static and dynamic localization and spatial transcription tasks. We evaluate eleven audio-model configurations and a language-model agent using specialist tools. The best static distance-estimation and dynamic direction-tracking accuracies are only 21.90% and 41.54%, respectively, while the lowest word error rates across four transcription tasks range from 59.15% to 73.10%. Ablation studies and additional experiments reveal that existing models are not yet able to effectively leverage peer information and struggle to handle multi-speaker and dynamic scenarios. These findings highlight the gap between access to distributed acoustic information and its effective use for spatial speech understanding. We release the dataset and evaluation code to support further research on collaborative and distributed spatial hearing.

0.32

FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs

cs.CV cs.LG ·2026-10-08 ·Kyeong-Rae Kim, Sungnyun Kim, Tae-Hyun Oh
benchmarkcode/data similarity 0.68
Abstract · Project page: https://byulharang.github.io/FloorSAV/

While 3D spatial reasoning in dynamic egocentric environments is crucial for embodied intelligence, audio-visual large language models (AV-LLMs) lack explicit mechanisms to process and internalize global geometry directly from raw sensory streams. Existing approaches either require costly fine-tuning or underutilize the model's cross-modal reasoning capacities. In this paper, we propose FloorSAV, a novel framework that explicitly grounds spatial audio-visual context by rendering a dynamic 2D floormap. By integrating 3D point clouds, camera trajectories, spatial audio cues, and semantically grounded object landmarks, we inject this floormap into the AV-LLM as a synchronized stream with an egocentric video. AV-LLMs utilize their multi-modal capabilities to jointly reason over visual, auditory, and geometric cues in a single inference with floormap interpretation guidance. We further introduce SAVED-Bench (Spatial Audio-Visual Egocentric Benchmark with Dynamic Agents), constructing essential tasks of spatial capability in real-world scenarios: dynamic relativity, regional, and path reasoning QAs. FloorSAV improves AV-LLMs' spatial reasoning on various tasks from both SAVED-Bench and SAVVY-Bench. Studies with ground-truth floormaps demonstrate the substantial potential of FloorSAV with accurate spatial information.

0.19

Audible World Models: Spatially Aware Sound Generation for 3D Worlds

cs.CV cs.SD ·2026-09-29 ·Duowen Chen, Jinjin He, Gouthaman KV, Sandeep Bangalore Venkatesh, Bo Zhu
top venue similarity 0.67
Abstract · Accepted at the 40th Conference on Neural Information Processing Systems (NeurIPS 2026)

Text- and image-conditioned world generators can create visually rich 3D environments, yet these worlds often remain silent or rely on soundtracks synthesized solely from text or rendered video. Although such audio can convey what should be heard, it lacks an explicit representation of where sound sources are located and how their perceived sound should vary with listener movement. We introduce Audible World Models, a training-free framework that incorporates sound into the generated world state. Starting from a text prompt, our system constructs a panoramic 3D proxy, separates it into semantic layers, identifies sound-producing foreground objects and ambient background regions, and synthesizes dry audio for each sound label. It then anchors these sources to reconstructed geometry and renders listener-dependent spatial audio using geometric acoustic propagation. By explicitly linking semantics, geometry, and sound propagation, the framework maintains persistent source locations while adapting the rendered audio to changes in listener viewpoint and motion. Experiments across 80 generated scenes demonstrate substantial gains in spatial consistency over text-, video-, and panorama-conditioned baselines, while preserving competitive semantic alignment. VLM-based assessments and human evaluations further indicate that our soundtracks are preferred for their audio-visual consistency, spatial plausibility, and motion-dependent behavior.

0.57

SAS: Scene-Annotated AudioSet Strong for Joint Scene-Event Analysis

cs.SD ·2026-11-06 ·Michel Olvera, Paraskevas Stamatiadis, Romain Serizel, Slim Essid, Gaël Richard
similarity 0.66
Abstract

Context is central to human auditory perception, as listeners interpret sounds within their acoustic environment to resolve ambiguity. However, large-scale datasets that jointly capture sound events and acoustic scenes remain limited. We address this by augmenting AudioSet Strong with clip-level scene annotations under two taxonomies for joint sceneevent modeling. A corpus-scale analysis shows that scene identity is better captured by events distinctive to a scene than by those that merely occur frequently. Reweighting event co-occurrences toward scene-specific associations produces representations that better align with human-derived taxonomies, while also exposing scenes where the acoustic and semantic views diverge. This geometric divergence predicts classification errors, while supervised validation confirms that the proposed scene annotations provide meaningful complementary information for developing scalable context-aware machine listening systems.

0.18

BinauralVAE: Spatial Audio Reconstruction For World Models

cs.SD cs.LG ·2026-09-06 ·Luis Vitor Zerkowski, Luiz Velho
code/data similarity 0.65
Abstract · 17 pages, 7 figures

Embodied artificial intelligence has historically very much relied on visual perception, leading to a proliferation of multiple vision-centric world models. However, this reliance fails to capture spatial understanding in its entirety and can even present vulnerabilities in environments with visual occlusions, low-light conditions, or blackouts-scenarios, where acoustic information becomes a critical alternative for spatial awareness and navigation. Despite its potential, research into realistic spatial audio and particularly the development of audio-centric world models remains sparse. In this technical report, we introduce BinauralVAE: a flexible, open-source pipeline (https://github.com/Luizerko/BinauralVAE) that explores multiple models for spatialized audio reconstruction, progressing from fundamental baselines to advanced, mathematically grounded architectures. Our approach evaluates various Variational Autoencoder architectures -- including complex-valued variants -- to learn robust latent representations of binaural signals. Developed alongside AudioWorldSim, our methodology leverages realistic acoustic data captured as a simulated robot navigates an environment. This pipeline establishes a foundation for state representation in a future audio-based world model, designed to map the direct causal connection between navigational actions and their resulting acoustic consequences, and helping to enable sound as an essential complementary modality for spatial knowledge acquisition.

0.27

SAIL: Spatial Audio Intelligence with Large Language Models via Disentangled Acoustic-Spatial Encoding and Dual-Stream Q-Former

cs.SD cs.AIeess.AS ·2026-09-28 ·Zhengding Luo, Jinyang Wu, Haozhe Ma, Yanghao Zhou, Woon-Seng Gan +1 more
similarity 0.65
Abstract

Spatial audio large language models (LLMs) enable embodied agents, wearable assistants, and immersive systems to recognize sound events, localize sources, and reason about their spatial relationships. However, existing spatial audio LLMs often rely on early fusion of acoustic and spatial features and source-agnostic token representations. These designs make it difficult to preserve the correspondence between individual sound events and their spatial attributes, particularly in multi-source scenes. To address this limitation, we propose SAIL, a Spatial Audio Intelligence framework with LLMs that preserves acoustic-spatial structure and source-level correspondence from audio encoding to LLM alignment. SAIL introduces a Disentangled Spatial Audio Transformer that represents Mel-spectrogram and interaural phase difference features as separate acoustic and spatial streams. Source-discriminative task queries further learn event, direction, and distance information for each source. A Dual-Stream Q-Former then aligns the two streams with the LLM using acoustic and spatial queries organized by source slots. Compared with the early-fusion baseline, SAIL achieves consistent improvements in dual-source sound event detection, direction and distance estimation, and spatial reasoning. These results demonstrate the importance of structured, source-discriminative audio representations for multi-source spatial understanding and reasoning.

0.45