CS Paper Radar

Spatial DJ: Spatial Audio Control for Arbitrary Loudspeaker Layouts

Anastasia Geleta, Margarita Geleta

cs.SD cs.HCeess.AS ·submitted 2026-10-31 05:40 UTC · v1

Abstract

Object-based spatial audio promises render-anywhere playback, yet deploying it in a new venue still requires two tasks that are rarely addressed together: acquiring the physical loudspeaker geometry and enabling creators manipulate spatial audio while listening through the target loudspeaker system rather than a headphone proxy. We present Spatial DJ, a system for on-site loudspeaker layout acquisition and real-time spatial audio rendering and control, using head-worn AR in a coordinate frame aligned to the physical loudspeakers. In an 18-loudspeaker room, we evaluate calibration accuracy, repeatability, and control latency, and conduct an 18-participant listener study comparing diffuse and orbiting point-source rendering configurations applied to vocal and instrumental layers. Orbiting point-source conditions received higher ratings of perceived immersion, energy, and novelty than the diffuse baseline. Beyond this controlled setting, four representative deployments illustrate reuse of the same system across heterogeneous loudspeaker venues.

Comment
20 pages, 6 figures

Closest papers

Nearest neighbours by meaning across the whole index.

PLACE: Positional Latent Adaptation via Conditioned Embeddings for Binaural Audio Generation

cs.SD cs.MMeess.AS ·2026-09-30 ·Tiernon Riesenmy, You Zhang, Gautam Bhattacharya, Andrea Fanelli
similarity 0.65
Abstract · Submitted to ICASSP 2027

We present PLACE, a method that extends the pretrained any-to-audio model AudioX for binaural generation from arbitrary combinations of text, video, and optional audio prompts. PLACE augments video conditioning with Perception Encoder Core features, aligns text and video representations to derive spatial cues, and applies a conditioning-dependent low-rank transformation to the generated latent. The adapter is supervised via decoded-audio interaural level and time difference objectives. Trained on MRSAudio, PLACE improves most metrics over ViSAGe on FAIR-Play and achieves an improved SpatialCLAP score over SpatialSonic on the BEWO-1M Single Static test split. Listener evaluations favor PLACE for video-to-audio and out-of-distribution text-to-audio generation, demonstrating flexible multimodal control and improved spatial consistency.

0.28

Audible World Models: Spatially Aware Sound Generation for 3D Worlds

cs.CV cs.SD ·2026-09-29 ·Duowen Chen, Jinjin He, Gouthaman KV, Sandeep Bangalore Venkatesh, Bo Zhu
top venue similarity 0.63
Abstract · Accepted at the 40th Conference on Neural Information Processing Systems (NeurIPS 2026)

Text- and image-conditioned world generators can create visually rich 3D environments, yet these worlds often remain silent or rely on soundtracks synthesized solely from text or rendered video. Although such audio can convey what should be heard, it lacks an explicit representation of where sound sources are located and how their perceived sound should vary with listener movement. We introduce Audible World Models, a training-free framework that incorporates sound into the generated world state. Starting from a text prompt, our system constructs a panoramic 3D proxy, separates it into semantic layers, identifies sound-producing foreground objects and ambient background regions, and synthesizes dry audio for each sound label. It then anchors these sources to reconstructed geometry and renders listener-dependent spatial audio using geometric acoustic propagation. By explicitly linking semantics, geometry, and sound propagation, the framework maintains persistent source locations while adapting the rendered audio to changes in listener viewpoint and motion. Experiments across 80 generated scenes demonstrate substantial gains in spatial consistency over text-, video-, and panorama-conditioned baselines, while preserving competitive semantic alignment. VLM-based assessments and human evaluations further indicate that our soundtracks are preferred for their audio-visual consistency, spatial plausibility, and motion-dependent behavior.

0.57

MESH: Multi-receiver Evaluation of Spatial Hearing across Egocentric and Allocentric Views

eess.AS cs.SD ·2026-11-12 ·Wen Wen, Bohan Li, KwanHo Tam, Yu Xi, Jing Peng +2 more
datasetcode/data similarity 0.60
Abstract · 20 pages. Wen Wen and Bohan Li contributed equally. Evaluation code: https://github.com/Wenanzhi/MESH-Benchmark . Dataset: https://huggingfa…

Spatial hearing in multi-receiver environments requires identifying who speaks, where they are, and how they move. Existing benchmarks evaluate spatial prediction, transcription, tracking, and distributed speech processing, but offer limited evaluation of whether speech signal remains associated with the correct physical source across receivers and motion. We introduce MESH, which evaluates this association through egocentric and allocentric views of shared acoustic scenes. The egocentric view uses local recordings and peer metadata, while the allocentric view combines synchronized recordings from multiple receivers in a shared 3D world frame. MESH contains nearly 40,000 questions across 13 static and dynamic localization and spatial transcription tasks. We evaluate eleven audio-model configurations and a language-model agent using specialist tools. The best static distance-estimation and dynamic direction-tracking accuracies are only 21.90% and 41.54%, respectively, while the lowest word error rates across four transcription tasks range from 59.15% to 73.10%. Ablation studies and additional experiments reveal that existing models are not yet able to effectively leverage peer information and struggle to handle multi-speaker and dynamic scenarios. These findings highlight the gap between access to distributed acoustic information and its effective use for spatial speech understanding. We release the dataset and evaluation code to support further research on collaborative and distributed spatial hearing.

0.32

SPHERE: A Multi-Attribute No-Reference metric for Binaural Speech Quality And Spatial Fidelity

cs.SD eess.AS ·2026-10-30 ·Bala Manikya Prasad Puram, Chandan K A Reddy, Artem Dementyev
similarity 0.60
Abstract

In this paper, we introduce SPHERE (Spatial and Perceptual Human Evaluation and Rating Estimator), a multidimensional no-reference perceptual evaluation framework for binaural speech. SPHERE defines nine distinct perceptual and spatial attributes that characterize the immersive listening experience. We detail the crowdsourcing protocol and evaluation template used to collect ground-truth human ratings. To demonstrate the tractability of automating this framework, we evaluate six baseline configurations across single-branch and Siamese architectures utilizing Mel-spectrograms, complex STFTs, and pretrained spatial embeddings (PhaseCoder). Experimental results demonstrate that these baseline estimators achieve strong correlation with human subjective ratings, proving the feasibility of automated multidimensional spatial audio quality assessment.

0.23

Audiovisual Disparities in VR: Impact on Spatial Perception

cs.GR ·2026-11-06 ·Edurne Bernal-Berdun, Mateo Vallejo, Qi Sun, Ana Serrano, Diego Gutierrez
top venue similarity 0.59
Abstract · 2025 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)

Virtual reality (VR) experiences often leverage rich and spatialized multimodal environments to increase immersion and engagement. This demands a consistent spatial perception of audiovisual stimuli, since perceived discrepancies can disrupt the sense of presence. In this work, we investigate the consequences of two types of spatial audiovisual disparities: true disparity, where there is a measurable spatial offset between auditory and visual cues, and perceptual disparity, where users report misalignment despite cues being colocated. Unlike most previous studies that employed controlled but simplified experimental setups, our research focuses on complex, realistic VR environments, allowing us to assess the actual implications for VR content design. Our experiments indicate that users are highly sensitive to true audiovisual disparities in controlled environments, detecting even minor misalignments. However, when engaged in additional tasks within realistic settings, their ability to notice such discrepancies diminishes significantly. We also observed that previously found perceptual disparities persist in complex audiovisual environments. However, we identify self-initiated head rotations as a key factor; its absence prevents the effect entirely. We hope our findings offer practical insights for designing more immersive and perceptually coherent VR experiences.

0.58

Numerical and perceptual validity of synthetic Head-Related Transfer Functions at scale

eess.AS ·2026-08-17 ·Katarina C. Poole, Lorenzo Picinali
unusual similarity 0.59
Abstract · Version 2 clarifies in the pdf this is a preprint submitted to JASA and awaiting review

Individually measuring head-related transfer functions (HRTFs) at scale remains a central challenge for personalised spatial audio, motivating growing interest in synthetic HRTFs. We evaluated the numerical, computational, and behavioural validity of synthetic HRTFs, generated through the boundary element method simulation using Mesh2HRTF, against measured and KEMAR HRTFs using the Extended SONICOM dataset. Across 200 subjects, synthetic HRTFs deviated less from measured than KEMAR in interaural time and level differences, but residual errors, together with elevated spectral distortion, concentrated at low, rear elevations. This is consistent with the omission of torso geometry from the synthesis pipeline. Two computational models revealed a corresponding pattern of predicted localisation errors, with synthetic HRTFs positioned between measured and KEMAR. In a virtual reality localisation task (N = 20), synthetic HRTFs matched measured on every polar metric, while KEMAR was significantly worse. However, behavioural error clustered around the front-back midline regardless of condition, not at the low elevations implicated numerically or by the models. A separate spatial release from masking task (N = 18) showed no effect of HRTF type. Together, these results indicate that high-resolution synthetic HRTFs preserve behavioural localisation performance, despite discrepancies between the numerical/model-predicted bias and the spatial pattern of behavioural error.

0.30

Reproduction of Simulated Acoustics Scenes Using Acoustics Room Transformer for speech intelligibility tests in Clinics

eess.AS physics.med-ph ·2026-11-13 ·Ali Fallah, Kirsten C Wagener, Pascal Pflueger, Steven van de Par
similarity 0.58
Abstract · This paper is submitted to Acta Acustica and is currently under review

An Acoustic Room Transformation (ART) method is proposed for clinical applications to render simulated acoustic environments using six loudspeakers in a reverberant room. In typical clinical settings, testing rooms are not acoustically treated and only a limited number of loudspeakers are available. The ART approach enables control of the playback-room reverberation, allowing speech intelligibility tests to be performed under realistic and perceptually compensated conditions. The method focuses on the two most common Speech Reception Threshold (SRT) configurations: S0N90 and S0N90. One front and one side loudspeaker reproduce the direct sound components, while the remaining four loudspeakers (two in front and two behind the listener) generate the reverberant sound field. The direct sound is optimized in terms of energy, and the reverberant field is adjusted based on both energy and interaural coherence. An additional direct-sound gain is applied to match the clarity index (C50) of the reproduced signals to that of the reference target. Results from speech intelligibility tests demonstrate that, with ART processing, the reproduced scenarios achieve SRT values very close to those of the reference signal in both S0N0 and S0N90 conditions and therefore allow for comparable speech test results independent of the particular test room acoustics.

0.27

CSAVocoder: A Causal Spatial Audio Vocoder Towards Real-Time Spatial Audio Generation

eess.AS ·2026-08-26 ·Zhiyuan Zhu, Han Wang, Wenxiang Guo, Yu Zhang, Changhao Pan +2 more
top venue similarity 0.58
Abstract

Spatial audio vocoders are able to convert mel-spectrograms produced by generative models into spatial audio waveforms. Most neural vocoders are designed for monaural audio, and direct extensions to spatial audio can degrade spatial quality by ignoring inter-channel cues. We present CSAVocoder, a causal GAN-based spatial audio vocoder that jointly optimizes waveform fidelity and spatial rendering. Our framework introduces a Spatial Adaptor that fuses multi-channel mel-spectrograms with dynamic source-listener pose information, together with a spatial consistency discriminator that supervises inter-channel cues. To meet real-time requirements, we design a strictly causal, stateful generator that supports efficient streaming inference with constant memory overhead. Experiments on large-scale spatial audio datasets show that CSAVocoder improves spatial fidelity at competitive audio quality and real-time performance.

0.61