CS Paper Radar

From Sound to Occupancy: Privacy-Aware Audio Sensing for Buildings

Myeongwon Chae, Seonghyuk Son

cs.SD cs.LG ·submitted 2026-11-08 19:11 UTC · v1
code/data

Abstract

Occupancy-based controls regulate lighting and HVAC according to whether a space is in use. Passive infrared (PIR) sensors, the most common choice, respond to motion and can report an occupied room as vacant when its occupants remain still. Sound offers an alternative cue, since seated occupants type, turn pages and cough, but microphones raise privacy concerns. We therefore study a two-stage design: an on-device encoder maps each second of audio to a short embedding, and a server-side temporal model turns the stream of embeddings into an occupancy state. To our knowledge, no publicly available dataset pairs office audio with occupancy labels, so we evaluate the two stages separately on public data. The encoder is trained on synthetic scenes that mix empty-room recordings, room impulse responses, office sounds and read speech, and tested on unseen rooms, recordings and speakers for whether a second contains a sound made by a person. On 11 real office recordings (DCASE 2013) it reaches an AUC of 0.87, against 0.76 for log-mel statistics. Attacks on the transmitted embeddings recover at most 1% of the spoken words, against 94% from log-mel frames, but still identify the speaker among 40 in 69 to 77% of the utterances. Noise-floor normalization with adversarial training, an 8-dimensional embedding, or sending only class probabilities lowers this to 38 to 55% without hurting detection. On HPDmobile, which pairs home occupancy labels with an amplitude envelope of the audio, a causal temporal model pretrained with a hierarchical contrastive loss matches hand-crafted features (AUC 0.75) and does better when only one labelled day per home is available. These results make the two-stage design a practical basis for privacy-aware acoustic occupancy sensing, with speaker identity as the one leak still to close.

Comment
19 pages, 4 figures, 7 tables. Code: https://github.com/AaronChae/occupancy-audio-privacy

Closest papers

Nearest neighbours by meaning across the whole index.

Removing Speech, Keeping Activities: Speech Removal for Privacy-Preserving Acoustic Sensing in Assisted Living

cs.SD cs.CRcs.HC ·2026-09-02 ·Pavlos Nicolaou, Christos Efstratiou
similarity 0.67
Abstract · 44 pages, 2 figures, 6 tables. Submitted to Engineering Applications of Artificial Intelligence

Acoustic sensing can monitor the daily activities of older adults without cameras or wearables, yet the possibility of capturing private conversations remains a barrier to its deployment. The engineering application addressed in this paper is privacy-preserving acoustic monitoring of daily activities in assisted living. Our contribution in artificial intelligence is a speech-removal front end: a U-Net that masks the short-time Fourier transform of ambient audio and returns a waveform in which speech is suppressed and the sounds needed for activity recognition are kept. The model is trained only on synthetic mixtures of public sounds and speech, so no speech has to be recorded in the target homes. We evaluate it with group-disjoint cross-validation on environmental and domestic sound datasets mixed with reverberant speech, on speech from unseen corpora and on real home recordings, and measure privacy with three voice activity detectors, Whisper speech recognition and a speaker-verification attack. With speech 5 dB above the background, the macro-averaged F1 score of activity recognition is 0.574 and 0.661 after removal on the two datasets, compared with 0.233 and 0.178 without processing, and higher than that of speech enhancement, speech separation and detector-gated muting at every speech level. Whisper recovers 5.2% of the spoken words, compared with 89.0% without processing and a chance floor of 2.2%. On the real recordings, the speech proportion estimated by a voice activity detector falls from 8.5% to 2.7%. The method reduces, but does not eliminate, spoken content and speaker identity.

0.60

Towards Privacy-Preserving Thermal Human Perception from Dataset to Deployment

cs.CV eess.SP ·2026-10-10 ·Kailai Sun, Yue Cui, Xinwei Wang, Zijie Gao, Baoshen Guo +6 more
code/datahot direction similarity 0.62
Abstract

Indoor human perception is the foundation for smart homes and healthcare. However, RGB cameras raise privacy concerns in private indoor environments, while low-resolution thermal cameras provide limited spatial and semantic information. Large-scale indoor thermal datasets and real-world deployments remain limited. Here, we introduce Indoor RGB-T, a large-scale paired RGB--thermal dataset for indoor human perception. The dataset covers 20 indoor rooms and contains 90,576 80x62 thermal frames, 87,016 temporally matched RGB--thermal pairs, and 115,557 annotated human bounding boxes. We establish benchmarks for robust thermal occupant-state detection with 0.75 mAP and occupancy estimation with an accuracy of 0.847 in unseen rooms. We further evaluate RGB-to-thermal and thermal-to-RGB translation, achieving a temperature MAE of 0.654 C and an R2 of 0.923. Finally, we deploy the thermal perception model on a commercial sleep-monitoring system in the real world. Field testing with 100 participants and deployment of approximately 1,000 devices further demonstrate its practical feasibility. We identify practical challenges that motivate future research directions, including environmental thermal interference, sensing range and multi-person occlusion. The code, trained models and partial dataset are available at https://github.com/kailaisun/RGBT and https://huggingface.co/skl24/RGBT.

0.41

MESH: Multi-receiver Evaluation of Spatial Hearing across Egocentric and Allocentric Views

eess.AS cs.SD ·2026-11-12 ·Wen Wen, Bohan Li, KwanHo Tam, Yu Xi, Jing Peng +2 more
datasetcode/data similarity 0.58
Abstract · 20 pages. Wen Wen and Bohan Li contributed equally. Evaluation code: https://github.com/Wenanzhi/MESH-Benchmark . Dataset: https://huggingfa…

Spatial hearing in multi-receiver environments requires identifying who speaks, where they are, and how they move. Existing benchmarks evaluate spatial prediction, transcription, tracking, and distributed speech processing, but offer limited evaluation of whether speech signal remains associated with the correct physical source across receivers and motion. We introduce MESH, which evaluates this association through egocentric and allocentric views of shared acoustic scenes. The egocentric view uses local recordings and peer metadata, while the allocentric view combines synchronized recordings from multiple receivers in a shared 3D world frame. MESH contains nearly 40,000 questions across 13 static and dynamic localization and spatial transcription tasks. We evaluate eleven audio-model configurations and a language-model agent using specialist tools. The best static distance-estimation and dynamic direction-tracking accuracies are only 21.90% and 41.54%, respectively, while the lowest word error rates across four transcription tasks range from 59.15% to 73.10%. Ablation studies and additional experiments reveal that existing models are not yet able to effectively leverage peer information and struggle to handle multi-speaker and dynamic scenarios. These findings highlight the gap between access to distributed acoustic information and its effective use for spatial speech understanding. We release the dataset and evaluation code to support further research on collaborative and distributed spatial hearing.

0.32

AcousticAgent: A Floorplan-Centric Simulation Platform for Distributed Multi-Room Speech Processing

eess.AS ·2026-10-16 ·Zhiqi Ai, Han Cheng, Shiyi Mu, Yongjin Zhou, Shugong Xu
dataset similarity 0.57
Abstract

Existing acoustic simulation tools either generate room impulse responses (RIRs) efficiently in parameterized rooms or rely on preconstructed three-dimensional scenes, making it difficult to build large numbers of editable residential multi-room environments. We present AcousticAgent, a floorplan-centric acoustic simulation platform that converts structured floorplans into editable, acoustically connected multi-room scenes. AcousticAgent jointly models room geometry and connectivity, furniture, materials, sound sources, and microphone configurations. A scene-oriented Acoustic Materials Database maps semantic elements to acoustic parameters, while a floorplan-aware hybrid geometrical-acoustics solver generates multichannel RIRs and reverberant audio. The platform also supports directional and moving sources, distributed microphones, and VLM-assisted scene construction from floorplan images and text. Using AcousticAgent, we construct FP-RIR, a dataset covering 15,376 residential floorplans with diverse room topologies, propagation conditions, and source--receiver configurations. We validate its acoustic fidelity against real-world measurements and evaluate whole-home source localization, speech enhancement, and far-field speech processing under single-device and distributed microphone settings. The results demonstrate the effectiveness of floorplan-aware modeling for large-scale residential acoustic simulation and the benefits of distributed spatial processing in multi-room environments. AcousticAgent and FP-RIR provide scalable and configurable resources for developing and evaluating residential speech systems.

0.11

Reverberant Speech Conditioning for Material-Aware Acoustic Parameter Mapping

cs.SD ·2026-11-09 ·Yi Wu, Olha Townsend, Qianqian Fang, Magdalena Fuentes, You Zhang
benchmark similarity 0.57
Abstract

Dense maps of room acoustic parameters (T30, EDT, C50, DRR) describe how a room sounds at every position. Scene-wide estimation predicts such maps from a floor plan but requires a reference impulse response, which is hard to obtain. Our system instead receives a floor plan and easily recorded reverberant speech from one talker-microphone pair. We compare four speech-derived conditioning inputs: MFCCs, an embedding from the blind estimator BERP, BERP's scalar reverberation-time estimate, and an impulse response reconstructed by Rec-RIR. Our main benchmark is the MRAS dataset, where material assignments vary at fixed geometry. The reconstructed impulse response performs best among the four inputs, reducing T30 error from 31.5% with the floor plan alone to 19.9%, 83% of the reduction achieved by the reference response. We find that adding speech mainly sets the overall level of the map, with little improvement in predicting how the acoustics vary across the room.

0.16

ARPEGE: Dataset and Methods to Estimate Acoustic and Geometric Parameters of Rooms from Sound

cs.SD eess.AS ·2026-10-30 ·Jean-Daniel Pascal Prieto, Prerak Srivastava, Cédric Foy, Antoine Deleforge
dataset similarity 0.56
Abstract

Can one hear the walls of a room? To foster audio signal processing research around this question we release ARPEGE, a dataset containing hundreds of ambisonic spherical-array room impulse responses recorded in 6 empty rooms, annotated by the relative positions of sources, microphones and all acoustic reflectors. The dataset includes multiple room sizes, surface materials, orientations and distances, as well as recordings of human speakers and handclaps. Two state-of-the-art methods on image source localization and blind room parameter estimation are demonstrated on the data. Initial code for re-simulating the measured room impulse responses is provided, a longer term goal being to digitally twin the dataset.

0.58

Quieter Than the Room: Representation Drift and Task Robustness in Speech Encoders

cs.SD cs.CL ·2026-09-23 ·Vsevolod Kovalev, Pranay Manocha
similarity 0.56
Abstract

Non-speech interference can change a speech representation without causing comparable task loss. We test eight frozen encoders on four tasks, adding non-speech sounds throughout recordings, during speech, or in pauses. Under whole-recording interference, embedding drift tracks task loss across seven sounds, with mean Spearman correlations of 0.81-0.88. Moving the same sound between speech and pauses changes this pattern. At quiet to moderate levels, pause interference produces larger drift, while speech interference usually causes greater loss on intent recognition, speaker verification and speech recognition. Emotion recognition shows a weaker placement effect. Pause interference also changes speech-frame representations beyond the injected region. Even below the estimated recording background, interference can change embeddings as much as repeated speech takes do. Drift helps rank the effects of different sounds, but larger drift does not consistently indicate greater task loss.

0.28

RMS-AQA: A Two-Stage Spatial Audio Question Answering Benchmark for Real-World Domestic Environments

cs.SD eess.AS ·2026-10-01 ·Peihao Chen, Qing Wang, Lichun Fan, Yufeng Hao, Zhifeng Kong +10 more
benchmarkcode/data similarity 0.56
Abstract · Project page: https://github.com/rmsaqachallenge/rmsaqa-code

Embodied assistants in domestic environments must infer what happened, where and when it occurred, and how to respond. To address this, we introduce RMS-AQA, a spatial audio question answering (SAQA) benchmark for real-world domestic environments. The benchmark features a two-stage question-answering (QA) format to comprehensively assess the ability of audio-language models (ALMs) to first ground audible sound events and subsequently perform complex spatio-temporal reasoning based on that grounding. To maximize acoustic realism, our dataset combines authentic real-world first-order Ambisonics (FOA) recordings with high-fidelity synthetic data generated using measured room impulse responses (RIRs). Furthermore, we provide a lightweight spatial plug-in that injects FOA-format data into frozen audio-language backbones. Experimental results reveal that the primary challenges stem from concurrent sources, far distance, and sim-to-real domain gap between RIR-synthesized and authentic recordings.

0.27