CS Paper Radar

Hearing's Blind Spots: Toward Perceptual Compute Allocation in Neural Text-to-Speech

Aayush Gupta

cs.SD eess.AS ·submitted 2026-10-06 06:41 UTC · v1
code/data

Abstract

A speech synthesizer produces a detailed waveform, but a listener need not hear every detail. Strong sounds can mask weaker neighbors, creating context-dependent "blind spots" in hearing. Could neural text-to-speech (TTS) spend less computation on those details? We investigate a first step: how the placement of spectral degradation affects completed speech from Piper and Kokoro. At matched time-frequency bin-removal budgets, a simple low-energy-first policy achieves higher mean wideband PESQ than random deletion and the tested psychoacoustic masking criteria. At 50% removal in the spreading-matrix comparison, energy, masking, and random policies score 4.46, 3.60, and 1.19 on Piper, and 3.42, 2.10, and 1.22 on Kokoro. Spectrograms and masking maps expose where these perturbations fall and why a model's below-threshold regions cannot automatically be treated as inaudible. Calibration diagnostics show substantial threshold sensitivity, while a small single-listener pilot establishes no transparent removal budget. CPU profiles identify waveform decoding as the main measured kernel cost, but a tested dynamic-INT8 Kokoro decoder runs slower than FP32 on both CPUs. The results establish energy ranking as a strong oracle baseline and motivate testing perceptual allocation inside a decoder. They do not yet demonstrate saved computation: the decisions here use already synthesized audio, and quality metrics are evaluated at 16 kHz. We release scripts, results, audio examples, and a listening-study instrument.

Comment
9 figures. Code, corpus, audio examples, aggregate results, and listening-study instrument: https://github.com/Deployment-inc/deep-perceptual-compute-allocation

Closest papers

Nearest neighbours by meaning across the whole index.

Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection

cs.AI ·2026-09-03 ·Maryam Abbasihafshejani, Murtuza Jadliwala
similarity 0.59
Abstract

Whisper is a widely used foundation model for automatic speech recognition (ASR), but its generative decoder can produce fluent hallucinated transcripts for inputs containing little or no speech. We propose a training-free, inference-time method to reduce these hallucinations using low-rank projection of decoder activations. A compact hallucination-associated subspace is estimated from non-speech calibration data, and decoder hidden states are projected away from this subspace during inference. We evaluate two variants: always-on, which applies projection to all inputs, and gated, which applies it only when Whisper predicts that an input is likely non-speech. Across non-speech benchmarks, always-on projection reduces average hallucination rate (HR) from 31.31% to 2.44%, a 92.21% relative reduction, while gated projection reduces HR to 3.74%, an 88.05% relative reduction, with lower false rejection of genuine speech. On LibriSpeech, gated projection increases absolute word error rate (WER) by 0.33-4.39 percentage points and yields false-rejection rates (FRR) of 0.41--9.97% across model and split settings. These results show that low-rank activation projection can substantially suppress Whisper hallucinations without retraining, while providing a controllable trade-off between hallucination suppression and speech recognition performance.

0.24

Silence-the-Mimic: Accelerating Imperceptible Perturbation Generation Against Voice Cloning

eess.AS ·2026-09-30 ·Runqiu Xu
published similarity 0.58
Abstract · Accepted at IEEE SLT 2026

Deep neural network-based Voice Conversion (VC) and Text-to-Speech (TTS) models have rapidly advanced, enabling realistic voice cloning with minimal input data. Such capabilities raise serious concerns over unauthorized cloning of speaker identities and the associated privacy and security risks. Current imperceptible adversarial protection methods rely on quality control losses that are highly sensitive to hyperparameter tuning and computationally expensive due to lengthy optimization. To address these limitations, we propose a fast protection method that generates perceptually constrained perturbations in the frequency domain under a psychoacoustic masking-based constraint. Our approach strictly enforces perceptibility bounds during adversarial training, eliminating the need for iterative quality balancing and significantly reducing computational cost. Experiments on multiple state-of-the-art VC and TTS models show that STM achieves competitive or superior protection performance with substantially better perceptual quality and up to $45.3\times$ speedup over existing white-box baselines. These results demonstrate the effectiveness of frequency-domain perturbations with perceptual constraints as a practical paradigm for protecting against voice cloning.

0.27

Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representations

cs.SD cs.AI ·2026-09-03 ·Yoto Fujita, Simon Leglaive, Laurent Girin
code/data similarity 0.58
Abstract · Presented at the 19th International Workshop on Acoustic Signal Enhancement (IWAENC), Sep 2026, Cremona, Italy

Most previous work on speech enhancement (SE) based on masked generative modeling relied on discrete token representations of audio signals, obtained using neural audio codecs (NACs). However, a recent study has shown that continuous latent representations of NACs can be advantageous for SE in terms of speech quality and intelligibility. In this work, we propose masked autoregressive SE (MARSE), a method for SE based on iterative decoding of masked clean speech frames using continuous NAC representations of speech. In particular, we investigate a set of different decoding policies, ceteris paribus, that is, using the same DNN (a Conformer model), the same NAC (the DAC codec) and the same training setup. The results show that MARSE enables a flexible trade-off between SE performance and computational cost. Audio examples and code are available online.

0.35

AESSI: An Around-Ear Silent Speech Interface for Cross-Day Online Reuse without Test-Day Calibration

cs.HC ·2026-09-18 ·Xiran Xu, Mochu Dong, Yujie Yan, Chenxi Wang, Yu Jiao +1 more
similarity 0.58
Abstract

Silent speech interfaces (SSIs) enable private communication without audible speech and may support people with post-stroke dysarthria. Everyday reuse requires articulation-related representations that generalize across days despite sensor repositioning and physiological changes. We present AESSI, an around-ear SSI using masked-context representation pretraining (MCRP): a student predicts teacher representations from masked time-frequency inputs to encourage robustness to recording variability. We collected 44 electrophysiological recordings from 24 participants for 25 everyday Mandarin sentences. With six participants' complete final recordings held out, AESSI achieved 92.24 percent mean accuracy without test-day calibration. AESSI exceeded the best adapted baseline by 50.57 percentage points. At least 21 days after each participant's last recording, five participants each completed 50 independently randomized online tests without calibration, achieving 98.0 percent overall accuracy. Median preprocessing and inference time in CPU replay was 31.70 ms. These results demonstrate end-to-end system operation and support online reuse without test-day calibration. Demo is included with the paper.

0.30

Mask-Based Speech Enhancement for Spatial Audio: A Comparison of Ambisonics, Beamforming, and Microphone Channels

eess.AS ·2026-09-16 ·Sheli Hendel, Boaz Rafaely, Dorothea Kolossa
similarity 0.58
Abstract

Mask-based speech enhancement is widely used for suppressing noise and interference, but its performance in spatial audio algorithms with multichannel output has not been studied extensively. In such settings, speech enhancement must improve speech quality while preserving spatial cues that are essential for localization, spatial awareness, and spatial release from masking. In this work, we systematically compare time frequency masking applied to three signal representations: microphone signals, beamformer outputs, and Ambisonics signals. Performance is evaluated in terms of speech quality, intelligibility, binaural cue preservation, and reverberation preservation. Results reveal a clear trade-off between enhancement and spatial fidelity: beamformer-domain masking achieves the highest speech enhancement scores, while Ambisonics-domain masking better preserves the spatial attributes of the residual interference. All methods preserve the target's localization cues.

0.15

Perceptual Quality Loss or Loss of Perceptual Quality?

eess.AS cs.LG ·2026-09-28 ·Danilo de Oliveira, Tal Peer, Maurício do V. M. da Costa, Timo Gerkmann
similarity 0.57
Abstract · Submitted to ICASSP 27

Contemporary deep speech enhancement (SE) models are often trained with specific auxiliary terms in the loss function as a way to improve their performance in terms of perceptual metrics. Nevertheless, a higher score on a perceptual metric does not necessarily correlate with an improved listening experience. Through objective and subjective experiments, we assess the performance of SE models trained with two different types of auxiliary PESQ loss terms. The numerical evaluation on a suite of standard metrics suggests that, while models optimized for PESQ naturally obtain higher PESQ scores in the test set, for most other metrics the scores do not significantly change. In some cases, the PESQ loss even results in worse PESQ scores on mismatched data. A formal listening experiment reveals that the models without a PESQ loss were generally preferred over models that include it, across all settings. Finally, we analyze the relative importance of PESQ in the composite metrics CSIG, CBAK and COVL, and find that PESQ dominates all of them. Our study highlights the perils of over-reliance on PESQ and stresses the importance of a complete evaluation procedure for SE.

0.14

Perceptually Better, Semantically Worse: Measuring Speech Enhancement Impact on LLM-Based Voice Systems

cs.SD eess.AS ·2026-08-31 ·Randy Frans Fela, Pejman Mowlaee
top venuebenchmark similarity 0.57
Abstract · Accepted at EMNLP 2026. 12 pages, 4 figures, 8 tables

Speech enhancement (SE) is commonly applied as a preprocessing step in spoken AI pipelines under the assumption that better audio quality improves downstream task performance. Whether SE-induced distortions propagate to downstream LLM task performance remains an open question. We introduce Output Divergence Rate (ODR), which measures how often SE changes an LLM's intent classification relative to clean speech, and benchmark five conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades. Every condition produces ODR significantly above zero ($p < 0.001$, binomial test). MetricGAN{+} more than doubles ODR versus unenhanced noisy speech (0.318 vs. 0.135) despite improving PESQ, and unmitigated echo reaches an ODR of 0.836 through speaker substitution, a failure WER cannot capture. Audio quality metrics range from near-zero to moderate correlation with ODR (SQUIM-MOS $ρ=-0.068$, PESQ $ρ=-0.467$). The MetricGAN{+} and echo results replicate across ASR architectures, indicating that standard audio quality metrics are insufficient for LLM pipeline quality.

0.47

How large an effect would we have seen? Positive controls, detection floors and dataset pitfalls in scalp-EEG speech decoding

eess.SP q-bio.NCq-bio.QM ·2026-10-14 ·Blessing Isoyiza Adeika, Onyema Osuagwu
similarity 0.57
Abstract · 22 pages, 4 figures, 8 tables plus supplementary tables. To be submitted to Journal of Neural Engineering

Objective. Studies that decode imagined speech from scalp electroencephalography (EEG) commonly validate their pipelines with a positive control, usually overt speech, and report null results without stating what effect size the pipeline could have detected. Approach. Using ds003626 ("Thinking out loud"), KaraOne and Track 3 of the 2020 International Brain-Computer Interface Competition, we audited decoding of overt speech, articulator preparation and imagined speech with artifact-channel subsets, band-limited decoding and trial selection, under a protocol that enforces split integrity by assertion. We injected class-dependent effects of known amplitude into the real recordings to measure the detection floor of five decoder families, and we checked the ds003626 labels against the dataset's acquisition protocol. Main results. On ds003626, two lip electrodes decoded overt speech at 0.7430 and the eight external channels at 0.7068, against 0.4231 for the 128-channel scalp montage, and no decodable signal survived in the 8-30 Hz band. On the other two corpora, within-subject accuracy rose monotonically with frequency into the range dominated by muscle activity. Band-power logistic regression detected an injected 8-30 Hz desynchronization of 0.54 uV under a criterion fixed in advance, whereas a compact convolutional network (EEGNet) detected nothing within subject up to 4.31 uV. In ds003626, the condition codes are the reverse of the intuitive reading, the class encoding changed between public releases, and condition is a deterministic function of time within each session. Significance. In these corpora the overt-speech positive control is dominated by myogenic signal and cannot validate a decoding pipeline. We recommend that speech-decoding studies report band-limited and artifact-channel accuracy, the direction of accuracy across frequency, and a measured detection floor.

0.39