CS Paper Radar

HAMMER: Harmonic-Aware Parallel Context Modeling and Discriminator-Free Perceptual Optimization for Speech Enhancement

Shang-Fu Chen, Szu-Wei Fu, Sung-Feng Huang, Rong Chao, Wen-Huang Cheng, Yu Tsao

eess.AS ·submitted 2026-09-18 00:33 UTC · v2
publishedcode/data

Abstract

Recent speech enhancement systems combine self-attention and Mamba to capture global interactions and long-range dependencies. Yet these hybrids usually operate as sequence mixers and do not explicitly exploit harmonic periodicity, a strong cue for preserving voiced speech under noise. Perceptual optimization poses another challenge. PESQ is non-differentiable, so many methods train auxiliary metric discriminators that increase complexity and introduce adversarial instability. We propose \ours, a harmonic-aware and discriminator-free speech enhancer built around two components. (i) The Time-Frequency Harmonic-aware Attention-Mamba (TF-HAM) block runs self-attention and bidirectional Mamba in parallel along both spectrogram axes, then applies a speech-adapted autocorrelation feed-forward network to encode local periodic structure. (ii) Metric-explicit perceptual refinement (MEPR) combines differentiable PESQ and log-likelihood-ratio losses to expose perceptual metric structure without a learned surrogate. On VoiceBank+DEMAND, \ours achieves 3.69 PESQ and 4.41 COVL with only 2.39\,M parameters, outperforming or matching discriminator-based systems. Inference-time perceptual contrast stretching further raises PESQ to 3.79 without retraining. The source code will be available at https://github.com/shangfuu/HAMMER.git.

Comment
1 page, 3 figures, accepted to ISCSLP 2026

Closest papers

Nearest neighbours by meaning across the whole index.

HC-AINet: An Auditory-Inspired Network with Harmonic Compensation for Low-Complexity Speech Enhancement

eess.AS ·2026-10-22 ·Nan Li, Xiaoxiong Wu, Huawei Liu, Ziping Zhao
similarity 0.67
Abstract · 12 pages, 5 figures, 9 tables

Speech enhancement on resource-constrained devices requires noise suppression with limited computation and minimal loss of spectral detail. Auditory-inspired subband processing reduces computation, but its smooth gains have limited ability to distinguish harmonic peaks from neighboring noise. We propose a harmonic-compensated auditory-inspired network (HC-AINet) for low-complexity speech enhancement. A triangular equivalent rectangular bandwidth (ERB) filterbank supplies compact features to a lightweight network comprising inverted-residual units, temporal Conformer layers, and parallel frequency-gated convolutions. The network predicts global and harmonic gains together with fusion weights. A harmonic-location map derived from the noisy spectrum confines the harmonic contribution to selected bins on the original frequency grid, while the global branch controls the broader spectral envelope. The compensated magnitude, noisy phase, and a magnitude-derived minimum-phase cue jointly condition a compact phase estimator for waveform reconstruction. On VoiceBank-DEMAND, the 64-band model achieves a PESQ of 3.20 and a STOI of 0.95 with approximately 0.27 million parameters.

0.20

Rethinking Batch Composition: Promoting Feature Diversity for Speech Enhancement

eess.AS ·2026-10-15 ·Mingchi Hou, Ina Kodrasi
similarity 0.65
Abstract

Speech enhancement systems are commonly trained using randomly sampled mini-batches, which may contain limited diversity in intrinsic acoustic characteristics of the clean speech signals. Motivated by previous findings linking clean speech characteristics to enhancement performance, we propose a feature-stratified mini-batch sampling strategy that divides training examples into quantiles according to a clean-speech characteristic and enforces equal representation within each mini-batch. Experiments across two large-scale English and Spanish speech datasets and a smaller dataset containing neurotypical and pathological speech demonstrate improvements over standard random mini-batch sampling across multiple objective enhancement metrics. Ablation studies demonstrate that the gains are robust across training runs, while their magnitude depends on the acoustic characteristic used for stratification. The proposed strategy requires no modification to the enhancement model or training objective, providing a simple and complementary approach to speech enhancement training.

0.11

Parallel Time-Band Mixing with Learned Observation-Adding for Robust ASR Front-Ends

cs.SD cs.AI ·2026-08-31 ·Xingyu Shen, Runze Wang, Wei-Ping Zhu, Benoit Champagne
top venuehot direction similarity 0.63
Abstract · Accept by Interspeech 2026

Speech enhancement is often used as a front-end for robust ASR, yet recurrent temporal and cross-band modules introduce sequential dependencies that reduce parallel efficiency. In this paper, we present a sequence-parallel band-split enhancement front-end built on a Parallel Time-Band Mixer (PTBM) block that eliminates within-block recurrent unrolling. PTBM integrates intra-band temporal mixing and per-frame cross-band attention within a unified parallel architecture, enabling efficient contextual modeling across both time and frequency dimensions. The system retains the mask-plus-residual reconstruction interface and introduces learned Observation-Adding (LOA) to suppress ASR-sensitive artifacts without development-set tuning. Experiments on DNS Challenge and CHiME-4 with frozen Whisper back-ends show that the proposed front-end consistently reduces word error rate relative to recurrent band-split baselines while requiring only 0.96 M parameters and 0.58 GMAC/s for the front-end network.

0.55

Silence-the-Mimic: Accelerating Imperceptible Perturbation Generation Against Voice Cloning

eess.AS ·2026-09-30 ·Runqiu Xu
published similarity 0.63
Abstract · Accepted at IEEE SLT 2026

Deep neural network-based Voice Conversion (VC) and Text-to-Speech (TTS) models have rapidly advanced, enabling realistic voice cloning with minimal input data. Such capabilities raise serious concerns over unauthorized cloning of speaker identities and the associated privacy and security risks. Current imperceptible adversarial protection methods rely on quality control losses that are highly sensitive to hyperparameter tuning and computationally expensive due to lengthy optimization. To address these limitations, we propose a fast protection method that generates perceptually constrained perturbations in the frequency domain under a psychoacoustic masking-based constraint. Our approach strictly enforces perceptibility bounds during adversarial training, eliminating the need for iterative quality balancing and significantly reducing computational cost. Experiments on multiple state-of-the-art VC and TTS models show that STM achieves competitive or superior protection performance with substantially better perceptual quality and up to $45.3\times$ speedup over existing white-box baselines. These results demonstrate the effectiveness of frequency-domain perturbations with perceptual constraints as a practical paradigm for protecting against voice cloning.

0.27

BLINC: Blind Calibration For Training-Free Speech Enhancement Adaptation

eess.AS cs.SD ·2026-09-18 ·Tobias Raichle, Ekaterina Gavrilko, Bin Yang
similarity 0.63
Abstract · This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may…

Speech enhancement (SE) models degrade under domain shifts and have to adapt to unseen target domains during deployment. Most existing test-time adaptation (TTA) methods for SE do so by adapting a subset of the model weights using a self-supervised loss, which requires backpropagation at test-time and permanently alters the model. We instead recalibrate the prediction itself and propose BLINC, a training-free TTA method that remaps the predicted time-frequency mask onto a bimodal target distribution by histogram matching. At test-time, the target distribution is parameterized from blind features of the noisy recording, so neither a reference distribution from a classical algorithm nor online metric optimization is involved. BLINC improves the overall quality of both evaluated SE models on almost every target condition and matches or exceeds the loss-based TTA baselines at minimal overhead.

0.23

Controlling Speaking Rate in Autoregressive TTS via Activation Steering

cs.SD cs.CLcs.LGeess.AS ·2026-09-27 ·Francesco Verdini, Antonis Asonitis, Aref Farhadipour, Marzieh Razavi, Pierre-Edouard Honnet +2 more
publishedbenchmark similarity 0.62
Abstract · Accepted at IEEE SLT 2026. 8 pages, 3 figures, 5 tables

Autoregressive text-to-speech (TTS) systems synthesize natural speech but, once trained, offer little control over speaking rate. We show that speaking rate can be steered at inference time, without retraining, by clamping a single decoder block's activation along a discovered speed axis. A decoder-block analysis recovers the rate axis, a neutral operating point, and a per-step intensity scale; at inference, the activation's projection onto this axis is set to a fixed scalar. Learning this direction from synthetically time-stretched and time-compressed speech yields rate control that largely preserves speaker identity, generalizes across model architectures, and maintains high naturalness in objective and human evaluations. Unlike standard additive steering, which breaks at the slow extreme, clamping remains stable on all three systems tested; at moderate targets, the better rule depends on the model. Finally, we show that rate information is decodable across layers but causally steerable only within a mid-depth window, and demonstrate the effectiveness of our approach on the public Seed-TTS-Eval benchmark.

0.35

GrainSpeech: Less Context, More Detail for Compact Speech Synthesis

eess.AS cs.AIcs.SDeess.SP ·2026-09-16 ·Zitao Liang, Chang Gao
code/data similarity 0.62
Abstract

Compact acoustic models face a challenging quality-capacity trade-off. We investigate two factors in this regime: encoder context and Mel-spectrogram supervision. A receptive-field-scaling study shows that expanding self-attention beyond 15 phonemes provides no consistent gains in pitch, energy, or duration prediction. Guided by this finding, we introduce a fixed-receptive-field convolutional encoder that reduces the respective prediction errors by 36.0%, 17.3%, and 3.4%. We further show that directly transferring image-domain gradient-variance supervision restores fine-scale variation but degrades predicted quality, motivating a Mel-specific formulation with axis-specific gradients, overlapping local statistics, and log-domain variance matching. GrainSpeech contains only 264.8K parameters and achieves 17.9x real-time Mel generation on a microcontroller (MCU), while attaining UTMOS scores comparable to substantially larger models with less than 1.5% of their parameters. Source code and demos are available at https://github.com/lab-emi/GrainSpeech.

0.38

UltraM2M: Leveraging Text Transcripts and Mixture Constraints for Weakly-Supervised Speech Enhancement

eess.AS ·2026-10-04 ·Liu, Jiachen, Wu, Fulin, Wang +1 more
similarity 0.62
Abstract · in submission

We propose UltraM2M, a weakly-supervised speech enhancement algorithm building upon the recent unsupervised mixture-to-mixture (M2M) algorithm. M2M realizes unsupervised speech enhancement by training deep neural networks on a set of real-recorded noisy-reverberant multi-channel mixture signals to estimate target speech and non-target signals. The two estimated signals are penalized by a so-called mixture-constraint (MC) loss, which constrains them to reconstruct the observed mixture signals. Although shown to be effective, the mixture constraint may be too weak to enable sufficient noise reduction. To deal with this, UltraM2M extends M2M by further leveraging text transcripts of real-recorded mixtures to design an automatic speech recognition (ASR) loss to penalize the estimated speech signal. The ASR loss can be viewed as a form of weak supervision that could help unsupervised enhancement. Evaluation results on the CHiME-4 dataset show the effectiveness of UltraM2M.

0.18