CS Paper Radar

Configurable-Bandwidth Time-Frequency Modeling for Efficient Full-Band Speech Enhancement Across Sampling Rates

Ui-Hyeop Shin, Wooseok Kim, Hyung-Min Park

eess.AS ·submitted 2026-09-24 12:22 UTC · v1
hot direction

Abstract

Speech enhancement systems are often developed for a fixed sampling rate, while time-frequency models become more expensive as the number of frequency bins increases. We propose TF-Refiner, a sampling-frequency-independent model that decouples the deep analysis bandwidth from the full-band input and output. A deep encoder processes the band below a configurable cutoff, while a shallow decoder combines the encoded features with input-dependent high-band queries and predicts local complex filters applied to the original noisy STFT. A single parameter set trained at 16 and 48 kHz is evaluated at various sampling rates. On VoiceBank+DEMAND, the universal model outperforms the rate-specific counterparts in PESQ, STOI, and log-spectral distance across the evaluated rates, including rates unseen in training. Random-cutoff training enables inference-time selection of cost-quality operating points without retraining or changing the output bandwidth. These results support configurable analysis bandwidth as a practical design choice for multi-rate full-band enhancement.

Comment
5 pages, 5 figures, 4 tables. Submitted to ICASSP 2027

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond Model Size: Redesigning LiSenNet for embedded speech enhancement

eess.AS cs.LGcs.SD ·2026-09-24 ·Clément Laroche, Rasmus Kongsgaard Olsson
similarity 0.67
Abstract

Deploying real-time speech enhancement on resource-constrained devices requires meeting strict latency, memory, and energy constraints. Microcontroller NPUs can accelerate neural inference under these constraints, but only through a restricted set of operators in static, integer-quantized graphs. Recent speech-enhancement networks have reduced parameter counts and MACs to levels nominally suitable for microcontrollers, but their operators and execution patterns often remain incompatible with restricted NPUs. We address this gap by redesigning LiSenNet, a 37k parameter sub-band dual-path model, for the STM32N6570-DK Neural-ART accelerator. We replace its recurrent bottleneck with convolutional frequency and temporal mixers, reformulate unsupported operations as static int8-compatible primitives, and use bounded decoder activations to preserve quality after quantization. On VoiceBank-DEMAND, the final NPU-compatible model matches or exceeds the recurrent LiSenNet baseline, reaching PESQ 3.08 versus 3.01 in FP32 and 3.01 versus 2.93 in int8. Deployed on a microcontroller, it processes each 16 ms input hop in 4.83 ms, corresponding to a real-time factor of 0.30. Stateless receptive-field recomputation is an order of magnitude slower at the same frame rate despite higher accelerator utilization. These results show that parameter count and operator compatibility, quantization range, and persistent streaming state must be co-designed to achieve efficient real-time speech enhancement on restricted NPUs.

0.15

Does per-frame early exit pay? A compute-matched study of dynamic depth for on-device speech enhancement

eess.AS cs.LGcs.SD ·2026-09-24 ·Clément Laroche, Riccardo Miccini
similarity 0.66
Abstract

Deep learning-based speech enhancement is increasingly deployed on-device in hearing aids, headsets, and earbuds. Most of these devices, however, can only accelerate static int8 graphs, so a depth-varying network must be implemented as several graphs, orchestrated by a policy. In this paper, we supervise every intermediate depth of one causal model, then we fine-tune its output heads to guarantee that deeper outputs are never worse than shallower ones. Using this training protocol, we can derive a family of static models that are more Pareto-efficient than their equivalently-sized counterparts trained from scratch on the same budget. Specifically, we achieve up to 0.11 higher PESQ for equivalent compute, and match the best PESQ at 30% less compute. We then quantize the models to int8 and measure the latency-quality frontier on an STM32N6 microcontroller. On VoiceBank-DEMAND, the dynamic enhancer lies on the same frontier as the static models, rather than trading quality for dynamic execution. Running the policy on the companion Cortex-M55 takes only 26 $μ$s per frame, while splitting the enhancer into separate NPU graphs adds 2.2% latency overhead. The cost of dynamic execution is therefore small.

0.24

Multirate State Space Models for End-to-End Processing of Pulse Density Modulated Speech Signals

cs.SD eess.AS ·2026-08-28 ·Ludovic Boulanger, Sean U. N. Wood
similarity 0.64
Abstract

Deep neural networks (DNNs) based on state-space models (SSMs) are increasingly applied to speech processing, but typically operate on pulse-code-modulated (PCM) audio. This constrains deployment on low-power, always-on edge devices, which commonly use single-bit pulse-density-modulated (PDM) micro-electromechanical (MEMS) microphones for their noise robustness, low cost, and variable sampling rates that enable low-power operation. In fact, converting PDM to PCM requires low-pass filtering and decimation, imposing costly overhead on resource-constrained hardware. While prior works have attempted to process PDM signals directly, they require long training times and generalize poorly across sampling rates. In this paper, we show that the SSM has two key properties that remediate these issues: its continuous-time parametrization allows it to produce a consistent representation of the input audio signal, regardless of the modulation strategy and sampling rate, and its long-term memory enables this representation to be aggressively downsampled without needing any anti-aliasing operations. We then propose a novel end-to-end PDM speech processing architecture that uses an SSM to encode the input audio signal into a modulation- and sampling-rate-invariant latent representation. We show that our proposed architecture achieves robust speech classification and enhancement gains at low-power sampling-rates (512 kHz) and similar performance to state-of-the-art algorithms operating on PCM data when tested on standard PDM sampling-rates of 2 MHz. Moreover, we show that the SSM's output can be downsampled by more than 65,000 times, thus significantly reducing the number of processing timesteps in downstream layers.

0.17

Bridge, Distill, Refine: High-Fidelity One-Step Generation for Multi-Bandwidth Speech Extension

eess.AS ·2026-10-16 ·Zhiqi Ai, Xu Shen, Mou Wang, Yongjin Zhou, Shugong Xu
similarity 0.63
Abstract · 5 pages, 4 tables, 1 fugure

Speech bandwidth extension (BWE) aims to recover missing spectral content from band-limited speech. However, most generative BWE methods are designed for fixed input bandwidths and rely on iterative sampling, limiting their flexibility and efficiency. We propose BDR-BWE, a three-stage generative framework for multi-bandwidth speech extension. First, to enable conditional generation from the observed band-limited spectrum, we design a cutoff-aware conditional flow bridge in the compressed complex STFT domain, focusing generation on the missing high-frequency band while preserving observed spectral information. Second, progressive consistency distillation compresses the multi-step flow transport into a one-step generator. Third, adversarial refinement further improves high-frequency detail and harmonic coherence. BDR-BWE achieves state-of-the-art log-spectral distance (LSD) on VCTK across all evaluated bandwidths with one-step inference, while maintaining competitive perceptual quality and generalizing well to EARS. Audio samples are available at https://aizhiqi-work.github.io/bdr-bwe-demo.

0.23

Realizing Low-Precision Storage and Sparse Execution in a Folded FPGA Speech Enhancer

eess.AS cs.ARcs.LG ·2026-10-26 ·Clément Laroche
similarity 0.63
Abstract · 5 pages, 3 figures, 2 tables. Submitted to IEEE ISCAS 2027

Hardware-aware training often optimizes FPGA resources, latency, or bit operations, but folding changes what compression saves: shared arithmetic decouples precision, weight storage, and scheduled work. We derive training objectives from the measured costs of a folded streaming speech-enhancement accelerator. Mixed-precision weights reduce on-chip storage; structured channel sparsity removes entire operations from the engine schedule. We deploy a bit-exact implementation on the Kria KV260. The best sparse model eliminates 47.3% of convolution MACs, cutting frame time by 45% (1.86 to 1.03 ms) and dynamic energy by 60% (83 to 33 $μ$J/frame) against a storage-priced model measured in the same session, with higher speech quality. Relative to the most efficient dense build measured separately, frame time falls by 42% and energy by 48%. Reducing weight width from 8 to 6 bits lowers dynamic power by 27% but changes frame time by less than 5%. Plain channel group-lasso removes more work than cycle-cost-weighted lasso. Thus, folded-accelerator training should price storage and removable work separately, rather than rely on a single bit-operation metric.

0.32

HC-AINet: An Auditory-Inspired Network with Harmonic Compensation for Low-Complexity Speech Enhancement

eess.AS ·2026-10-22 ·Nan Li, Xiaoxiong Wu, Huawei Liu, Ziping Zhao
similarity 0.63
Abstract · 12 pages, 5 figures, 9 tables

Speech enhancement on resource-constrained devices requires noise suppression with limited computation and minimal loss of spectral detail. Auditory-inspired subband processing reduces computation, but its smooth gains have limited ability to distinguish harmonic peaks from neighboring noise. We propose a harmonic-compensated auditory-inspired network (HC-AINet) for low-complexity speech enhancement. A triangular equivalent rectangular bandwidth (ERB) filterbank supplies compact features to a lightweight network comprising inverted-residual units, temporal Conformer layers, and parallel frequency-gated convolutions. The network predicts global and harmonic gains together with fusion weights. A harmonic-location map derived from the noisy spectrum confines the harmonic contribution to selected bins on the original frequency grid, while the global branch controls the broader spectral envelope. The compensated magnitude, noisy phase, and a magnitude-derived minimum-phase cue jointly condition a compact phase estimator for waveform reconstruction. On VoiceBank-DEMAND, the 64-band model achieves a PESQ of 3.20 and a STOI of 0.95 with approximately 0.27 million parameters.

0.20

GigaEdge Spench: An SDK for Practical Streaming Speech Enhancement

cs.LG eess.AS ·2026-11-09 ·Artem Sokolov, Artem Khaidarov, Yana Lysova, Alexander Gladkikh, Vitaliy Yulis
code/data similarity 0.61
Abstract · Additional checking required

We present GigaEdge Spench, an open-source SDK for streaming speech enhancement on CPU servers, mobile devices, and in browsers. Applications can build enhancement channels from neural denoising, normalization and dynamics blocks, sample-rate conversion, and acoustic echo cancellation where supported. These stages can be combined, while normalization can also run without a neural model. The release includes an 8 kHz INT8 denoiser and discriminative and hybrid 48 kHz FP32 models. A shared C++ core supports ONNX Runtime and TensorFlow Lite through native interfaces, Android and iOS SDKs, and WebAssembly integration. Independent stream contexts share model resources, and consecutive frames from one stream can be processed in a single inference call. We evaluate the released models on EARS-WHAM v2 and compare their quality and processing costs with RNNoise and DPDFNet2. All tested native and Web configurations process files faster than real time, although larger chunks require more time to collect input. We also measure how many streams a shared model can process on a CPU. Our contribution is a configurable deployment system and an experimental study of how model choice, chunk size, and execution environment affect its use. All reported SDK timing uses ONNX Runtime. Index Terms: speech enhancement, streaming inference, cross-platform deployment, temporal frame grouping, computational efficiency

0.26

Tracing Decoder Artifacts for Compact Synthetic Speech Screening

cs.SD cs.AIeess.AS ·2026-09-25 ·Yi Chen Liu, Jian Liu
similarity 0.61
Abstract

Recent advances in speech synthesis and voice cloning have increased the need for reliable synthetic-speech detection, yet high-accuracy detectors increasingly rely on large pretrained models that are costly to invoke on every recording. Rather than replacing such detectors, we investigate a compact front-end screen that processes all inputs cheaply and forwards only suspicious recordings for more expensive analysis. To enable lightweight screening without a large learned encoder, we exploit spectral traces introduced by speech-generation operations. We analyze how learned upsampling and inverse short-time Fourier transform synthesis can produce predictable spectral artifacts and measure their presence directly in generated waveforms. Because the strength of these artifacts varies across generators, we combine decoder-guided spectral measurements with complementary descriptors of short-time spectral shape and temporal variation in a compact gradient-boosted tree. Across seven speech generators and two human-speech sources, the proposed screen achieves an equal error rate of 0.021\% with an estimated model storage of 151 KiB. When used as the first stage of a simulated cascade with a 1.15-billion-parameter detector, it reduces estimated detection energy by 84.4\% while operating at a 0.050\% synthetic-speech miss rate, demonstrating the potential of decoder-guided acoustic evidence for low-cost front-end screening.

0.30