CS Paper Radar

Domain-Incremental Learning for Generative Speech Enhancement

Manjunath Mulimani, Annamaria Mesaros, Minje Kim, Jesper Rindom Jensen

eess.AS cs.SD ·submitted 2026-09-28 11:06 UTC · v1

Abstract

We propose a domain-incremental learning framework for generative speech enhancement (SE) that learns from a sequence of datasets or domains recorded under diverse acoustic conditions. Fine-tuning a pretrained model on continuously evolving domains leads to catastrophic forgetting of previously acquired knowledge, while zero-shot generalization often fails to adequately adapt to unseen domains. To address these challenges, we first develop a novel language model-based generative SE model that we then use as a pretrained backbone and incrementally adapt it to acoustically mismatched domains using lightweight domain-specific Low-Rank Adaptation. The proposed framework enables the model to acquire enhancement capabilities for new domains while preserving performance on previously learned domains. Evaluated on four heterogeneous speech datasets, our approach effectively adapts to new domains without forgetting previously learned domains.

Comment
Submitted to the IEEE International Conference of Acoustics, Speech, and Signal Processing (IEEE ICASSP 2027)

Closest papers

Nearest neighbours by meaning across the whole index.

Parameter isolation with domain-specific experts for incremental audio classification

eess.AS cs.LGcs.SD ·2026-09-13 ·Jongyeon Park, Do-Hyeon Lim, Sang-won Park, Hong Kook Kim, Kyungdeuk Ko +2 more
similarity 0.75
Abstract · 5 pages, 3 figures, 3 tables. Accepted to the Detection and Classification of Acoustic Scenes and Events (DCASE) Workshop 2026

To successfully deploy a model in time-varying environments such as streaming data prediction and sensing control, domain-incremental learning (DIL) has attracted attention since it aims to adapt a previously trained model to newly arriving domains, while reserving knowledge from earlier domains without accessing their data. Incremental learning across domains can be regarded as a recurrent update, in which the current model is obtained by updating the model carried over from previous domains. Conventional DIL approaches that rely on domain-invariant feature learning and weight regularization gradually overwrite or constrain parameters learned in previous domains, leading to catastrophic forgetting. Instead, this paper proposes a new domain-specific parameter-isolation architecture that retains all past domains. The proposed architecture mitigates catastrophic forgetting through a full-order recurrent update, constructing a new expert using domain-specific data conditioned on all previously frozen models. To achieve this, we incorporate data-free generative replay to reconstruct previous-domain data and cross-domain feature generation to recover later expert features missing from earlier domain samples. Finally, we apply the proposed model architecture to domain-agnostic incremental learning for audio classification, as defined in the DCASE 2026 Challenge Task 7. Consequently, we achieve micro and macro accuracies of 78.4% and 78.9%, respectively, representing increases of 33 and 25 percentage points over the Challenge baseline. Ablation studies are conducted to examine the effectiveness of each processing component in terms of classification accuracy.

0.22

DriftSE: Speech Enhancement with Generative Drifting

cs.SD cs.AIeess.AS ·2026-09-10 ·Liang Xu, Diego Caviedes-Nozal, W. Bastiaan Kleijn, Longfei Felix Yan, Rasmus Kongsgaard Olsson
code/data similarity 0.71
Abstract · 11 pages, 3 figures

We propose DriftSE, a novel one-step generative framework for speech enhancement formulated as a latent distribution equilibrium problem. During training, the drifting field aligns the generator's pushforward distribution with the clean speech manifold through drifting in a latent domain. During inference, the drifting process is discarded, enabling one-step generation. We establish that its enhancement quality depends fundamentally on the choice of latent representation. Semantic latents preserve phonetic structure but fail to capture physical acoustic cues, whereas acoustic latents reconstruct the physical signal but risk linguistic hallucination. Therefore, we introduce dual-latent drifting, performing parallel drifting in both semantic and acoustic latents to simultaneously preserve phonetic intelligibility and acoustic fidelity. Additionally, we demonstrate that DriftSE enables fully unpaired training by aligning latent distributions rather than exact point-wise targets. Consequently, DriftSE facilitates cross-dataset learning in the absence of paired noisy-clean samples. Moreover, DriftSE exhibits broad architectural flexibility across different generator backbones. Extensive evaluations on additive denoising and convolutive dereverberation demonstrate robust one-step enhancement across both offline and real-time causal settings. Notably, DriftSE achieves state-of-the-art word error rates across all four evaluated datasets while strictly operating at 1 NFE. Code and audio examples are available online.

0.65

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

eess.AS cs.CL ·2026-09-16 ·Mohan Shi, Zilai Wang, Natarajan Balaji Shankar, Kaiyuan Zhang, Eray Eren +1 more
published similarity 0.71
Abstract · Camera ready for IEEE SLT 2026 (figure1 revised)

Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Speech Recognition (ASR) performance on general-domain speech. However, adapting them to domain-shifted speech, such as child or dialectal speech, remains challenging under limited target-domain data. Given the dominant role of the LLM in Speech-LLMs, with cross-entropy loss applied only at the LLM output, the speech encoder may receive insufficient adaptation to new acoustic conditions. In this paper, we propose Encoder Awakening via Adapters (EAVA), a simple yet effective domain-adaptive fine-tuning method for Speech-LLM-based ASR. First, lightweight adapters are inserted into each encoder layer and trained exclusively, enabling target-domain acoustic knowledge to be incorporated into the encoder while preserving its pre-trained knowledge. Second, the full model is jointly fine-tuned on the target domain with LoRA applied to the LLM. Experiments on three domain-shifted ASR datasets, covering child and dialectal speech, show that EAVA consistently outperforms vanilla fine-tuning and other baselines, achieving new state-of-the-art performance.

0.31

RemixIT-TSE: Progressive Synthetic-to-Real Adaptation for Target Speech Extraction via Target-Aware Supervision and Remixing

cs.SD ·2026-09-28 ·Yu Wang, Haixin Guan, Shuang Wei, Yanhua Long
code/data similarity 0.69
Abstract · 5 pages, 1 figure

Target Speech Extraction (TSE) in real-world conversational scenarios suffers from severe performance degradation due to the domain gap between synthetic training data and complex acoustic environments, where signal-level ground truth is typically unavailable. To address this challenge, we make the first attempt to extend RemixIT from speech enhancement to TSE and propose a progressive synthetic-to-real adaptation framework for real-world TSE with two fine-tuning stages. The first stage leverages region-wise speaker similarity and silence constraints within a target-aware adaptation framework to jointly optimize the model using synthetic and weakly supervised real-world data, injecting real-world traits while preserving synthetic-learned capabilities. The second stage further adapts the model using only real-world data through our RemixIT-TSE, where quality-filtered teacher pseudo targets, which guarantee reliable student training, provide signal-level supervision via SI-SNR loss. Experiments on the real conversational evaluation set (EVAL-2) of the SLT 2026 REAL-TSE Challenge, the proposed method achieves a 6.53% relative TER reduction, together with relative improvements of 21.84% in speaker similarity, 9.89% in DNSMOS-P808, and 4.10% in target-activity F1 over the source-domain baseline, demonstrating its effectiveness under unseen real-world conditions. Source code at https: //github.com/YuWang-Speech/RemixIT-TSE.

0.32

Conflict-Aware Meta-Learning for Generalizable Speech Deepfake Detection

eess.AS ·2026-11-05 ·Siqing Qin, Kong Aik Lee, Youzhi TU, Eng Siong Chng, Man-Wai Mak
similarity 0.69
Abstract · submit to TASLP; 13 pages

Speech deepfake detectors built on large self-supervised learning front-ends achieve high in-domain accuracy, yet suffer degradation under domain shift caused by unseen synthesizers, transmission codecs, or unfamiliar acoustic conditions. Meta-learning, particularly model-agnostic meta-learning for domain generalization (MLDG), addresses this issue by simulating domain shifts. However, applying MLDG directly introduces a bi-level optimization challenge that remains unsolved. First, effective domain-shift emulation requires genuine distributional gaps between meta-splits, yet such gaps can frequently produce gradient conflicts, destabilizing the bi-level optimization. Second, the stochastic nature of mini-batch gradients causes the reference direction to fluctuate across iterations, resulting in an optimization trajectory that fails to achieve domain generalization. We propose a conflict-aware meta-learning (CAML) framework, which addresses both challenges through three complementary components. Firstly, base domain gradient surgery (base-DGS) introduces an asymmetric projection that treats the meta-train gradient as a primary anchor. Secondly, EMA-DGS replaces the unstable anchor with its exponential moving average (EMA), providing a stable optimization trajectory anchor that substantially reduces domain-shift-induced conflict detections. Thirdly, layer-wise DGS (LW-DGS) further extends the framework to individual parameter tensors, enabling fine-grained, layer-specific intervention that targets conflict-prone layers while leaving well-aligned parameters intact. Experimental results demonstrate that the proposed CAML framework achieves lower EER than the compared baselines on most cross-dataset evaluations across multiple cross-dataset benchmarks and architectures, achieving a maximum relative improvement of 45.2% in terms of EER.

0.08

Rethinking Batch Composition: Promoting Feature Diversity for Speech Enhancement

eess.AS ·2026-10-15 ·Mingchi Hou, Ina Kodrasi
similarity 0.69
Abstract

Speech enhancement systems are commonly trained using randomly sampled mini-batches, which may contain limited diversity in intrinsic acoustic characteristics of the clean speech signals. Motivated by previous findings linking clean speech characteristics to enhancement performance, we propose a feature-stratified mini-batch sampling strategy that divides training examples into quantiles according to a clean-speech characteristic and enforces equal representation within each mini-batch. Experiments across two large-scale English and Spanish speech datasets and a smaller dataset containing neurotypical and pathological speech demonstrate improvements over standard random mini-batch sampling across multiple objective enhancement metrics. Ablation studies demonstrate that the gains are robust across training runs, while their magnitude depends on the acoustic characteristic used for stratification. The proposed strategy requires no modification to the enhancement model or training objective, providing a simple and complementary approach to speech enhancement training.

0.11

Test-time adaptation for speech enhancement with an autoregressive speech prior

cs.SD cs.AI ·2026-09-03 ·Sofiene Kammoun, Simon Leglaive, Xavier Alameda-Pineda, Timo Gerkmann
code/data similarity 0.69
Abstract · Submitted to IWAENC 2026

Test-time adaptation (TTA) offers a promising direction for improving speech enhancement models under mismatched acoustic conditions, without requiring access to labeled target data. In this work, we propose a single-utterance TTA method that regularizes a pretrained speech enhancement model using an autoregressive prior trained on clean speech latent representations extracted from a neural audio codec. Adaptation is performed by minimizing the Kullback-Leibler divergence between the enhanced speech distribution and the clean speech prior. Experiments across multiple noisy speech datasets show consistent improvements in speech quality, particularly under training-testing noise mismatch conditions. Code and audio examples are available online.

0.50

Cleaner Speech, Weaker Generalization: Revisiting Pitt-Derived Benchmarks for Alzheimer's Disease Detection

cs.SD cs.AI ·2026-08-31 ·Luqi Sun, Shreeram Suresh Chandra, Lin Zhang, You-Jin Li, Brian MacWhinney +3 more
similarity 0.68
Abstract

Speech-based Alzheimer's disease (AD) detection increasingly relies on speech-enhanced and curated versions of the Pitt Corpus, where speech enhancement, sample selection, and demographic balancing are often treated as beneficial preprocessing steps. However, whether these transformations improve real-world AD detection or instead affect model generalization and prediction behavior remains unclear. In this work, we revisit the role of speech preprocessing and dataset curation across widely used benchmarks for speech-based AD detection. We evaluate the speech quality of different datasets, the cross-dataset generalization of multiple deep learning models under matched and mismatched enhancement settings, and the behavior of several recent large audio-language models (LALMs). Experimental results show that across multiple supervised speech models, speech-enhanced datasets often improve in-domain performance while reducing robustness in cross-domain evaluation. Matched enhancement between training and test data alleviates, but does not eliminate, this degradation. LALMs show a similar sensitivity: enhanced datasets induce stronger class imbalance and prediction shifts than unprocessed data. These results suggest that speech preprocessing and dataset curation can substantially influence downstream AD detection behavior, indicating that ``cleaner'' speech datasets are not necessarily more reliable for real-world AD detection.

0.22