CS Paper Radar

AD-Mamba: Architecture-Complementary Knowledge Distillation for Efficient and Robust Audio Classification

Junyu Wang, Tianrui Wang, Meng Ge, Longbiao Wang, Xiaobao Wang, Jianwu Dang

eess.AS ·submitted 2026-10-20 06:30 UTC · v1

Abstract

Recent advancements in large-scale audio classification have been primarily driven by Transformer-based architectures due to their strong capacity in modeling long-range dependencies. However, their inherent quadratic complexity limits scalability and efficiency, especially when processing long or sparsely informative audio streams. Mamba, a recently proposed state space model (SSM), offers a promising alternative by enabling efficient long-range modeling with linear sequence scaling. Nevertheless, existing Mamba-based audio classification methods still lag behind mainstream Transformer-based models, suggesting that effective training strategies for fully exploiting the audio modeling potential of SSMs remain underexplored. To address this issue, we propose AD-Mamba, an architecture-complementary multi-stage knowledge distillation framework that transfers the local modeling strength of CNNs and the global contextual modeling capability of Transformers into a Mamba-based student model. Specifically, the CNN-guided stages encourage the student to adapt to local spectro-temporal structures, while the Transformer-guided stage further refines global semantic discrimination and contextual modeling. Experimental results on AudioSet show that AD-Mamba achieves 50.3% mean average precision (mAP) with only 30M parameters, outperforming representative Transformer-based and Mamba-based audio classification baselines with lower computational cost. Further evaluations under long-form audio, sparse-information, and noise-interference conditions show that AD-Mamba consistently outperforms representative CNN and Transformer baselines. When transferred to FSD50K, ESC-50, and DCASE16, the proposed model also achieves superior performance, demonstrating strong generalization across audio classification tasks.

Closest papers

Nearest neighbours by meaning across the whole index.

DAVSS: Distilled Audio-Visual State Space Models

eess.AS ·2026-08-20 ·Saurabhchand Bhati, Mrudula Athi, Amit S. Chhetri, James Glass
similarity 0.73
Abstract

State-space models (SSMs) distilled from transformer teachers combine the performance of transformers with the efficiency of SSMs. We extend the Transformer-SSM knowledge distillation to a multimodal setting and propose the Distilled Audio-visual State-Space (DAVSS) model. The DAVSS model, 14M parameters, is 12 times smaller compared to transformer-based models such as CAV-MAE, and still outperforms them. DAVSS improves over the existing audio-visual models by: 1) Finer input resolution: using smaller patch sizes process the input, compensating for the smaller model size by increasing input sequence lengths. This is supported by the observation that a larger patch size results in lower performance. 2) Deeper joint modeling: utilizing a larger portion of the model (30%) for joint audio-visual processing, compared to <5% in CAV-MAE, enabling deeper cross-modal interaction without significantly increasing the computational cost associated with the concatenated audio-visual tokens.

0.13

Toward a Universal Local Speech Feature Extractor Through Distillation

cs.CL ·2026-10-17 ·Jessica Yang, Marcelo Beramendi Caballero, Ju-Chieh Chou, Karen Livescu
benchmark similarity 0.72
Abstract

Self-supervised speech models commonly consist of a set of convolutional neural network (CNN) layers used as a local feature extractor, followed by transformer layers. The CNN layers are computationally demanding, and are also similar across models and resemble the much simpler mel spectral features. We hypothesize that the CNN layers can be replaced by a simple, universal model. To test this hypothesis, we learn a two-layer feature extractor via distillation from one or more teacher models (HuBERT, data2vec, and WavLM). Experiments on SUPERB(-SG) benchmark tasks show that our model largely retains the performance of the teacher models and can also be used in an unseen model (wav2vec 2.0). Our feature extractor requires only 3% of the MACs of the CNNs, resulting in a $>$20% overall inference runtime reduction.

0.17

ResMamba: A Dual-Path Convolution-Mamba Model for Music Genre Classification

cs.SD cs.AIeess.AS ·2026-10-31 ·Yingyu Zhang, Pu Qu
similarity 0.72
Abstract

Music genre classification remains a fundamental yet challenging task in music information retrieval, requiring models that capture local time--frequency patterns and long-range temporal dependencies. To address this, we propose ResMamba, a hybrid architecture that augments a ResNet-style four-stage backbone with Mamba blocks. Building upon the selective structured state space sequence model, ResMamba first adapts the selective SSM into a 4D-compatible residual Mamba block that captures global dependencies at linear complexity. We then design the ConvMamba dual-path block, which combines this adapted Mamba branch with a parallel convolutional branch through a learned $1\times1$ fusion, jointly modeling global sequential context and local spectro-temporal patterns. Furthermore, we adopt a multi-scale fusion head that aggregates hierarchical representations across all four stages. Experiments on the GTZAN dataset demonstrate that our full ResMamba model achieves a 5-fold cross-validation accuracy of $88.99\% \pm 1.58\%$, surpassing the strongest published method in our comparison (88.64\%). All models are trained from scratch without any pretraining, external data, or data augmentation. Beyond GTZAN, ResMamba also generalizes to the larger FMA benchmarks under the same training protocol, attaining $64.52\% \pm 1.88\%$ on FMA-Small under 10-fold cross-validation --- the best published from-scratch result --- and $65.75\%$ on FMA-Medium under the official artist-level split, surpassing the widely cited SVM baseline of Defferrard \textit{et al.} (63\%).

0.12

UniAE-MoE: A Unified Audio Encoder via Mixture of Experts

cs.SD cs.AI ·2026-09-30 ·Shengbo Cai, Zhisheng Zhang, Zichao Nie, Jing Peng, Jingran Xie +1 more
benchmark similarity 0.70
Abstract

Large Audio Language Models (LALMs) rely on effective audio encoders for multi-task performance. We introduce UniAE-MoE, a unified audio encoder designed to model cross-domain audio representations and achieve outstanding downstream understanding performance via a Mixture-of-Experts (MoE) architecture. Specifically, we explore mainstream audio encoders and integrate those from Qwen2-Audio and Audio-Flamingo 3, which demonstrate superior downstream capabilities. To facilitate effective model fusion, we improve our encoder using SwiGLU with shared experts to decouple encoder networks, and we further introduce a two-stage instruction-tuning strategy to better adapt the model to diverse downstream tasks. Moreover, we propose the task-specific data scaling (TSDS) technique to enhance UniAE-MoE's understanding capabilities. On the XARES-LLM benchmark, UniAE-MoE attains a score of 0.802, achieving state-of-the-art performance. It also delivers top-tier performance in the official Interspeech 2026 Audio Encoder Capability Challenge, further demonstrating robust generalization across diverse audio tasks. Together, these results validate the effectiveness of UniAE-MoE for unified audio understanding across speech, music, and general audio domains.

0.13

FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation

cs.CL cs.SD ·2026-08-25 ·Feiyu Shen, Fenglong Xie, Junjie Li, Kun Xie, Lei Xie +11 more
code/datahighly cited similarity 0.69
Abstract · 20 pages, 3 figures. In this revision, the author list is ordered alphabetically by given name and an author-contribution statement is added…

A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio language model with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-language model to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.

0.54

MambaVoice: Lightweight Audiovisual Singing Voice Separation Via A Hybrid Mamba-Transformer Model

cs.SD ·2026-09-22 ·Adithi Shankar, Gopika Krishnan, Gloria Haro, Xavier Serra, Martín Rocamora
published similarity 0.69
Abstract · 6 pages, Accepted at ISMIR 2026

Isolating a target singing voice from a music video remains challenging, particularly in the presence of multiple vocalists and dense instrumental accompaniment. We propose MambaVoice, a lightweight audiovisual framework that leverages a hybrid Mamba--Transformer architecture for targeted singing voice separation. The model jointly encodes audio and visual streams using an attention-based band-split audio encoder and a spatio-temporal graph convolutional network (ST-GCN) for facial motion features. These modalities are fused through a multiplicative gating mechanism, enabling visual cues to selectively modulate audio representations. The fused features are processed by a hybrid backbone that combines Transformer self-attention with Selective State Space Models (SSMs), achieving efficient long-range temporal modeling with linear complexity. We evaluated MambaVoice on the Acappella and URSing datasets under challenging conditions, including mixtures with interfering singers. At 16.2 million parameters, the model demonstrates comparable performance, achieving 14.18 dB SDR on Acappella and strong cross-dataset performance on URSing, comparable to larger models at a fraction of the parameter count. These findings highlight the effectiveness of hybrid SSM--attention architectures for scalable, efficient audiovisual source separation, suggesting they are well-suited as lightweight components within larger pipelines. We conduct a perceptual study that further supports our improvements in objective metrics. We provide our implementation online.

0.37

Task-Aware Joint Pruning and Distillation for Efficient Audio Deepfake Detection

cs.SD cs.AI ·2026-10-04 ·Miao He, Peng Cheng, Zhongjie Ba, Qing Wen, Li Lu +2 more
top venue similarity 0.68
Abstract · 6 pages, 4 figures, Proceedings of Interspeech 2026

Advances in speech synthesis have made deepfake speeches increasingly convincing, posing growing threats to security. While self-supervised learning (SSL) based detectors achieve state-of-the-art performance, their computational demands (typically 300M+ parameters) prevent deployment on resource-constrained devices. Existing compression methods, designed mainly for content-centric tasks, struggle to maintain competitive performance when directly adapted to deepfake detection. We propose a Task-Aware Joint Pruning and Distillation framework that combines cross-domain knowledge distillation with movement-guided structured pruning to transfer forgery-discriminative knowledge and preserve critical structures under aggressive compression. Our framework reduces the model to 31.9M parameters with 6.3$\times$ FLOPs reduction, with an average performance drop of only 1.30\% across multiple datasets compared to the uncompressed baseline, demonstrating strong potential for on-device deployment.

0.32

Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding

cs.SD ·2026-09-16 ·Benedikt Mayrhofer, Enrique Orozco Olivares, Franz Pernkopf, Philipp Aichinger, Martin Hagmüller
similarity 0.68
Abstract · 7 pages, 4 figures, 4 tables. Accepted at IEEE Spoken Language Technology Workshop (SLT) 2026

Self-supervised learning (SSL) has improved speech representations, yet performance degrades in pathological domains such as electrolaryngeal (EL) speech, and the computational footprint of SSL models limits their applicability in real-time, on-device deployment. We propose a multi-teacher knowledge distillation framework to train a lightweight, streaming content encoder that generalizes across healthy (HE) and EL speech. Two teachers are distilled progressively: a frozen SSL model providing discrete phonetic cluster targets from HE speech, and an EL-fine-tuned speech recognition model supplying continuous bottleneck feature targets. Evaluated via downstream speech recognition, our approach reduces the EL word error rate to 21.2%, compared to 39.3% for the strongest zero-shot SSL baseline. Among causal convolutional, Transformer, Conformer, and Mamba-based student architectures, a Mel-Conformer achieves the best combination of EL accuracy and computational efficiency. The final encoder contains 21.9,M parameters and runs at a real-time factor of 0.30 under ONNX Runtime on a single CPU core.

0.30