CS Paper Radar

Neural Multichannel Distant Speaker Diarization and Source Separation with Beta Speaker Activity Prior

Sicheng Mao, Mathieu Fontaine, Anthony Larcher, Roland Badeau

cs.SD ·submitted 2026-08-21 08:35 UTC · v1
top venuetheory

Abstract

Distant speaker diarization remains challenging due to adverse acoustic conditions, varying numbers of speakers and overlapping speech. While data-driven approaches have shown strong performance, model-driven methods offer a compelling alternative by leveraging spatial information from multichannel recordings. This paper is motivated to propose a Bayesian diarization model for a model-driven method called neural FCASA to enhance its robustness. Specifically, we propose a beta prior over speaker activity and hence a variational lower bound objective that can be seen as a regularized continuous speaker activity score in place of the original cross-entropy loss to train the diarization model. Our experiments show significant improvements in terms of Diarization Error Rate by at least 3% (16% relatively) and Jaccard Error Rate by at least 4% (20% relatively) on the AMI dataset compared to the baseline.

Comment
Accepted by Interspeech 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Neural Multichannel Distant Speaker Diarization with Heavy-tailed Source Separation Model

cs.SD cs.AI ·2026-09-10 ·Sicheng Mao, Baihan Li, Mathieu Fontaine, Anthony Larcher, Roland Badeau
published similarity 0.80
Abstract · Accepted by IEEE SLT 2026

Distant speaker diarization remains challenging due to difficult acoustic environments, varying numbers of speakers and overlapping speech. Model-driven methods are proposed to exploit the speech source features in multi-channel recordings that help diarization. This paper generalizes a neural model that jointly learns to perform blind source separation and diarization over speech mixtures (neural FCASA) with heavy-tailed models. The popular Gaussian distribution has been applied for variance modeling in the original source separation model, which we replace with two families of heavy-tailed models (Leptokurtic Generalized Gaussian distribution and Student's t distribution) to better capture the heavy-tailedness in speech signals. Thanks to the Gaussian scale mixture model, we are able to unify the proposed method and the original one under the same form of learning objective. Our experiments show consistent large improvements in Diarization Error Rate (DER) and Jaccard Error Rate (JER) compared to the baseline on various corpora.

0.27

Beyond DER: Speaker Counting in Crowded End-to-End Diarization

eess.AS ·2026-09-23 ·Lahiru Samarakoon, Hongxu Zhu
publishedhot direction similarity 0.71
Abstract · Accepted to IEEE SLT 2026

Speaker diarization must solve two problems: counting how many speakers are present in a given conversation, and assigning speech to each one. This becomes harder in crowded conversations with five or more speakers. We evaluate how end-to-end neural diarization models count speakers, and find systematic under-counting in crowded recordings. The standard diarization error rate (DER) hides this failure, because it is duration-weighted and barely penalizes the dropped, low-activity speakers. We therefore also report the Jaccard error rate (JER) and explicit counting metrics. We propose a gated loss that couples speaker existence with frame activity. This loss can be computed in two ways, duration-weighted or speaker-weighted, and the speaker-weighted variant, used as a regularizer, reduces the under-count. On real-world crowded recordings our method clearly improves the counting metrics, lowering JER by about 6% relative and DER by about 9% relative, while leaving sparse recordings unharmed.

0.59

S2SND-v2: Unified Online and Offline Speaker Diarization with Speaker-Adapted w2v-BERT 2.0

cs.SD ·2026-10-26 ·Jincheng Wang, Li Li, Weixin Zhu, Yannan Wang, Tao Yu +2 more
similarity 0.66
Abstract · Submitted to ICASSP 2027. 5 pages, 2 figures, 2 tables

This paper presents S2SND-v2, a unified online and offline speaker diarization framework built on S2SND that improves acoustic feature extraction and the use of the resulting representations. For acoustic feature extraction, we introduce a speaker-adapted w2v-BERT 2.0 front end and compare full and pruned construction routes. To better use these features, overlap-aware activity-query masking improves speaker embedding extraction under overlap, while late-speaker training aligns unknown-speaker training with online inference conditions. Combining these improvements, the pruned system achieves competitive average diarization error rates of 14.52% online and 12.03% offline across eight test sets.

0.12

Training-Free Affinity Fusion of Neural and Embedding-Based Speaker Diarization

cs.SD cs.LG ·2026-09-30 ·Yehoshua Dissen, Joseph Keshet, Eduard Golshtein
similarity 0.66
Abstract · submitted to ICASSP 2027

Speaker diarization systems based on speaker embeddings and neural diarization exploit complementary forms of speaker information, but their intermediate representations are not directly compatible. We introduce Training-Free Affinity Fusion (TFAF), which integrates the speaker structure inferred by a neural diarizer into an embedding-based diarization system. The neural speaker partition is used to condition local speaker representations, from which we construct a continuous affinity matrix and combine it with the embedding-based acoustic affinity before a single global clustering step. The method requires no additional training, shared embedding space, speaker-label alignment, or hard transfer of the neural diarizer's speaker count. Experiments on AMI and CALLHOME show consistent DER improvements over both constituent systems; on AMI, fusion also improves speaker-attributed transcription. Ablations show that the neural speaker partition accounts for most of the gain, while retaining the continuous embedding-based affinities provides additional benefit over hard partition fusion.

0.22

SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation

eess.AS cs.CLcs.LG ·2026-10-03 ·Séverin Baroudi, Hervé Bredin, Ricard Marxer
benchmarkcode/data similarity 0.64
Abstract · Submitted to ICASSP 2027

Self-supervised learning (SSL) is standard for speech representation learning, but mainstream models are designed around single-speaker audio, limiting their usefulness in multi-speakers scenarios. We present SepRQ, an open-source SSL framework that replaces masked prediction with a pseudo-source-separation objective over frozen random-projection codebooks. By adopting a novel mask-free, multiresolution approach, SepRQ achieves state-of-the-art performance in Speaker Diarization and Speech Separation on the SUPERB benchmark, surpassing WavLM and other cocktail-party derived SSLs at both Base and Large scales, while requiring only 85.68M inference parameters. SepRQ also demonstrates strong performance across target-speaker tasks requiring enrollment (such as Target-Speaker Automatic Speech Recognition), and on the challenging multi-domain DIHARD 3 diarization dataset. Notably, we report strong separation capabilities on three-speaker mixtures (WSJ0-3Mix), where current SSL literature struggles. While cocktail-party SSLs remain scarce and closed-source, limited to C-HuBERT and the enrollment-based SA-WavLM, we open-source SepRQ to the community.

0.26

Learning Speaker-Activity-Conditioned WavLM Speaker Embeddings with Real Conversational Data

eess.AS ·2026-10-18 ·Prachi Singh, Nguyen Hai Phong, Petr Pálka, Jiangyu Han, Marc Delcroix +3 more
published similarity 0.64
Abstract · Accepted to SLT 2026

End-to-end neural diarization with vector clustering (EEND-VC) is a strong framework for long-form conversational diarization, where speaker embeddings are used to link local speaker decisions across windows. Conventional EEND-VC systems extract these embeddings by cutting out regions assigned to each local speaker, preferably using only non-overlapped speech. This works well when enough clean target-speaker speech is available, but it discards useful speaker evidence in overlapped regions and becomes unreliable when a local speaker appears mostly in overlap. We propose a speaker-activity-conditioned embedding extractor that adapts a pretrained WavLM-MHFA speaker verification model to extract target-speaker embeddings directly from full conversational windows. Inspired by the recently proposed diarization-conditioned Whisper (DiCoW) model for target-speaker ASR, frame-level diarization-dependent transformations inject local speaker-activity information into intermediate WavLM representations while preserving the pretrained initialization. Experiments on multiple conversational diarization benchmarks show that fine-tuning with limited real meeting data reduces speaker-confusion errors and improves EEND-VC diarization performance. The results further indicate that real conversational data provides an effective alternative to large-scale simulated-mixture training for diarization-aware speaker embedding extraction.

0.16

Spatially-Guided Speaker Extraction Based on Relative Position Cues

cs.SD eess.AS ·2026-11-01 ·Aviad Eisenberg, Shlomo E. Chazan, Sharon Gannot
similarity 0.64
Abstract

We present a multi-channel target speaker extraction (TSE) framework that is guided by relative spatial cues rather than explicit speaker-dependent references, such as clean enrollment utterances or precise directions of arrival (DOA). The proposed approach encodes the target speaker through discrete, interpretable relations between concurrent speakers, such as left and right or near and far, referred to as the Relative Position Cue (RPC). This relational conditioning effectively leverages spatial information while avoiding explicit DOA estimation, thereby enabling more robust operation under realistic conditions. We evaluate the proposed method across multiple datasets encompassing diverse acoustic conditions, including directional and diffuse babble noise, varying numbers of microphones, mixtures with different numbers of speakers, and dynamic scenarios involving moving speakers. Experimental results demonstrate that the RPC-based approach consistently achieves strong extraction performance and generalizes well across all tested configurations, establishing it as a practical and interpretable alternative for spatially informed speaker extraction.

0.31

Pandemonium: Longform Target-Speaker Speech Recognition with Cross-Speaker Interactions

eess.AS cs.CLcs.SD ·2026-11-07 ·Alexander Polok, Dominik Klement, Ivan Medennikov, Antoine Laurent, Petr Pálka +7 more
code/data similarity 0.64
Abstract · Submitted to ICASSP 2027

Diarization-conditioned target-speaker ASR (TS-ASR), in particular Diarization-conditioned Whisper (DiCoW), is an accurate and data-efficient approach to multi-talker transcription. It has, however, three structural limitations: the model transcribes each speaker independently, operates on short (30 s) windows, and optimizes its diarization and TS-ASR components separately. We address the first two. We integrate the diarization conditioning into a longform Parakeet backbone (DiCoP) and add cross-speaker attention, which lets the model attend across the speaker dimension so that speakers are no longer decoded in isolation, and an STNO mask refiner that re-estimates the diarization-derived conditioning. As conversational data with genuine multi-speaker interaction is scarce, we scale training with a pseudo-labeling pipeline that mines 5.4k hours of in-the-wild multi-speaker audio from YODAS2. Combining these with synthetic and real data yields state-of-the-art results across several meeting benchmarks. We open-source the resulting models, DiCoP and Pandemonium, and the pseudo-labeled data.

0.18