CS Paper Radar

Beyond EER: Multi-Dimensional Evaluation of Information Leakage in Speaker De-Identification

Seungmin Seo, Oleg Aulov, P. Jonathon Phillips, Kevin Mangold, Jonathan Eskin

cs.SD cs.AI ·submitted 2026-09-16 13:50 UTC · v1
published

Abstract

Speaker de-identification (SDID) aims to preserve privacy by concealing speaker identity while maintaining speech utility. However, current evaluations often reduce privacy to a single dimension - biometric verification performance - typically measured by Equal Error Rate (EER). This narrow focus ignores critical leakage channels, such as soft biometric inference, embedding-level re-identification, and structural template similarity, which threaten the unlinkability and irreversibility of biometric references. We propose a holistic evaluation framework across five complementary metrics: (i) EER, (ii) soft biometric leakage score , (iii) cumulative match characteristic re-identification analysis, (iv) canonical correlation analysis and Procrustes embedding alignment, and (v) intelligibility via word error rate and semantic similarity. Evaluating five SDID systems from the IARPA ARTS program, we demonstrate that these metrics capture independent dimensions of information leakage. Our results indicate that reliance on a single metric can misrepresent the privacy properties of an SDID system.

Comment
Accepted to IJCB 2026 (Main Track)

Closest papers

Nearest neighbours by meaning across the whole index.

Identity by Design, Demographics by Accident: Demographic Leakage and Suppression in Behavioral Biometric Embeddings

cs.CR ·2026-08-28 ·Iyadh Khan, Patrick Nilackshan, Mohamed Aathif, Mohamed Theesan, Sandareka Wickramanayake +1 more
benchmark similarity 0.69
Abstract

Behavioral biometric authentication (BBA) systems use deep learning models to transform biometric signals, such as eye movements, voice, keystroke/touchstroke dynamics, and gait, into identity embeddings for user authentication. While designed to encode identity, these embeddings may inadvertently reveal sensitive demographic attributes, including gender, age, and height. Consequently, an adversary with access to the authentication model can infer demographic information from biometric signals, including those of users unseen during training or enrollment. In this paper, we present the first systematic audit of demographic leakage in BBA systems, evaluating 11 models across 9 datasets spanning four biometric modalities. We further benchmark four post-hoc suppression methods-Incremental Variable Elimination (IVE), Hilbert-Schmidt Independence Criterion (HSIC), Adversarial Encoder-Decoder (AED), and Protected Attribute Suppression System (PASS) to assess their ability to mitigate demographic leakage while preserving authentication utility. Our analysis reveals substantial variation in leakage and suppressibility across modalities, model architectures, and learning objectives. While voice embeddings exhibit high leakage that can be effectively suppressed, keystroke/touchstroke embeddings exhibit lower leakage but are considerably more difficult to sanitize. These findings highlight a fundamental privacy risk in behavioral biometric authentication and provide insights into the factors governing demographic information suppressibility.

0.29

VIPBench: A Human-Aligned Benchmark for Voice Identity Perception in the Age of Voice Cloning

cs.SD cs.AIcs.CYcs.MM ·2026-11-03 ·Xudong Tang, Matthew Groh
publishedbenchmarkcode/data similarity 0.68
Abstract · Accepted at NeurIPS 2026, Track on Evaluations and Datasets (poster). 30 pages, 12 figures, 22 tables. Dataset and code: https://huggingface…

Speaker-recognition benchmarks test how accurately a model recovers the speaker label of a recording, treating voice identity as a property of the person who produced it. However, these benchmarks do not evaluate how humans perceive voice identity. Whether a listener hears two voices as the same person has consequences for intellectual property rights, cybersecurity, and everyday life. We introduce the Voice Identity Perception benchmark (VIPBench), which contains 124,876 same-or-different judgments from 1,290 listeners on 9,800 recording pairs from 100 public figures. The pairs include real recordings, voice clones generated by a text-to-speech system, and morphs between two speakers' voices. We also define four tasks that evaluate the alignment between speaker-representation models and human identity perception. The first two tasks measure how well a model's similarity score predicts the proportion of listeners who respond "same" and the majority response. The other two measure whether a model and listeners find the same speaker pairs similar, and whether a model's relationship to listener judgments holds equally for real and synthetic recordings. We evaluate eleven publicly available speech models. Listener judgments diverge from speaker labels most often for voice clones: in 41% of responses, listeners judged a clone and a real recording of the same speaker to be different people. No model reaches the noise ceiling; the best model explains 61% of the explainable variance. Models trained with speaker labels align with listeners better than self-supervised models do, yet a model with a 12% error rate on speaker labels aligns with listeners as well as models with error rates below 2%. VIPBench enables the evaluation of speaker models against human perception on real, cloned, and morphed speech, and our results motivate training speaker models directly on listener judgments.

0.36

The Unseen Threat: How Complementary Combinations of Acoustic Features Compromise Speech Privacy

eess.AS ·2026-11-12 ·Jule Pohlhausen, Jörg Bitzer
top venue similarity 0.66
Abstract · Accepted at SPSC 2026 - 6th Symposium on Security and Privacy in Speech Communication

The interaction of acoustic features in leaking private information remains underexplored. This work focuses on time-domain features, i.e., broadband RMS level and pitch, alone and combined with temporally smoothed and subsampled Mel spectrograms. Privacy is assessed using informed automatic speech recognition and speaker-verification attacks, while utility for conversational analysis is evaluated using voice activity detection and speaker diarisation. RMS level and pitch alone exposed limited information, but fusion with 500 ms-smoothed Mel spectrograms enhanced utility while substantially increasing speech-content leakage. Specifically, the combination with RMS level showed that fine-grained temporal dynamics can amplify leakage from coarsely sampled spectral features. Speaker-identity protection remained limited for all tested features. Overall, these findings indicate that privacy assessments performed on individual acoustic features may underestimate leakage after feature fusion.

0.24

Rethinking Automated Voice Similarity by Shifting from EER to Embedding Geometry

eess.AS cs.SD ·2026-09-27 ·Szu-Chi Chen, Jia-Kai Dong, Yi-Cheng Lin, Sung-Feng Huang, Hung-yi Lee
similarity 0.66
Abstract · 5 pages. Submitted to ICASSP 2027

Speaker verification (SV) models are commonly assumed to better capture nuances among speaker characteristics as verification accuracy improves, leading to their widespread use as automated proxies for human voice similarity in speech generation tasks. However, by establishing a human perceptual alignment metric and conducting systematic analysis, we demonstrate that perceptual alignment is governed far more by how a model is trained (its learning objective) than by how well it performs (EER). Notably, standard margin-based classification losses (e.g., AAM-Softmax) yield substantially lower perceptual alignment than prototypical metric losses, while EER itself fails to track human judgment, directly challenging the community's implicit assumption. We trace this divergence to embedding geometry, where a model's effective dimensionality ($d_{\mathrm{eff}}$) tracks perceptual alignment with a $-0.95$ rank correlation, revealing that the dimensional spread favored by classification losses fundamentally clashes with the low-dimensional nature of human voice perception. Imposing a dimensionality bottleneck compresses $d_{\mathrm{eff}}$ and raises perceptual alignment ($ρ_{\mathrm{align}}$) from 0.08 to 0.74, establishing a principled geometric criterion for evaluating voice similarity.

0.33

Character Identity is not Speaker Identity: KyaraBench and KyaraEmbed for Character Verification

eess.AS cs.SD ·2026-10-05 ·Joonyong Park, Jerry Li
benchmark similarity 0.64
Abstract

A dubbed character keeps its identity while the voice actor changes, so character identity and speaker identity are distinct properties of one recording, yet speaker verification measures only the latter. To address this gap, we propose KyaraBench, a benchmark that scores character voice directly instead of speaker voice, built from 85 human-audited identities in a dubbed anime corpus. It poses two challenging conditions: one that swaps the performer under a fixed character, and one that fixes the performer under changing characters. Listening studies with 78 participants provide human reference scores for cross-performer verification and same-actor discrimination. Speaker-verification baselines show increased errors under these character-specific conditions. We then train KyaraEmbed, a compact encoder using multilingual character supervision, same-actor negatives, and a language-alignment term. The model achieves the best performance on all character-specific conditions in the main comparison. We release the benchmark, protocol, and encoder publicly.

0.19

Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correction

eess.AS cs.CL ·2026-08-23 ·Hermann Yepdjio Nkouanga, Minwei Luo, Maggie Wigness, Suresh Singh
top venue similarity 0.64
Abstract · Accepted to INTERSPEECH 2026

While cascaded multi-talker ASR (MT-ASR) leverages state-of-the-art foundation models, its performance is often capped by speaker leakage during separation. Prior correction strategies primarily focus on lexical re-labeling for speaker attribution. We propose a complementary pruning-based paradigm that robustly identifies and removes leakage artifacts. Our method utilizes a pre-trained speaker diarization model as a multimodal verifier to prune transcribed segments satisfying a tripartite consensus of temporal containment, lexical cross-validation, and temporal alignment. Results on LibriMix, LibriSpeechMix, and the AMI Meeting corpus show our algorithm consistently reduces cpW ER across diverse overlap conditions. Specifically, on subsets with high speaker leakage, our method achieves relative cpW ER reductions of up to 29%, highlighting its effectiveness in enhancing the reliability of cascaded MT-ASR transcripts in complex acoustic environments.

0.29

A Unified Uncertainty-Aware Back-End for Speaker Verification: Scoring, Normalization, and Calibration

cs.SD ·2026-09-01 ·Junjie Li, Kong Aik Lee
similarity 0.64
Abstract

Speaker verification back-ends commonly combine similarity scoring, score normalization, and calibration. However, speaker embeddings extracted from real-world utterances have trial-dependent reliability because of factors such as duration, noise, and channel variation. Existing uncertainty-aware methods primarily improve the speaker encoder or the initial similarity score, while the estimated uncertainty is typically not propagated through subsequent normalization and calibration. We represent each utterance by a speaker embedding, interpreted as a posterior mean, together with its covariance as an uncertainty estimate. We present a unified uncertainty-aware back-end comprising uncertainty-aware cosine scoring, uncertainty-aware AS-Norm (UAS-Norm), and uncertainty-aware Quality Measure Function calibration (UQMF). Covariance information is incorporated throughout this pipeline to adjust score scaling, cohort statistics, normalized-score combination, and calibration features. Experiments with ECAPA-TDNN and ResNet show consistent EER reductions and improved target--non-target separation across both architectures.

0.39

Reducing Speaker Residual by Considering Pinhole Effect in Voice Anonymization

eess.AS ·2026-09-20 ·Zeyan Liu, Weili Jiang, Liping Chen, Kong Aik Lee, Boyu Zhao +2 more
top venue similarity 0.63
Abstract · Accepted to INTERSPEECH 2026

Voice anonymization aims to protect privacy by suppressing speaker identity while preserving linguistic content and prosody. However, residual speaker attributes in non-identity representations may still increase linkability and weaken privacy protection. To this end, this paper proposes a fine-tuning strategy with a pinhole loss for well-trained voice anonymization frameworks to further reduce residual speaker attributes. Inspired by the pinhole effect, the pinhole loss measures the linkability of anonymized utterances from the same source speaker. By minimizing this loss, linkability is reduced, thereby improving privacy protection. Experiments on multiple anonymization frameworks, pseudo-speaker generation methods, and datasets show improved privacy protection while maintaining utility. Audio samples can be found in https://anonymous.4open.science/r/Pinhole-loss-fine-tunning-4628.

0.31