CS Paper Radar

Silence-the-Mimic: Accelerating Imperceptible Perturbation Generation Against Voice Cloning

Runqiu Xu

eess.AS ·submitted 2026-09-30 20:00 UTC · v1
published

Abstract

Deep neural network-based Voice Conversion (VC) and Text-to-Speech (TTS) models have rapidly advanced, enabling realistic voice cloning with minimal input data. Such capabilities raise serious concerns over unauthorized cloning of speaker identities and the associated privacy and security risks. Current imperceptible adversarial protection methods rely on quality control losses that are highly sensitive to hyperparameter tuning and computationally expensive due to lengthy optimization. To address these limitations, we propose a fast protection method that generates perceptually constrained perturbations in the frequency domain under a psychoacoustic masking-based constraint. Our approach strictly enforces perceptibility bounds during adversarial training, eliminating the need for iterative quality balancing and significantly reducing computational cost. Experiments on multiple state-of-the-art VC and TTS models show that STM achieves competitive or superior protection performance with substantially better perceptual quality and up to $45.3\times$ speedup over existing white-box baselines. These results demonstrate the effectiveness of frequency-domain perturbations with perceptual constraints as a practical paradigm for protecting against voice cloning.

Comment
Accepted at IEEE SLT 2026

Closest papers

Nearest neighbours by meaning across the whole index.

TTS-Guard: Black-Box Ownership Verification of Text-to-Speech Models via Adaptive Adversarial Speaker-Pair Fingerprints

cs.SD ·2026-09-20 ·Xubin Yue, Zhenhua Xu, Zhebo Wang, Mengting Li, Zijie Zhou +3 more
top venue similarity 0.72
Abstract

The rapid maturation of zero-shot Text-to-Speech (TTS) models has turned high-quality voice cloning into a widely available capability, raising acute concerns over unauthorised replication, fine-tuning and resale of proprietary speech models. Yet ownership verification for TTS remains largely open: speech is a continuous waveform whose perturbations are easily destroyed by routine signal processing, and the human auditory system imposes a much tighter perceptual budget than vision. We present \textbf{TTS-Guard}, a black-box ownership verification framework for TTS models built on \emph{adversarial speaker-pair fingerprints}. TTS-Guard(i) selects key speaker pairs in a \emph{dual} embedding space for architecture-agnostic stealth;(ii) optimises a perturbation through an \emph{adaptive curriculum} of shadow models covering fine-tuning, pruning, quantisation and distillation; and (iii) aggregates black-box queries into a calibrated \emph{Verification Confidence Score}. On five mainstream TTS systems, TTS-Guard reaches an average Fingerprint Success Rate of $96.4\%$ at a False Positive Rate of $5.8\%$, while preserving intelligibility and naturalness. The fingerprint remains effective against ten audio attacks, six model modifications, and two state-of-the-art adversarial purifiers.

0.42

Audio-Driven Adversarial Defense for 3D Talking Face Generation with totally Visual Fidelity Preservation

cs.CV cs.MM ·2026-08-31 ·Rui-Qing Sun, Chen-Hao Cui, Hui-Yang Zhao, Tian Lan, Zhijing Wu +1 more
similarity 0.70
Abstract

The rapid development of generative portrait models has raised growing concerns about privacy leakage and identity misuse. In particular, audio-driven 3D talking face generation can reconstruct a reusable 3D portrait of a target person from a monocular video and animate it with arbitrary speech, making realistic identity impersonation alarmingly practical. Existing proactive defenses mainly operate in the visual domain by injecting subtle perturbations into acial regions to disrupt identity acquisition. However, such perturbations often compromise visual quality due to the strong structural priors and social sensitivity of human faces, and are easily weakened by common real-world transformations such as resizing. To overcome these limitations, we propose an imperceptible audio defense for audio-driven 3D talking face generation by shifting protection from the visual modality to the audio modality. Specifically,we exploit psychoacoustic masking to hide protective perturbations within perceptually masked frequency regions of the speech signal, thereby reducing perceptual distortion while suppressing reliable facial animation. Extensive experiments demonstrate that the proposed method effectively degrades 3D talking face generation while preserving favorable perceptual quality. These findings highlight psychoacoustically guided audio perturbations as a practical and promising direction for privacy-preserving portrait protection.

0.15

Detector-Guided Vocoder Training for Adversarial Text-to-Speech Synthesis

cs.SD ·2026-11-03 ·Jiayan Cui, Davide Salvi, Paolo Bestagini, Stefano Tubaro
published similarity 0.69
Abstract · Accepted at WIFS 2026

In this paper, we propose a novel Text-to-Speech (TTS) framework designed to perform adversarial attacks against speech deepfake detection systems. Given a TTS model and a detector capable of classifying its outputs as synthetic, our approach fine-tunes the vocoder by integrating the detector into the TTS training loop. Unlike traditional adversarial methods that rely on post-hoc perturbations or external noise injection, our approach enables the generation of synthetic speech that is intrinsically adversarial, capable of lowering the target detector's reliability without requiring additional post-processing steps. We evaluate the framework in both white-box and black-box settings, showing its effectiveness against known and unseen detectors. We also propose a multi-detector strategy that allows the system to simultaneously learn to deceive multiple detectors, showcasing the scalability of the method. Experimental results confirm the efficacy of our approach, which has the potential to impact forensic security and detection countermeasures. At the same time, subjective listening tests reveal that the proposed method also enhances synthesized speech quality.

0.24

Preemptive defense against re-identification attacks on voice anonymization via adversarial perturbation

eess.AS ·2026-10-05 ·Michele Panariello, Yibo Bai, Massimiliano Todisco, Nicholas Evans
published similarity 0.69
Abstract · Accepted to IEEE Spoken Language Technology (SLT) 2026

Voice anonymization (VA) is used to conceal the voice identities in speech data. Performance is usually estimated using automatic speaker verification (ASV) as a proxy to judge the capability of an attacker to re-identify speakers after anonymization. The defender anonymizes test utterances; the attacker compares them to equally anonymized reference utterances to infer voice identity, with degraded ASV performance indicating successful anonymization. Thus, the defender's protection is one-sided and only applied to test utterances via VA. Though unexplored so far, there is an opportunity to enhance anonymization by protecting reference utterances too. We present a new, preemptive VA paradigm: reference utterances are protected using adversarial noise to degrade their potential to infer voice identity. Preemptive protection does not degrade the perceived quality of reference utterances and is independent of the specific ASV system used for identity inference. By combining preemptive protection and regular test-side VA, ASV equal error rates can be increased from 14\% to 45\% (near-perfect privacy).

0.30

TransferBreaker: Breaking Adversarial Transferability in Fine-Tuned Speech Recognition

cs.LG ·2026-10-06 ·Mojtaba Nafez, Aref Mousavi, Mohammad Ebrahim Mahdavi, Mobina Poulaei, Kiarash Kiani Feriz +1 more
top venuecode/data similarity 0.68
Abstract · 39 pages, 5 figures, 22 tables. Accepted at NeurIPS 2026

Many organizations fine-tune publicly available pretrained Automatic Speech Recognition (ASR) models and deploy them in black-box settings, assuming limited access provides protection. We show this assumption is fragile: adversarial perturbations crafted on the public base model transfer effectively to fine-tuned target models, severely degrading performance and posing concerns for safety-critical applications. We propose TransferBreaker, a unified fine-tuning framework that suppresses adversarial transfer by integrating Base Adversarial Fine-Tuning, which restricts adversarial training to base-effective perturbations; Latent Jacobian Regularization, which enforces latent-space invariance by suppressing adversarially sensitive directions; and HybridGrad-AFT, which improves robustness against adaptive attacks by interpolating transferable perturbations from base and target gradients. We theoretically justify all components and evaluate TransferBreaker across three languages and four large ASR models, reducing adversarial WER from 92.6 to 27.8. Our code is publicly available at https://github.com/rohban-lab/TransferBreaker.

0.48

MUTE: Multi-Level Alignment Uncoupling Against Talking-Head Exploitation for Voice Protection

cs.SD cs.CVcs.LG ·2026-11-09 ·Donghyun Kim, Jin Hong, Seungmin Kim, Dain Kim, Junseok Kwon +1 more
top venuecode/data similarity 0.68
Abstract · Accepted at NeurIPS 2026. 33 pages, including appendix and checklist. Project and code: https://github.com/jindl465/MUTE

Talking-head generation models, which synthesize realistic facial animations from audio, are increasingly vulnerable to misuse in multimodal deepfake scenarios. Protecting such systems remains challenging, as talking-head models fundamentally rely on precise audio-visual alignment, while effective audio protection methods remain largely underexplored. In this work, we propose MUTE, an audio protection framework that uncouples the underlying audio-visual alignment through a multi-level strategy. Specifically, MUTE combines (1) Alignment-level Disruption, which directly perturbs cross-modal attention to disrupt structural alignment, and (2) Representation-level Degradation, which perturbs temporal audio embeddings to degrade their representations. To improve robustness, perturbations are constrained in the STFT domain to high-energy regions, making them resistant to post-processing, denoising, and adaptive attacks. An optional speaker-level objective further mitigates potential bypass via TTS-based resynthesis. Extensive experiments demonstrate that MUTE consistently degrades lip synchronization across both white-box and black-box talking-head models while preserving perceptual audio quality. The proposed method remains effective under real-world transformations and can be combined with image-based approaches to provide stronger multimodal protection. The source code and project page are available at https://github.com/jindl465/MUTE.

0.34

GhostPrefix: Synchronization-Free Universal Prefix Attacks for ASR Manipulation

cs.LG ·2026-10-30 ·Shengding Liu, Ashwin Innuganti, Qiben Yan
code/data similarity 0.67
Abstract

We present GhostPrefix, the first synchronization-free universal adversarial prefix attack against automatic speech recognition (ASR) systems. Unlike prior audio attacks, which are typically crafted per utterance, require synchronization with the victim waveform, or target narrow objectives, GhostPrefix learns a short, reusable audio segment prepended to arbitrary live speech to exert real-time control over ASR outputs without knowledge of the future utterance or temporal alignment. A key challenge is maintaining stable adversarial effects across unknown, evolving speech contexts. GhostPrefix combines token-level losses with a novel representation-level objective for stable, input-agnostic effects. GhostPrefix supports three attack modes: (i) prompted prefix attacks for semantic steering, (ii) targeted prefix attacks for transcript hijacking, and (iii) untargeted prefix attacks for transcription degradation. To remain perceptually inconspicuous, we regularize prefixes toward benign environmental-sound distributions and improve robustness against over-the-air distortions. We evaluate GhostPrefix across 13 open-source ASR models from four families. In digital settings, it achieves average attack success rates of 97.6% for prompted steering and 93.4% for targeted hijacking, while driving word error rate to 1.18 under untargeted attacks. Over the air, it achieves success rates of 91.5% and 89.5% for prompted and targeted modes, respectively. These manipulations propagate into downstream AI-assisted pipelines, including voice assistants and AI note-taking tools, causing systematic semantic misinterpretations. GhostPrefix exposes synchronization-free universal prefixes as a practical, under-explored threat to real-world ASR and downstream applications. Artifacts and demos are available at https://ghostprefix.github.io/.

0.15

Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer

cs.SD cs.AIcs.LGcs.MM ·2026-08-16 ·Ivan Mikheev, Viacheslav Vasilev, Anna Dmitrienko, Alexey Letunovskiy, Ivan Kirillov +2 more
benchmark similarity 0.67
Abstract

Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.

0.04