CS Paper Radar

Audio-Driven Adversarial Defense for 3D Talking Face Generation with totally Visual Fidelity Preservation

Rui-Qing Sun, Chen-Hao Cui, Hui-Yang Zhao, Tian Lan, Zhijing Wu, Xian-Ling Mao

cs.CV cs.MM ·submitted 2026-08-31 15:22 UTC · v1

Abstract

The rapid development of generative portrait models has raised growing concerns about privacy leakage and identity misuse. In particular, audio-driven 3D talking face generation can reconstruct a reusable 3D portrait of a target person from a monocular video and animate it with arbitrary speech, making realistic identity impersonation alarmingly practical. Existing proactive defenses mainly operate in the visual domain by injecting subtle perturbations into acial regions to disrupt identity acquisition. However, such perturbations often compromise visual quality due to the strong structural priors and social sensitivity of human faces, and are easily weakened by common real-world transformations such as resizing. To overcome these limitations, we propose an imperceptible audio defense for audio-driven 3D talking face generation by shifting protection from the visual modality to the audio modality. Specifically,we exploit psychoacoustic masking to hide protective perturbations within perceptually masked frequency regions of the speech signal, thereby reducing perceptual distortion while suppressing reliable facial animation. Extensive experiments demonstrate that the proposed method effectively degrades 3D talking face generation while preserving favorable perceptual quality. These findings highlight psychoacoustically guided audio perturbations as a practical and promising direction for privacy-preserving portrait protection.

Closest papers

Nearest neighbours by meaning across the whole index.

MUTE: Multi-Level Alignment Uncoupling Against Talking-Head Exploitation for Voice Protection

cs.SD cs.CVcs.LG ·2026-11-09 ·Donghyun Kim, Jin Hong, Seungmin Kim, Dain Kim, Junseok Kwon +1 more
top venuecode/data similarity 0.72
Abstract · Accepted at NeurIPS 2026. 33 pages, including appendix and checklist. Project and code: https://github.com/jindl465/MUTE

Talking-head generation models, which synthesize realistic facial animations from audio, are increasingly vulnerable to misuse in multimodal deepfake scenarios. Protecting such systems remains challenging, as talking-head models fundamentally rely on precise audio-visual alignment, while effective audio protection methods remain largely underexplored. In this work, we propose MUTE, an audio protection framework that uncouples the underlying audio-visual alignment through a multi-level strategy. Specifically, MUTE combines (1) Alignment-level Disruption, which directly perturbs cross-modal attention to disrupt structural alignment, and (2) Representation-level Degradation, which perturbs temporal audio embeddings to degrade their representations. To improve robustness, perturbations are constrained in the STFT domain to high-energy regions, making them resistant to post-processing, denoising, and adaptive attacks. An optional speaker-level objective further mitigates potential bypass via TTS-based resynthesis. Extensive experiments demonstrate that MUTE consistently degrades lip synchronization across both white-box and black-box talking-head models while preserving perceptual audio quality. The proposed method remains effective under real-world transformations and can be combined with image-based approaches to provide stronger multimodal protection. The source code and project page are available at https://github.com/jindl465/MUTE.

0.34

AudioLift: Audio-Driven Single Image to Full-Head Gaussian Avatar via Multi-view Video Generation

cs.CV ·2026-10-22 ·Zhuoyuan Jiang, Weijie Lyu, Xiaoyuan Wang, Tiancheng Shen, Ming-Hsuan Yang
code/data similarity 0.70
Abstract · Project page: https://zhuoyuanjiang.github.io/audiolift-page/

Existing 3D talking avatar methods mainly rely on parametric face models including 3DMM or FLAME, whose compact motion representations limit their ability to synthesize extreme expressions and to model accessories such as headwear. We introduce AudioLift, a two-stage framework that uses generated multi-view videos as the intermediate representation for audio-driven avatar generation without parametric face models or per-instance optimization. Our framework combines a synchronized multi-view video generation stage with an inconsistency-robust Gaussian reconstruction stage to recover coherent full-head geometry from a single portrait image. Extensive experiments demonstrate that AudioLift produces 3D talking avatars capable of capturing more extreme facial expressions than existing methods, with improved geometric completeness and strong multi-view consistency across the full $360^\circ$ viewing range.

0.16

Silence-the-Mimic: Accelerating Imperceptible Perturbation Generation Against Voice Cloning

eess.AS ·2026-09-30 ·Runqiu Xu
published similarity 0.70
Abstract · Accepted at IEEE SLT 2026

Deep neural network-based Voice Conversion (VC) and Text-to-Speech (TTS) models have rapidly advanced, enabling realistic voice cloning with minimal input data. Such capabilities raise serious concerns over unauthorized cloning of speaker identities and the associated privacy and security risks. Current imperceptible adversarial protection methods rely on quality control losses that are highly sensitive to hyperparameter tuning and computationally expensive due to lengthy optimization. To address these limitations, we propose a fast protection method that generates perceptually constrained perturbations in the frequency domain under a psychoacoustic masking-based constraint. Our approach strictly enforces perceptibility bounds during adversarial training, eliminating the need for iterative quality balancing and significantly reducing computational cost. Experiments on multiple state-of-the-art VC and TTS models show that STM achieves competitive or superior protection performance with substantially better perceptual quality and up to $45.3\times$ speedup over existing white-box baselines. These results demonstrate the effectiveness of frequency-domain perturbations with perceptual constraints as a practical paradigm for protecting against voice cloning.

0.27

Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation

cs.CV ·2026-09-29 ·Bangxun Tang
code/data similarity 0.69
Abstract · 19 pages, 8 figures, 5 tables. Under review

We present RGOR (Reference-Grounded Oral Refinement), an audio-driven lip-sync framework that renders the mouth of the specific person being dubbed rather than a generic one. Existing lip-sync systems follow the audio closely and keep the face recognizable, yet the mouth they render is an average mouth: the shape and texture of the lips, the arrangement of the teeth, and how much of them shows as the mouth opens are not that person's. The problem persists because nothing in current training or evaluation asks for the person's own mouth: perceptual losses accept any plausible mouth, face identity is carried mostly by the skin around it, and the released inference code of inpainting systems uses the unmasked target frame as the reference, which hides the gap. To address this, RGOR conditions every generated frame on frames from separate enrollment recordings of the same person and on HD patches of the mouth that bypass the VAE, and trains the generator against a paired judge that compares each rendered mouth with the person's reference and learns to reject a realistic mouth of someone else. We further build an evaluation protocol and use it to compare open-source and commercial lip-sync systems on held-out identities. Experiments show that RGOR achieves the best or second-best result on most metrics, and preserves the person's own lip and dental detail while keeping synchronization and the rest of the face intact.

0.26

No Way To Steal My Face: Proactive Defense Against Identity-Preserving Personalized Generation

cs.CV cs.CR ·2026-10-14 ·Lizhi Xiong, Jun Li, Ziqiang Li, Weiwei Jiang, Zhangjie Fu
top venuecode/data similarity 0.66
Abstract

Recent advances in diffusion models have enabled high-fidelity, identity-preserving image generation for personalized applications such as digital avatars and virtual try-on systems. However, their reliance on sensitive facial reference images raises growing privacy concerns. Existing defense mechanisms are primarily designed for training-based personalization and struggle to generalize to emerging training-free approaches, due to fundamental differences in their identity integration paradigms. To bridge this gap, we propose IDGuardian-the first generalizable and model-agnostic identity protection framework capable of defending against both training-based and training-free personalization methods. IDGuardian abstracts the personalization process into two critical stages: identity extraction and identity injection. It then introduces crafted adversarial perturbations to simultaneously disrupt both stages. Specifically, it degrades the identity features extracted by external encoders and establishes an adversarial conceptual bridge that misdirects the generative trajectory away from the target identity. Extensive experiments show that IDGuardian effectively protects identity across various personalization pipelines and model architectures, while remaining robust to post-processing, adaptive attacks, and cross-dataset generalization. Our code is available at https://github.com/OpenAscent-L/IDGuardian.git.

0.68

KVeil: First-Place Solution to ECCV 2026 LifeGenIP Challenge on Unlearnable Videos

cs.CV ·2026-10-09 ·Boheng Li, Yueming Xing, Yiming Li, Jie Zhang, Han Qiu +1 more
publishedcode/data similarity 0.66
Abstract · Technical report for the winning solution (1st place, both phases) of the ECCV 2026 LifeGenIP Challenge on Unlearnable Videos against Diffus…

The rapid development of personalized video generation has raised serious concerns over identity privacy, as a few facial videos of a person now suffice to reproduce their identity in synthetic videos. This report presents the first-place solutions to both phases of the ECCV 2026 LifeGenIP Challenge on Unlearnable Videos, which aims to protect facial videos from unauthorized customization under a limited and imperceptible perturbation budget. For Phase 1, we introduce Reference Capacity Collapse, a pair of complementary losses that degrade the information carried by the reference condition while keeping subsequent video representations dependent on it, together with a Motion-Guided EOT mechanism that improves robustness across unknown inference conditions. Phase 2 further considers tuning-based customization and unseen generative models. To address both threats within a single perturbation, we design a two-layer framework that protects the facial and background regions separately, constructing structure-aligned perturbations from a synthetic target identity in the former and strengthening the white-box disruption of Phase 1 in the latter. Our method achieves nearly perfect protection under white-box reference-based customization, and remains effective under other customization paradigms, various models, and temporal attacks. Both solutions are built upon our recent work LETHE. We hope our solutions can inspire future work on protecting facial videos against unauthorized customization. Our code is available at https://github.com/AntigoneRandy/KVeil.

0.46

DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation

cs.SD cs.AI ·2026-08-10 ·Wei Zhou, Wanyi Ning, Yinshang Guo, Qianxiao Fang, Haitao Qian +1 more
similarity 0.66
Abstract

Audio-visual speech enhancement under real-world conditions remains challenging due to unreliable visual inputs and the lack of large-scale training data with realistic acoustic conditions. Existing approaches usually fuse visual features directly into the separation network, making them vulnerable to degraded visual signals. In this paper, we present DAVE, a decoupled audio-visual enhancement framework for real-world speech separation. Firstly, to address the data scarcity issue, we construct DAVE-Corpus, a large-scale training corpus with 219,411 mixtures generated from public meeting corpora through combinatorial acoustic augmentation. Then, we introduce a progressive multi-objective optimization strategy to jointly improve speech separation, intelligibility, speaker identity preservation, and perceptual quality. We further develop a certified selective enhancement chain that applies scene routing, GAN-based denoising, and loudness normalization only within the no-reference partition, guaranteeing non-degradation of reference-based metrics. Experimental results on the Real-World Audio-Visual Speech Enhancement Challenge demonstrate the robustness of DAVE under both real-world mixed scenarios and visual degradation conditions.

0.04

Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling

cs.CV eess.AS ·2026-10-14 ·Jing-Xuan Zhang, Genshun Wan, Jianqing Gao, Jia Pan, Yumei Zhang +1 more
top venuebenchmarkcode/data similarity 0.65
Abstract · accepted to NeurIPS 2026

Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this work, we propose FLARE, a framework for robust visual speech representation learning via cross-modal distillation and joint face-lip modeling. FLARE leverages semantically rich Whisper representations as the acoustic teacher signal, converting them into discrete distillation targets through a random-projection quantizer and pretraining the model with a masked prediction objective. To capture complementary visual information, FLARE employs a dual-stream architecture that jointly processes lip region-of-interest (ROI) and full-face inputs, where the lip stream captures finegrained articulatory motion and the face stream encodes broader facial dynamics. During pretraining, auxiliary audio input is incorporated to facilitate cross-modal alignment, and modality dropout is applied to encourage robust visual-only representations. For downstream lipreading, FLARE serves as a visual encoder coupled with either a lightweight Transformer decoder or an LLM-based decoder. Evaluated on LRS2, LRS3, and the out-of-domain WildVSR benchmark, FLARE achieves strong lipreading performance, with word error rates of 11.7%, 13.2%, and 34.3%, respectively. A progressive design study confirms the contribution of each major component. Further analyses demonstrate that joint face-lip modeling improves robustness under degraded video resolution and large head pose variation, while yielding more phoneme-discriminative visual representations. Code and pretrained models are available at https://github.com/jxzhanggg/FLARE.

0.45