CS Paper Radar

Unveil the Mask: Jointly-Optimized Diffusion Masked Pretraining for Appearance-Agnostic Operator Verification

Katsuya Ogata, Ryoya Yoshiyama, Kawachi Hodaka, Yuta Nakashima

cs.CV ·submitted 2026-10-26 13:25 UTC · v1
benchmark

Abstract

Photorealistic avatar videos decouple the appearance of a rendered face from the person who operates it, which makes conventional face recognition unsuitable for identifying the operator. Verifying the operator therefore requires representations of how a face moves that are insensitive to what the avatar looks like. We propose a verification framework built without any external pretrained model. It uses two complementary modalities provided for each sample: facial landmark sequences and video frames. From the landmark modality, it learns motion representations by diffusion masked pretraining over grouped landmarks, guided by lightweight auxiliary tasks that predict behavioral cues such as motion intensity and eye openness, and turns them into operator embeddings with a contrastive linear probe. To generalize to unseen avatar generators, the pretraining input is randomly transformed to simulate how a generator may alter the operator's motion. From the video modality, it captures fine facial motion through differences between consecutive frames. The embeddings of the two modalities are combined into one. Our approach reaches an overall AUC of 0.769 on the WBA2026 benchmark. Linear probing shows that the video embedding retains little information about the avatar's identity, whereas the landmark embedding retains some, and fusing the two modalities improves verification in particular across speaking and listening modes.

Comment
Accepted at ACCV 2026 Workshop on Who Is Behind the Avatar? (WBA)

Closest papers

Nearest neighbours by meaning across the whole index.

Video Diffusion Avatars: Taming Video Diffusion Models into Controllable 3D Avatars

cs.CV ·2026-10-16 ·Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen +1 more
benchmark similarity 0.74
Abstract · Project webiste: https://simongiebenhain.github.io/video-diffusion-avatars/

Video diffusion models (VDMs) trained on internet-scale data encode a rich prior on appearance, motion, and physics that is difficult to replicate with explicit 3D rendering and costly 3D data capture. We introduce Video Diffusion Avatars (VDAs), a method that creates photo-realistic, multi-view consistent avatars from a single image by fine-tuning a VDM for precise control over camera viewpoint, facial expression, and upper-body pose. Departing from 3D morphable models (3DMMs), VDAs use a learned expression space that lifts the expressiveness ceiling of 3DMMs, e.g., for the tongue and mouth interior. However, learned expression codes exhibit heavy viewpoint bias, as our DISTRACT benchmark reveals. We show that this bias can be removed through disentanglement, enabling their use as driving signals for multi-view consistent avatars. Camera and upper-body pose are controlled jointly via proxy-mesh renderings, giving independent control over expression, pose, and viewpoint. A camera-triplet training strategy makes the approach efficient: our models train for 34 hours on a single A100 using only public multi-view data. VDAs outperform state-of-the-art single-image 3D avatars on novel-view reenactment, reducing face LPIPS by 29% and improving perceptual quality (JOD) by 66% on BecomingLit. A static 3DGS reconstruction probe further confirms that our generations are the most multi-view consistent. Built on a video prior, VDAs produce effects beyond the reach of explicit renderers, including physically plausible hair dynamics, shadows, reflections, and refractions.

0.22

DIME: Scaling Facial Representation Learning via Differential Masked Autoencoding

cs.CV ·2026-10-12 ·Hao Yu, Haoyu Chen, Hui Wei, Yan Jiang, Nicu Sebe +1 more
similarity 0.72
Abstract

Scaling up data and model capacity has driven substantial progress in visual representation learning. However, conventional pretext tasks exhibit diminishing returns on fine-grained facial analysis. We attribute this to a scaling bottleneck: these tasks model each image's appearance and are dominated by identity, pose, and illumination, so subtle within-identity facial differences contribute little to the training signal. We introduce DIME, a differential masked autoencoder for large-scale facial representation learning. DIME builds on a simple observation: two images of the same person share identity-specific appearance, so their difference exposes facial changes. DIME reconstructs both masked images in an identity-matched pair and, through an Explicit Differential Direction Supervision (EDDS) objective, aligns the direction of their reconstructed difference with the true difference, while source isolation makes each reconstruction depend only on its own image. Shared appearance cancels in this difference, and because EDDS ignores magnitude, subtle facial changes count as much as large pose or illumination changes. DIME uses identity labels only to form pretraining pairs, requires no dense annotations, and learns dense facial representations with emergent part-level semantics. We pretrain DIME on 20 million high-resolution facial images consolidated from existing face datasets. DIME's AU detection performance continues to improve as pretraining data scales from 6M to 20M images, with its BP4D F1 increasing by 5.9 points, while MAE gains only 0.5 points on the same subsets. DIME also outperforms specialized facial methods on AU detection, AU intensity estimation, cross-domain anti-spoofing, landmark localization, head pose estimation, face parsing, and face generation.

0.21

BEACON: Behavior and Appearance Control for Subject-Specific Video Generation

cs.CV cs.AI ·2026-09-07 ·Pokrzywa Baptiste, Nabyl Quignon, Yara Bahram, Muhammad Osama Zeeshan, Antitza Dantcheva +1 more
similarity 0.71
Abstract

Generating human-centric videos that preserve both visual identity and person-specific expressive behavior remains a fundamental challenge. In addition to reproducing appearance, a model must replicate the facial behaviors that characterize how a subject expresses emotion over time. However, most state-of-the-art methods condition generation on a single reference image, which contains no information about these temporal dynamics. As a result, they tend to preserve the subject's visual identity but often produce expressions with limited variation and weak subject specificity. To mitigate this issue, we introduce BEACON, a lightweight framework for person-specific video generation that produces more expressive videos by disentangling visual identity from expressive behavior. BEACON conditions generation on two complementary signals: a reference image encoding the identity and a reference video capturing subject-specific facial dynamics. By conditioning on these complementary signals, BEACON generates videos that better preserve both the subject's appearance and characteristic facial dynamics, while also supporting identity-expression transfer. Our experiments on the MEAD and RAVDESS datasets show that by fine-tuning on approximately 2,000 pairs and updating about 1% of the pretrained Wan video diffusion model, BEACON improves facial expressivity over state-of-the-art video generation methods while maintaining competitive identity preservation.

0.27

IDPrior: Decoupling Identity Generation from Facial Image Synthesis

cs.CV ·2026-11-05 ·Fadi Boutros, Meriç Demirörs, Darian Tomašević, Vitomir Štruc, Naser Damer
publishedcode/data similarity 0.70
Abstract · Accepted at WACV 2027

Large-scale identity-labeled face datasets are essential for training modern face recognition models, yet collecting and annotating such data is costly and raises privacy and ethical concerns. Existing synthetic face generation methods typically derive identities from predefined class labels, unconditional image generation, or predefined latent sampling strategies, without explicitly modeling the underlying distribution of facial identities. In this paper, we propose IDPrior, a two-stage framework that decouples identity generation from image synthesis. We first learn a diffusion prior in the embedding space of a pretrained face recognition model to model the distribution of facial identities and generate novel identity embeddings. These embeddings are then used to condition a pretrained identity-conditioned diffusion model (DM), enabling the synthesis of multiple images of the same identity. This modular design provides a scalable approach to identity generation while remaining compatible with existing and future identity-conditioned DMs. Experiments indicate that IDPrior generates diverse and well-separated identities while maintaining a closer correspondence to the distribution of authentic identities. Face recognition models trained on datasets generated using IDPrior-generated identities achieve competitive or improved performance compared with existing synthetic-data-based methods across multiple face verification benchmarks. Code and generated datasets are publicly available at https://github.com/fdbtrs/IDperturb.

0.38

ReXMo: End-to-End Video Character Replacement with Expression and Motion Preservation

cs.CV ·2026-11-03 ·Hualiang Wei, Wenhui Li
similarity 0.70
Abstract · Preprint

Video character replacement aims to replace actors while preserving facial expressions, motions, and scene dynamics. Existing methods typically rely on masked generation with pose guidance, limiting fine-grained expression and motion preservation in complex scenes. We propose ReXMo, an end-to-end framework that supports single-, multi-, and selective character replacement scenarios. ReXMo introduces paired triplet supervision, where synthesized driving videos and original videos share identical performances but different identities, enabling scalable training through synthetic data generation. An In-Context Diffusion Transformer learns identity transformation by jointly conditioning on driving videos, reference identities, and text prompts, while an Expression Enhancement Adapter improves facial expression preservation in multi-person scenes. Extensive experiments demonstrate that ReXMo outperforms existing methods in identity fidelity, motion consistency, expression preservation, and visual quality. Our dataset and code will be publicly released.

0.24

Learning Unified Video and Image Representation for Video Face Forgery Detection

cs.CV ·2026-08-13 ·Haotian Liu, Yang Liu, Guoying Zhao, Xiaobai Li
benchmarkcode/data similarity 0.68
Abstract

Face forgery detection is crucial for preserving the security and integrity of facial data given the rapid developments in face manipulation techniques and deep generative models. Existing methods for video face forgery detection typically assume that all frames in a forged video are manipulated, while detecting partially forged videos that contain only a subset of altered frames remains challenging. To address this issue, we propose a novel framework, UVIF, that utilizes additional annotated images to provide fine-grained supervision for detecting partial forgeries in videos. UVIF employs a unified encoder and a multi-task learning paradigm to jointly model facial videos and images for boosted video face forgery detection. A 2D backbone with temporal fusion modules is employed as the unified encoder. A pseudo labeling process is designed for video frames to bridge their representations with those of static images. A video-oriented feature alignment strategy is further introduced to reduce the distribution gap between videos and images. Extensive experiments on benchmark datasets demonstrate the effectiveness of our framework, which outperforms state-of-theart methods in detecting partially forged videos while introducing no additional computational overhead. Our code is available at https://github.com/haotianll/UVIF.

0.24

Beyond Ambiguous Visual Cues: Studying Physiological Disruptions and Cross-Modal Inconsistencies in Deepfake Videos

cs.CV ·2026-09-11 ·Chenxi Yang, Yassine Ouzar, Larbi Boubchir
similarity 0.68
Abstract

Recent deepfake detection studies increasingly suggest remote photoplethysmography (rPPG) signals as an authenticity cue. However, existing benchmarks lack physiological ground truth, and current detectors underexplore the cross-level relationship between facial features and physiological dynamics, often relying on late fusion or rPPG features alone. In this paper, we construct high-fidelity deepfake manipulations on established real rPPG datasets (COHFACE and UBFC-rPPG) to investigate how forgeries disrupt natural physiological signals and facial behavior at the same time. Building on this analysis, we propose a bidirectional co-attention fusion detector that jointly models rPPG and facial behavior tokens. This mechanism explicitly captures the cross-level dependencies between pulse dynamics and facial motion to learn a robust, joint authenticity representation. Extensive experiments using a subject-disjoint 5-fold evaluation demonstrate the superiority of our approach. Achieving a 92.80\% AUC on constructed datasets using face swapping and 96.78\% AUC on motion transfer, our model outperforms both the rPPG-only single modality baseline and the best feature-level fusion methods. Furthermore, transfer-learning result of the fusion detector on Celeb-DF-v2 while keeping both feature extractors fixed achieves 91.20\% accuracy and 86.08\% AUC, which suggests applicability under target-domain adaptation.

0.19

Photorealistic Novel View Synthesis of Human Faces using Next-Scale Transformers

cs.CV cs.LG ·2026-08-24 ·Federico Stella, Fei Jiang, Zhongshi Jiang, Zohar Barzelay, Emanuel Garbin +2 more
similarity 0.67
Abstract

Photorealistic novel view synthesis of people remains challenging at high spatial resolutions and across multiple target cameras, where preserving identity, fine appearance details, and geometric coherence is critical. We build on the next-scale autoregressive paradigm and adapt it for human-centric view synthesis by enabling higher image resolutions, multi-view outputs and stronger cross-view consistency in a single forward pass. We train on a synthetic dataset of human faces spanning diverse identities and apparel. Contrary to diffusion models, this paradigm does not need 2D pre-training and, thanks to its next-scale architecture, it benefits from lower-resolution, general-purpose pre-trainings, with the full-sized purpose-specific images being used only in the last training stages. This enables our architecture to converge with a smaller amount of purpose-specific training data, allowing us to use a smaller but more realistic training dataset. The resulting model produces sharp and realistic views, with the option to synthesize multiple novel viewpoints simultaneously for improved agreement across views. Empirically, we observe gains in perceptual fidelity and cross-view coherence on human subjects, demonstrating that next-scale autoregression is an effective backbone for scalable, multi-output human view synthesis. We also couple our pipeline with an existing transformer-based model for pixel-aligned 3D gaussian lifting from multi-view facial inputs, resulting in accurate and photorealistic 3D models of human faces.

0.17