CS Paper Radar

Long-Tail Adaptive Flow Matching with Explicit Conditional Consistency Guidance for Precise Multimodal Face Synthesis

Yushe Cao, Xuechao Zou, Xing Xi, Dianxi Shi, Chun Yu, Junliang Xing

cs.CV ·submitted 2026-09-14 06:35 UTC · v1

Abstract

Although diffusion-based methods have substantially improved the controllability of multimodal face synthesis, their semantic alignment remains suboptimal because most existing approaches rely on implicit latent-space objectives to model the relationship between denoising variables and multimodal conditions. Such implicit modeling is often insufficient to enforce precise correspondence between synthesized faces and conditional inputs, especially under long-tailed semantic mask distributions where rare attributes receive weak optimization signals. To address these limitations, we propose EC\textsuperscript{2}Face, a multimodal face synthesis framework that improves semantic alignment through explicit semantic supervision and distribution-aware optimization. First, we introduce Explicit Conditional Consistency Guidance (ECCG), which imposes direct consistency supervision in pixel space by decoding an approximate reverse estimate of the clean latent and explicitly aligning the synthesized image with textual descriptions and semantic masks. A temporal dynamic modulation function is further designed to adapt the supervision strength according to the timestep-dependent reliability of reverse estimation. Second, we propose Long-Tail Adaptive Flow Matching (LAFM), which reweights spatial optimization signals based on semantic attribute frequency, with normalized weights to maintain numerical stability during training. Importantly, all additional modules are used only during training and introduce no extra inference overhead. Extensive experiments show that EC\textsuperscript{2}Face consistently outperforms competitive baselines in both generation quality and semantic alignment, achieving a 29.38\% improvement in mask accuracy on rare attributes.

Comment
14 pages, 12 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Identity-Conditioned Latent Consistency Distillation for Face Synthesis

cs.CV ·2026-08-31 ·Tiago Kienen Chaves, Bernardo Biesseck, David Menotti
publishedcode/data similarity 0.68
Abstract · Accepted for presentation at the 2026 Conference on Graphics, Patterns and Images (SIBGRAPI)

Diffusion models have achieved strong results in high-fidelity image synthesis, but their iterative sampling process makes large-scale generation computationally expensive. This limitation is especially relevant when generating synthetic face datasets for face recognition, where a large number of subjects with many samples in different poses, expressions, ages, etc., are required. In this work, we show that identity-conditioned face synthesis can be performed at a substantially lower computational cost by a latent Consistency Model with few iterations, without compromising image quality. For training, we distill knowledge from the foundation Diffusion Model Arc2Face (teacher) by adapting its original text-to-image pipeline to an embedding-to-face setting, replacing textual prompts with ArcFace identity embeddings. Our distilled model (student) generates identity-conditioned face images with an average inference time of 0.4819 seconds per image, compared with 2.102 seconds for Arc2Face, resulting in a 4.36$\times$ speed-up. Quantitative results, based on FID scores, show that the distilled model remains competitive with Arc2Face across all evaluation protocols. On 100k generated images, it achieves near-parity on CelebA (13.921 vs. 12.928) and outperforms the teacher on WebFace42M (9.317 vs. 9.802). Further evaluations on Synth-500 and AgeDB show a moderate performance gap for the former but comparable results for the latter. These results indicate that Arc2Face can be accelerated through task-specific latent consistency distillation while preserving high image quality for large-scale synthetic face generation. Our proposal is publicly available at https://github.com/UFPR-IPASP-PR/FaceRec-IdentityConsistency.

0.33

FMA-Net: Flexible Modality Adaptation for Robust Multimodal Face Presentation Attack Detection

cs.CV ·2026-11-09 ·Yuhao Gao, Yichen Shi, Zitong Yu, Hongyang Wang, Hongrui Li +3 more
top venue similarity 0.67
Abstract

Multimodal face presentation attack detection (fPAD) improves robustness using complementary RGB, depth, and infrared (IR) cues, but deployment remains challenging when modalities are unavailable or incomplete. Existing flexible-modality methods mainly use zero-filling or modality omission, potentially discarding cross-modal information and introducing modality-dependent bias. We propose the Flexible Modality Adaptation Network (FMA-Net), a unified framework that compensates for missing modalities through instruction-guided latent-space translation. The Face Modality Instruct Translator (FMIT) estimates target modality observations from RGB inputs, while context-aware category enhancement uses learnable text prompts as modality-invariant semantic anchors. A Facial Feature Adapter (F2A) integrates global facial structure and local spoof-relevant cues to enhance discriminative representations. Synthetic-to-Real Face Domain Alignment (SR-FDA) reduces feature discrepancies between translated and real modalities, with benefits depending on the missing-modality pattern and data domain. Experiments on WMCA and PADISI-Face validate the framework. On WMCA with a 70% modality missing ratio, FMA-Net achieves a mean ACER of 3.22% across four intra-domain missing-modality protocols, compared with 4.24% for the strongest baseline by the same criterion. When both depth and IR are entirely unavailable, it achieves 5.23% ACER, compared with 5.99% for the best competing result. Cross-domain experiments also reveal limitations of synthetic-to-real alignment under dataset shifts. These results support FMA-Net as a practical approach to fPAD with flexible and incomplete modalities.

0.34

From Short Clips to Real-Time Streams: Identity-Aware Supervision and Context-Efficient Generation for Video Face Swapping

cs.CV ·2026-10-19 ·Zekai Luo, Yuwei Liang, Ruiqi Ma, Haoyu Huang, Xuehao Wang +6 more
similarity 0.66
Abstract · Project webpage: https://aim-uofa.github.io/StreamingSwap/

Video face swapping transfers the reference identity to a target video while preserving its other attributes and motion. Despite recent progress in reference-guided flow-based methods, jointly achieving faithful identity transfer, high fidelity, and real-time generation over long video streams remains challenging. To address this challenge, we rethink the task-specific structure of reference-guided video face swapping. First, face swapping changes only identity while preserving most visual content, yet flow-matching supervision is weakly sensitive to these changes. We therefore develop Joint Rec-ID Loss, which balances explicit identity supervision and latent reconstruction according to the noise level. Second, the target video already provides non-edited attributes and temporal dynamics, enabling generation with fewer spatial tokens and shorter temporal contexts through Face-RoI Generation and Short-Context Rollout Training. By integrating these designs, we propose StreamingSwap, a reference-guided flow-matching framework for real-time streaming video face swapping. We also introduce HumanFaceVid, comprising 1,000 hours of cinematic training data and evaluation sets of 200 short and 200 long videos. Experiments demonstrate that our method achieves competitive identity transfer and visual quality at 20 FPS on a single NVIDIA RTX 4090 GPU, with stable generation over videos up to 10 minutes long. Project webpage: https://aim-uofa.github.io/StreamingSwap/

0.32

A Generative AI Integrated Multimodal Framework for Low-Latency Multi-Camera Person Re-Identification

cs.CV cs.AIeess.IV ·2026-09-13 ·Leon Fernando, C Dombawala, P. Hettigoda, Vanodhya G. Warnasooriya, Ishara Neranjana +1 more
top venue similarity 0.66
Abstract · Presented at 2026 IEEE International Conference on Responsible Artificial Intelligence (IRAI)

Person re-identification (ReID) is essential for multi-camera surveillance and tracking, yet remains difficult due to viewpoint and illumination changes, occlusion, background clutter, and low resolution imagery. We propose a generative AI integrated multimodal ReID framework designed explicitly for robustness under missing cues and low latency deployment. The key idea is a cost aware early-exit cascade that prioritizes inexpensive, high confidence evidence and only triggers expensive modalities for ambiguous cases. Our system integrates (i) global visual embeddings from segmented person regions, (ii) automatically generated fine grained semantic attribute descriptions generated by vision-language models (VLMs), and (iii) optional facial embeddings when face observations are reliable. To optimize the balance between accuracy and latency, we use a cost aware early-exit cascade instead of fusing all modalities. Specifically, we first inspect the top-k retrieval results to determine whether the query is unambiguous. If the best match is clearly separated from the remaining candidates, we stop early and return the result to minimize latency; in ambiguous cases, we keep multiple hypotheses and invoke additional modalities (face/semantic) with adaptive reliability weighting to refine the decision. We report person re-identification performance using mAP and Rank-1 accuracy on the Market-1501 and DukeMTMC-reID benchmarks. The proposed adaptive early-exit cascade resolves 60.7% of DukeMTMC-reID queries and 68.4% of Market-1501 queries without invoking semantic reasoning, reducing computational overhead while maintaining competitive retrieval performance.

0.29

RoAR-VAE: Improving Latent Diffusability through Robust Prediction

cs.CV ·2026-10-19 ·Yang Yue, Fangyun Wei, Yitong Wang, Jinjing Zhao, Henry Zheng +2 more
benchmarkcode/datacommunity pick similarity 0.64
Abstract · Code is available at https://github.com/LeapLabTHU/RoAR_VAE

An effective visual tokenizer must preserve image fidelity while producing latents that downstream diffusion models can learn efficiently. Prior work improves latent diffusability through semantic alignment or joint tokenizer--diffusion training. Revisiting REPA-E, a representative approach, we find that removing its diffusion objective and dedicated generative blocks while retaining semantic prediction from corrupted latents yields comparable downstream generation performance under a matched setting. This observation motivates a general principle of robust prediction: generation-relevant information should remain recoverable from corrupted latents. We instantiate this principle with two complementary objectives. RobustAlign predicts pretrained semantic features from corrupted latents to preserve high-level structure, while RobustRecon reconstructs the clean image to retain fine-grained spatial and appearance information. Together, they form Robust Alignment and Reconstruction VAE (RoAR-VAE). Our analysis suggests that robust prediction improves diffusability by making useful information more accessible in high-variance latent directions, which diffusion models tend to learn earlier. Empirically, RoAR-VAE improves average generation benchmark performance by 2.19 points over the strongest baseline tokenizer under the same training budget, while improving small-text reconstruction accuracy by 16.93 percentage points over FLUX.1-VAE. These results demonstrate that RoAR-VAE achieves a favorable balance between reconstruction fidelity and downstream generative learnability.

0.33

Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioning

cs.CV cs.SDeess.AS ·2026-10-01 ·Gunwoo Lee, Yoori Oh, Yoseob Han
publishedcode/data similarity 0.64
Abstract · Accepted to BMVC 2026

Video-to-speech synthesis aims to generate natural-sounding speech from silent talking-face videos while ensuring phonetic accuracy. A fundamental challenge in this task is the inherent one-to-many mapping problem, where visual dynamics often lack sufficient information to uniquely determine the corresponding utterance. To address this, we propose Watch Your Speech (WYS), a video-to-speech synthesis framework that incorporates textual conditioning as an explicit linguistic cue to mitigate visual ambiguity. Our framework features an attention-based embedding fusion module that synergistically integrates textual context with video sequences, coupled with a conditional flow matching objective for high-fidelity speech generation. Extensive experiments on the LRS2 and LRS3 datasets demonstrate that WYS achieves superior performance, establishing new state-of-the-art results in audio-visual synchronization (LSE-C/D) while maintaining highly competitive textual accuracy (WER). Subjective evaluations further confirm that our model generates speech with near-human naturalness, validating the effectiveness of textual conditioning in content-controlled video-to-speech synthesis. Project page: https://github.com/gunwoo5034/Watch-your-Speech

0.41

TIB: Sample-wise Tempered Information Bottleneck for Multimodal Attribution beyond Alignment Assumption

cs.CV cs.AIcs.MM ·2026-11-08 ·Qiming Huang, Peixi Liu, Jianbo Jiao
published similarity 0.64
Abstract · Accpeted to NeurIPS2026

Multimodal (e.g. vision-language, in this work) attribution aims to interpret the models without access to ground-truth explanatory supervision. Existing multimodal attribution methods typically rely on paired modalities as proxy supervision, implicitly assuming that image-text pairs are semantically aligned. Information bottleneck (IB)-based attribution methods follow this general paradigm and instantiate it through a global sufficiency-compression objective. In realistic multimodal datasets, however, this assumption is frequently violated due to partial semantic misalignment and unreliable cross-modal correspondences. Through theoretical analysis, we show that, under such misalignment, information-bottleneck-based attribution objectives exhibit a structural limitation: by assuming equal reliability across all proxy supervision signals, they induce posterior over-contraction and forced explanations. To address this issue, we propose TIB, a sample-wise Tempered Information Bottleneck framework that adaptively modulates attribution strength according to cross-modal reliability, without assuming semantic alignment. Extensive experiments on controlled benchmarks and large-scale datasets show the effectiveness of the proposed approach, while also validating the identified failure mode.

0.28

RAGMesh with FaME-G2E: Long-Form Text-Driven 3D Face Generation and Editing

cs.CV ·2026-08-10 ·Hao Li, Ju Dai, Feng Zhou, Mengting Shi, Haofei Wang +4 more
similarity 0.64
Abstract

Text-driven 3D face generation and editing remains challenging due to the difficulty of translating long-form descriptions into fine-grained facial geometry. Existing methods primarily align global textual semantics with facial structures but often struggle to capture subtle local deformations, such as eyebrow tension, cheek contraction, and asymmetric mouth motions, resulting in limited geometric fidelity and editing precision. To facilitate fine-grained text-driven facial modeling, we first construct FaME-G2E, a large-scale multimodal dataset containing detailed text--mesh annotations and paired text--blendshape samples for unified 3D facial generation and editing. Based on this dataset, we propose RAGMesh, a retrieval-augmented framework that leverages text-correlated geometric priors to improve high-fidelity facial synthesis and editing. Specifically, the Multi-Scale Retrieval Fusion (MSRF) module retrieves semantically consistent global and regional facial priors and fuses them in the blendshape space, suppressing conflicting local deformations while preserving coherent deformation patterns. Furthermore, we introduce Adaptive RAG-guided Supervision (AdaRAGS), a region-aware constraint that explicitly aligns textual semantics with corresponding facial regions, enhancing regional controllability and editing accuracy. Extensive experiments on FaME-G2E demonstrate that RAGMesh achieves superior performance over state-of-the-art methods in local geometric accuracy, text-guided controllability, regional editing precision, and inference efficiency. Video demo is available at https://youtu.be/Yr0_XkpWcNk, and the source code and dataset will be released upon paper acceptance.

0.14