CS Paper Radar

DiffSSMEye: Conditional High-Frequency Eye Movement Synthesis with Structured State-Space Diffusion

Kamrul Hasan, Mehedi Hasan Raju, Oleg V. Komogortsev

cs.HC ·submitted 2026-11-12 16:07 UTC · v1

Abstract

Synthesizing high-frequency eye movements requires preserving long-range temporal dependencies and subject-specific oculomotor dynamics. However, methods based on generative adversarial networks (GANs) and variational autoencoders (VAEs) can struggle to preserve these characteristics simultaneously. We introduce DiffSSMEye, a conditional denoising diffusion probabilistic model with bidirectional Structured State Space Sequence (S4) layers for generating five-second gaze-velocity sequences at 1,000 Hz. Generation is conditioned on a coarse gaze reference, user identity, and task, where identity and task embeddings are incorporated through feature-wise linear modulation. Identity-preservation, multi-resolution spectral, and integrated-drift losses complement noise prediction to constrain biometric consistency, frequency structure, and displacement error. Subject-disjoint evaluation on GazeBase across seven tasks yields mean real--synthetic embedding cosine similarities of 0.91--0.95. DiffSSMEye also achieves the lowest median spatial errors among these baselines for horizontal and random saccade tasks. These results demonstrate improved spatial accuracy and biometric feature consistency for subject-specific gaze synthesis.

Comment
14 pages, 5 tables, 2 figures

Closest papers

Nearest neighbours by meaning across the whole index.

EyeMakeYou: Identity-, Task-, and Subjective-State-Conditioned Diffusion for High-Frequency Gaze Synthesis

cs.HC cs.CV ·2026-09-03 ·Kamrul Hasan, Mehedi Hasan Raju, Oleg V. Komogortsev
similarity 0.76
Abstract · 20 pages, 3 tables, 3 figures

Eye movement biometrics (EMB) is an emerging behavioral modality for user authentication, particularly in virtual- and augmented-reality systems, where gaze dynamics contain distinctive subject-specific features. However, robust EMB systems require diverse, high-quality gaze recordings that are expensive to collect and often unavailable at the scale needed for model development. Generative models can mitigate data scarcity, but existing methods either synthesize generic gaze behavior or personalize signals primarily by identity, without jointly representing the user's task and subjective state. Consequently, generated signals may appear visually realistic while failing to retain the behavioral properties required for biometric applications. To address this limitation, we propose EyeMakeYou, a multi-conditional denoising diffusion framework for subject-specific, high-frequency gaze synthesis. EyeMakeYou generates 5-s, 1000-Hz bivariate gaze-velocity sequences from an identity-removed reference trajectory and conditions the denoising process on an identity embedding, a task embedding, and self-reported ratings of overall difficulty, mental tiredness, and eye tiredness. Its objective combines diffusion noise prediction and identity preservation with multi-resolution spectral, drift-consistency, and event-weighted local-smoothness losses. Experiments on GazeBase show that EyeMakeYou achieves higher median spatial accuracy and greater real--synthetic similarity in the embedding feature space than the existing generative approaches, while retaining selected task-dependent associations between subjective reports and oculomotor features. These findings support conditional diffusion as a practical approach for augmenting gaze datasets for biometric and interactive applications.

0.48

Diffusion models for eye-gaze trajectory generation using position and velocity representations

cs.CV cs.AIcs.NE ·2026-09-01 ·Laxman Basnet, Alexander Szorkovszky, Pedro G. Lind, Anis Yazidi, Shailendra Bhandari
similarity 0.72
Abstract · 13 pages, 9 figures

Eye-tracking data are expensive to collect, requiring specialized hardware and controlled laboratory conditions, and difficult to share due to privacy regulations, which together limit the quantity and diversity of gaze data available for research. We investigate unconditional gaze generation using diffusion models in position and component-wise velocity representations. The models are trained and evaluated on eight-second visual-search trajectories from 28 participants, using a split of 23 training participants and five held-out validation participants, with three training seeds. Evaluation includes kinematic distributions, trajectory geometry, event statistics, and a temporal Wasserstein GAN with gradient penalty as a learned comparator. Across eight position-derived features, mean Jensen-Shannon divergence is $0.016 \pm 0.004$ for the position model and $0.037 \pm 0.002$ for the velocity model, with standard deviations calculated across training seeds. These scores do not establish that representation choice caused the difference. A separate comparison using the same evaluation method shows that the ordering of the models varies across training seeds. The position model also produces smaller feature discrepancies than a kinematic Markov baseline. Differences in saccade counts between real and generated data depend strongly on the velocity threshold used to detect events. The results provide quantitative reference measurements for unconditional gaze generation on this visual-search task and show how evaluation choices affect conclusions about generated gaze.

0.24

GazeFlow: From Human Gaze Behavior to Generative Egocentric Gaze Prediction

cs.CV cs.LG ·2026-09-29 ·Sheng Zhao, Weikai Lin, Yuhao Zhu
top venue similarity 0.67
Abstract · Accepted at NeurIPS 2026. 25 pages

Egocentric gaze prediction enables many downstream applications but remains challenging, as human gaze is inherently stochastic. This stochasticity is constrained by structured temporal dynamics alternating between fixations and saccades, top-down influences from tasks, and bottom-up visual saliency. Based on this observation, we introduce GazeFlow, a framework that directly models gaze as a joint distribution of temporal gaze positions conditioned upon both top-down and bottom-up information. In particular, GazeFlow uses conditional flow matching (CFM): a learned velocity field iteratively transports a Gaussian noise sample into a plausible gaze trajectory drawn from this joint distribution. The velocity field is conditioned on bottom-up visual features extracted by a video encoder and on top-down task information obtained by globally querying these features. On standard datasets, GazeFlow achieves state-of-the-art performance on per-frame metrics, and the generated trajectories align better with human gaze temporal dynamics.

0.47

EyeDiT: Anatomy-Grounded Multi-Modal Eye Generation and Simulation

cs.CV ·2026-11-18 ·Yonghan Lee, Zachary Willms, Kun Liu, Pushkar Anand, Ali Behrooz +3 more
code/data similarity 0.65
Abstract

Synthetic eye data are critical for rapid development of eye tracking (ET) in AR/VR applications. However, existing synthetic eye generators are largely image-centric and lack a high-quality underlying 3D anatomy of the eye, which limits their control mainly to the gaze and provides no anatomical labels. To address this problem, we propose EyeDiT, an anatomy-grounded, multi-modal 3D eye generation and simulation model trained with optical coherence tomography (OCT), labeled anatomy point clouds and near-eye images. Our pipeline first trains a Multi-Modal Anatomy Variational Autoencoder (VAE), which encodes an anatomy-aware latent representation from all input modalities. Then, our Eye Latent DiT is trained to generate latent samples conditioned on the optical axis, the pupil position and radius, and the subject identity. Together with Gaussian decoders for novel-view simulation, our model enables control over the pupil, subject identity and viewpoint for data synthesis. We show that our model outperforms previous eye image synthesis models in novel-view synthesis (24.9 against 21.8 dB PSNR from a single view), and that a public ET neural network trained on our synthetic data shows a smaller sim-to-real gap than when trained on data from other generators (3.5$^\circ$ vs. 8.4$^\circ$ gaze error), approaches a network trained on real data, and further improves ET performance when combined with real data (2.3$^\circ$ vs. 2.6$^\circ$). More information is available on the project page: https://lyhan12.github.io/eyedit

0.35

Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modeling

cs.CV ·2026-08-24 ·Virmarie Maquiling, Zhuojiang Cai, Enkelejda Kasneci
benchmark similarity 0.64
Abstract · 8 pages, 1 figure, 1 table

Gaze is increasingly used as an input signal for vision and multimodal models, yet no consensus exists on how to represent it across datasets. Raw traces preserve detail but are noisy and device-dependent, while coarse event labels are easy to model but can discard local motion structure. We formulate event-aligned, fixed-horizon angular displacement as an interpretable, event-conditioned motion vocabulary and compare it with event-only, spatial, absolute-angle, learned vector-quantized, and continuous representations. To assess transfer alongside target predictability and token collapse, our evaluation combines next-token prediction with target-domain regret, low-order target references, paired bootstrap, order sensitivity, motif overlap, and frozen structural probes. In an event-aligned headset benchmark, angular-motion tokens have lower target-domain regret than frozen-codebook VQ tokens in one transfer direction, while the reverse direction is inconclusive. The probes reveal complementary representation properties, and event-only tokens show that low perplexity can retain little motion information. On a third egocentric dataset, a matched comparison of I-VT, native, and frame-span interfaces shows that event construction materially changes transfer: native events have the lowest regret into EGTEA, while frame-span events have zero motif overlap and fail severely as a source. Motion-based tokenization therefore provides a compact representation for event-aligned egocentric gaze streams, while the evaluation identifies how target predictability and event construction shape cross-dataset conclusions.

0.51

Unveil the Mask: Jointly-Optimized Diffusion Masked Pretraining for Appearance-Agnostic Operator Verification

cs.CV ·2026-10-26 ·Katsuya Ogata, Ryoya Yoshiyama, Kawachi Hodaka, Yuta Nakashima
benchmark similarity 0.64
Abstract · Accepted at ACCV 2026 Workshop on Who Is Behind the Avatar? (WBA)

Photorealistic avatar videos decouple the appearance of a rendered face from the person who operates it, which makes conventional face recognition unsuitable for identifying the operator. Verifying the operator therefore requires representations of how a face moves that are insensitive to what the avatar looks like. We propose a verification framework built without any external pretrained model. It uses two complementary modalities provided for each sample: facial landmark sequences and video frames. From the landmark modality, it learns motion representations by diffusion masked pretraining over grouped landmarks, guided by lightweight auxiliary tasks that predict behavioral cues such as motion intensity and eye openness, and turns them into operator embeddings with a contrastive linear probe. To generalize to unseen avatar generators, the pretraining input is randomly transformed to simulate how a generator may alter the operator's motion. From the video modality, it captures fine facial motion through differences between consecutive frames. The embeddings of the two modalities are combined into one. Our approach reaches an overall AUC of 0.769 on the WBA2026 benchmark. Linear probing shows that the video embedding retains little information about the avatar's identity, whereas the landmark embedding retains some, and fusing the two modalities improves verification in particular across speaking and listening modes.

0.15

Disentangling Spurious Correlations in Vision-Language-Action Models via Predicting Domain-Invariant Latent Lookahead

cs.RO cs.LG ·2026-09-29 ·Junghyun Kim, Ngseo Kim, ChungWoo Lee, Seoyeon Lee, Woo-Jeong Baek +5 more
publishedcode/datahot direction similarity 0.62
Abstract · Accepted to CoRL 2026. Project website: https://dill-vla.github.io/

Vision-Language-Action (VLA) models remain brittle under visual distribution shifts, often relying on spurious correlations tied to domain-specific factors rather than task-relevant structure. We propose Domain-Invariant Latent Lookahead (DILL), a representation-learning framework that mitigates shortcut learning in VLA policies. Our key idea is to supervise policies with domain-invariant future latents learned from domain-transformed trajectory data. A Task-Domain Encoder is trained with contrastive objectives and Gaussian disentanglement regularization to separate task-relevant structure from domain-specific visual variation. The learned encoder then provides future latents for VLA policy learning through lookahead prediction and domain disentanglement, encouraging the policy to focus on task-relevant structure rather than incidental visual factors. Counterfactual task-view evaluations show that DILL reduces shortcut reliance, while LIBERO-Plus evaluations demonstrate improved visual robustness, with 69.1% average success, 11.4 percentage points above the strongest baseline. Real-world manipulation experiments further support DILL's applicability beyond controlled simulation. Complementary latent-space diagnostics show that these behavioral gains are accompanied by representations that better preserve task-consistent structure while suppressing domain-specific variation. Our project page is available at https://dill-vla.github.io/.

0.47

Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation

cs.CV ·2026-08-24 ·Chengqun Yang, Liang Xu, Yanping Li, Fulong Liu, Jingnan Gao +2 more
publishedcode/data similarity 0.62
Abstract · Accepted by ICME 2026

Text-driven human motion synthesis has made substantial development with two core modules of motion representation and generative architecture. For representation, Vector Quantization (VQ)-based methods compress motion data into discrete tokens while latent-based models operate directly in continuous space. However, both of these representations exhibit significant limitations. VQ-based methods suffer from inherent information loss, which compromises the quality, diversity, and generalization of generated motions, while continuous representation on holistic whole-body motion hinders part-level flexibility. For architecture, diffusion and autoregressive diffusion models have demonstrated their superiority, yet the fine-grained controllability over individual body parts is also limited. Thus, we propose a unified spatiotemporally decoupled framework named DeMoDiff, which jointly redesigns representation and architecture. To enhance representation extraction capabilities and offer greater part-level controllability, we present a spatial-temporal VAE that encodes each body joint rather than compressing the whole-body motion into a single latent space. Then, we incorporate spatial-temporal masking and attention mechanisms into an autoregressive diffusion generator, achieving both generative capability and controllable editability. Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that our model achieves state-of-the-art reconstruction performance and compelling motion generation results. Moreover, our framework demonstrates strong temporal and spatial editing capabilities, further validating its effectiveness. Our project page: https://rex0191.github.io/DeMoDiff/

0.38