CS Paper Radar

EgoHRV: Continuous Heart Rate Variability Estimation from Egocentric Systems for Autonomic Response and Skill Assessment

Berken Utku Demirel, Christian Holz

cs.CV eess.SP ·submitted 2026-08-19 09:10 UTC · v1
top venuecode/data

Abstract

Egocentric vision systems capture human behavior from visible cues, but overlook physiological indicators of autonomic states such as stress, engagement, and attention. Heart rate variability (HRV) is a widely used noninvasive marker of autonomic regulation under stress. HRV reflects small timing differences between successive heartbeats and has so far been out of reach for egocentric platforms, where motion and noise in gaze video mask exactly this fine-grained timing. We propose EgoHRV, a method that estimates HRV as well as heart rate (HR) from the gaze cameras that are already integrated into egocentric headsets. Our pipeline combines a 3D backbone with a novel low--high decomposition module that extracts the blood volume pulse (BVP) signal from gaze video. Our cross-domain pretraining aligns the frequency-domain representations of contact-based and camera-derived signals. This alignment gives EgoHRV the temporal precision to recover HRV from the subtle fluctuations in gaze video. EgoHRV achieves state-of-the-art accuracy for HR and HRV estimation from egocentric video, and its uncertainty-aware design improves downstream behavioral modeling. Integrating our HRV estimates and confidence measures into EgoExo4D's proficiency estimator raises accuracy by 17.8%. Beyond skill, continuous HRV estimation also opens egocentric systems to stress- and arousal-aware estimation tasks. Code: https://github.com/eth-siplab/EgoHRV

Comment
Accepted to the European Conference on Computer Vision (ECCV) 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Adapting Appearance-Based Gaze Estimation to Narrow-Range, Long-Duration Screen Viewing

eess.IV ·2026-10-08 ·Jordan Prescott, Kleanthis Avramidis, Shrikanth Narayanan
benchmark similarity 0.66
Abstract

Appearance-based gaze estimation offers a low-cost alternative to infrared eye tracking for screen-based behavioral and clinical applications, however existing models are typically developed for wide ranges of gaze angle and head pose. Prolonged screen viewing presents a distinct regime in which gaze remains near the screen center, head motion is limited, and calibration drift accumulates over time. In this work, we benchmark six published estimators along with a proposed differential-gaze model on 140 long-duration facial video recordings with synchronized eye tracking under subject-disjoint evaluation and a common budget of calibration frames. Differential estimation was the only static method that significantly improved upon a baseline predictor of each recording's mean gaze, and was further boosted by addition of temporal context. It also yielded the strongest agreement with reference fixations and saccades, demonstrating that low angular error alone could not validate eye movement reconstruction. These findings establish differential estimation as a promising foundation for reliable gaze tracking in long-duration, narrow-range settings.

0.07

EgoHMP: Achieving Precise Human Motion Prediction in 3D Scenes via Egocentric Cues

cs.CV ·2026-11-10 ·Xin Zhao, Pengzhan Zhou, Ziyi Li, Riheng Jia, Zhida Qin +1 more
top venue similarity 0.66
Abstract · Accepted at NeurIPS 2026 (Main Conference)

The complex interactive dependencies between human behavior and 3D scenes make accurately predicting human intent a major challenge for human motion prediction (HMP). While recent studies use human gaze to infer intent, gaze signals suffer from ambiguity, drift, and a reliance on eye-tracking hardware. Research in cognitive science indicates that egocentric views inherently encode potential interactions, while head motion serves as a crucial precursor in motion planning. Motivated by this, we propose EgoHMP, a novel framework that uses egocentric views and head motions as robust carriers of interaction intent. By translating this intent into an envisioning of future interactions, EgoHMP achieves precise HMP in 3D scenes. EgoHMP first employs a vision-motion alignment encoder to align the visual and motion feature spaces. Based on these aligned features, an egocentric intent predictor with a modality-aware modulation mechanism adaptively suppresses invalid interaction cues to predict an accurate interaction probability map. Furthermore, to enhance the stability and realism of the generated motions, we propose a contact-aware motion decoder. Specifically, inspired by the human cognitive process of planning trajectories prior to action execution, we first predict a global trajectory to serve as a self-prompt, effectively mitigating the optimization conflict in diffusion models. Subsequently, we introduce a multi-stage denoiser, which relieves motion artifacts by incorporating human contact modeling through a multi-stage optimization strategy. Finally, a GCN-based motion decoder is employed to synthesize physically plausible and semantically consistent motions. Extensive experiments demonstrate that EgoHMP achieves state-of-the-art performance.

0.23

OpenSAL360: Open-Source Crowdsourcing Platform for Omnidirectional Video Saliency Collection

cs.CV ·2026-09-18 ·Alexey Bryncev, Andrey Moskalenko, Kira Shilovskaya, Ivan Kosmynin, Dmitriy Vatolin
top venuedatasetcode/data similarity 0.65
Abstract · Accepted by ACM MM 2026

Omnidirectional video saliency prediction plays an important role in many immersive multimedia applications, including viewport-adaptive streaming and compression, foveated rendering, mesh simplification, perceptual quality assessment. Yet progress in this area remains constrained by the cost and complexity of collecting eye-tracking data with VR headsets, which makes large-scale dataset creation difficult to extend. We present OpenSAL360, the first open-source platform for scalable, low-cost 360° video saliency collection. Unlike conventional VR-based protocols, it requires only a standard screen, mouse, and internet connection, enabling parallel saliency data collection from common crowdsourcing assessors without specialized hardware. We validate our collection protocol against seven well-established VR eye-tracking datasets and conduct ablation studies on key interface, pre-, and post-processing parameters. To demonstrate the effectiveness and scalability of the proposed methodology, we collect and publicly release a saliency dataset covering 500 omnidirectional videos annotated by 2,000+ crowdsourcing assessors, making it, to the best of our knowledge, the largest dataset in this field. We make OpenSAL360 publicly available at https://github.com/msu-video-group/OpenSAL360.

0.61

EyeTAG: Eye Trajectory-Aware Gaze Estimation

cs.CV cs.LG ·2026-10-01 ·Jungmin Lee, Niamat Ullah, Yoseob Han
publishedcode/data similarity 0.64
Abstract · Accepted to BMVC 2026

Gaze estimation under natural head-eye motion underpins applications from driver monitoring to human-computer interaction. Single-frame methods predict each frame independently, so consecutive outputs fluctuate as jitter. Multi-frame methods reduce this, but they learn motion implicitly inside appearance features, so the gaze trajectory is never an explicit variable. We propose EyeTAG (Eye Trajectory-Aware Gaze Estimation), a causal multi-frame framework built around an explicit first-order gaze prior: at each step it differentiates its own recent predictions and feeds the resulting trajectory back as a compact kinematic token. Because differencing is translation-invariant in gaze space, this token carries subject-invariant motion rather than personal gaze offsets. Face and eye streams supply visual evidence, fused by cross-attention and a causal Transformer decoder. EyeTAG reduces the mean angular error by about 1.0$^\circ$ on Gaze360 and performs on par with the strongest baseline on EVE (2.56$^\circ$ vs. 2.58$^\circ$). Within-model ablations, which keep the encoder and the rest of the architecture fixed and vary only the gaze history, show that the differential formulation, rather than temporal context alone, removes the systematic saccade bias that persists even with an absolute gaze-history prior. Our code is available at https://github.com/peter8366/EyeTAG.

0.38

GazeAgent: VLM-Guided Implicit Gaze Calibration from Egocentric Multimodal Behavior for AI Glasses

cs.HC cs.CV ·2026-10-17 ·Yuhong Zhang, Khanh An Nguyen, Xinyu Tian, KungYi Chang, Abraham Padua +4 more
similarity 0.64
Abstract · 27 pages, 14 figures, 3 tables

AI glasses typically require explicit, user-specific eye-tracker calibration, and accuracy can drift when the device shifts during use. We present GazeAgent, a VLM-guided framework that continuously and adaptively improves gaze-calibration accuracy from naturally occurring behavior without asking users to interrupt their activities. GazeAgent identifies precise visual attention (PVA) events from multimodal streams of inertial measurement unit (IMU), gaze, and wearer-hand signals, validates their task relevance, and localizes functional targets as pseudo-calibration targets. In a 10-participant scripted study, the system detected all 820 predefined PVA opportunities and, under device disturbance, reduced median gaze error by 44.1% compared with explicit calibration. In a five-participant case study, autonomous calibration produced the lowest workload and interruption ratings, and all participants preferred it for routine use. By deriving task-critical targets from natural multimodal behavior, GazeAgent demonstrates the potential for continuous, adaptive, and low-burden gaze interaction on egocentric AI glasses.

0.08

From Laboratory to Road: Evaluating Wearable Gaze Accuracy for Driving

cs.CV ·2026-10-06 ·William Engel, Fabian Flohr
similarity 0.64
Abstract · Peer-reviewed and accepted as an Extended Abstract at the German Conference on Pattern Recognition (GCPR 2026). Presented as a poster at GCP…

Bird's-eye-view (BEV) representations have become a widely used interface between perception and planning in autonomous driving, but they encode what is in a scene, not what is behaviorally relevant to a human driver. Gaze offers a compelling behavioral signal for this gap, yet wearable eye trackers are routinely deployed as if their spatial output were ground truth, despite known sensitivity to head motion, illumination, and calibration drift. We present, to our knowledge, the first unified framework for quantifying wearable gaze accuracy under real driving conditions. Our on-road study contains 41 validated scenes in which one driver fixated a vehicle's license plate. Gaze error is measured as the angular difference between the plate center and the gaze direction estimated by the glasses. Separate indoor studies with the same driver and device systematically analyze how distance, illumination, head motion, target motion, and gaze eccentricity affect both systematic bias and gaze precision. The mean on-road error was 4.58 degrees. Applying an offset estimated from the indoor recordings reduced it to 1.10 degrees and improved all 41 scenes. Because this offset varied between sessions, reliable BEV supervision may require online recalibration and condition-dependent estimates of gaze uncertainty.

0.15

EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input

cs.CV cs.AI ·2026-08-16 ·Matteo Stoiber, Niels Buus Lassen
published similarity 0.64
Abstract · 16 pages. Accepted at the WearableAI Workshop, ECCV 2026 (Archival Track)

The use of multimodal LLMs (MLLMs) for egocentric video understanding with wearable devices is constrained by the token budget. Memory and compute cost scale with the number of visual tokens, and high-resolution video quickly becomes expensive to transmit and process at scale. Prior work (GazeLLM) addresses this by cropping the video around the camera wearer's gaze. This reduces the number of visual tokens by about tenfold while maintaining or improving the quality of full-resolution descriptions. However, this compression strategy depends on dedicated eye-tracking hardware, which is unavailable on consumer smart glasses. Building a software-only substitute poses a joint constraint: the predictor must be accurate enough to preserve downstream description quality, yet light enough to run on-device, within the power and compute budget of a smartphone. We address this with EgoGazeLite, a lightweight dual-process gaze predictor for egocentric video. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops show no significant difference from ground-truth-gaze crops. Equivalence is confirmed in all ten cases. EgoGazeLite achieves this at 15.7M parameters, 6.71 GFLOPs, and runs the full gaze-and-crop pipeline end-to-end in real time (21.6 ms/frame) on consumer accelerator hardware. Together, these results remove the need for eye-tracking hardware for token-efficient, gaze-conditioned egocentric video understanding with MLLMs.

0.26

Tracking Is Not Sensing: Artifact-Aware Thermal Video Respiration Estimation in Head-Free Marmosets

cs.CV ·2026-10-24 ·Penghao Wang, Sen Yang, Fenghua Xie, He Wang, Xindong Song
published similarity 0.63
Abstract · WACV 2027

Continuous physiological monitoring is essential for linking behavior, internal state, and neural activity in neuroscience and translational research. In common marmosets, a key nonhuman primate model for social neuroscience, respiratory rate provides a basic readout of arousal, stress, and physiological regulation. Contact sensors can provide accurate measurements, but they add preparation burden and may interfere with head-free behavioral protocols. Thermal video offers a noncontact alternative by capturing breathing-related temperature changes around the nose and mouth. However, in head-free marmosets, these cues are weak, localized, and easily corrupted by head motion, pose changes, and fur. Here we show that this setting exposes a broader challenge in video-based physiological sensing: visually reliable tracking does not necessarily imply physiologically valid measurement. Using synchronized thermal video and contact respiration recordings, we show that marmoset respiration is recoverable from thermal video but strongly ROI-sensitive. Under head-free motion, visually plausible nasal tracking can still produce traces inconsistent with contact respiration, demonstrating that tracking is not sensing. We therefore estimate respiratory rate from learned temporal signals extracted from thermal clips and introduce Selective Temporal Attenuation Regularization (STAR). STAR selectively attenuates high-motion thermal discontinuities through local temporal imputation during training, regularizing models toward respiration-consistent dynamics while leaving inference unchanged on raw video. Across multiple deep-learning backbones, STAR improves head-free estimation, reaching 2.23 bpm MAE compared with 20.76 bpm for tracked ROI analysis.

0.29