CS Paper Radar

GazeAgent: VLM-Guided Implicit Gaze Calibration from Egocentric Multimodal Behavior for AI Glasses

Yuhong Zhang, Khanh An Nguyen, Xinyu Tian, KungYi Chang, Abraham Padua, Yuchen Xu, Ying Choon Wu, Gert Cauwenberghs, Tzyy-Ping Jung

cs.HC cs.CV ·submitted 2026-10-17 01:31 UTC · v1

Abstract

AI glasses typically require explicit, user-specific eye-tracker calibration, and accuracy can drift when the device shifts during use. We present GazeAgent, a VLM-guided framework that continuously and adaptively improves gaze-calibration accuracy from naturally occurring behavior without asking users to interrupt their activities. GazeAgent identifies precise visual attention (PVA) events from multimodal streams of inertial measurement unit (IMU), gaze, and wearer-hand signals, validates their task relevance, and localizes functional targets as pseudo-calibration targets. In a 10-participant scripted study, the system detected all 820 predefined PVA opportunities and, under device disturbance, reduced median gaze error by 44.1% compared with explicit calibration. In a five-participant case study, autonomous calibration produced the lowest workload and interruption ratings, and all participants preferred it for routine use. By deriving task-critical targets from natural multimodal behavior, GazeAgent demonstrates the potential for continuous, adaptive, and low-burden gaze interaction on egocentric AI glasses.

Comment
27 pages, 14 figures, 3 tables

Closest papers

Nearest neighbours by meaning across the whole index.

From Laboratory to Road: Evaluating Wearable Gaze Accuracy for Driving

cs.CV ·2026-10-06 ·William Engel, Fabian Flohr
similarity 0.74
Abstract · Peer-reviewed and accepted as an Extended Abstract at the German Conference on Pattern Recognition (GCPR 2026). Presented as a poster at GCP…

Bird's-eye-view (BEV) representations have become a widely used interface between perception and planning in autonomous driving, but they encode what is in a scene, not what is behaviorally relevant to a human driver. Gaze offers a compelling behavioral signal for this gap, yet wearable eye trackers are routinely deployed as if their spatial output were ground truth, despite known sensitivity to head motion, illumination, and calibration drift. We present, to our knowledge, the first unified framework for quantifying wearable gaze accuracy under real driving conditions. Our on-road study contains 41 validated scenes in which one driver fixated a vehicle's license plate. Gaze error is measured as the angular difference between the plate center and the gaze direction estimated by the glasses. Separate indoor studies with the same driver and device systematically analyze how distance, illumination, head motion, target motion, and gaze eccentricity affect both systematic bias and gaze precision. The mean on-road error was 4.58 degrees. Applying an offset estimated from the indoor recordings reduced it to 1.10 degrees and improved all 41 scenes. Because this offset varied between sessions, reliable BEV supervision may require online recalibration and condition-dependent estimates of gaze uncertainty.

0.15

VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users

cs.CV ·2026-10-16 ·Noé Mosca, Mohamed Rayan Barhdadi, Muhammad Hamza, Joshua Yeh, Suryodaya Bikram Shahi +9 more
benchmarkcode/data similarity 0.73
Abstract · Code and dataset are available at https://harvard-ai-and-robotics-lab.github.io/VISTA

Blindness and low vision limit many people's ability to complete daily tasks, motivating AI assistants that provide useful guidance. Yet, existing benchmarks do not adequately test whether vision models can provide effective everyday assistance. In particular, i) existing benchmarks primarily rely on visual and textual inputs, with limited support for other modalities that are increasingly available in egocentric wearable devices such as gaze, inertial measurements, spatial trajectories, and others; and ii) existing evaluations emphasize scene understanding and question answering, with limited coverage of goal-conditioned guidance for blind and low-vision users across everyday tasks. To bridge this gap, we introduce VISTA (Visually Impaired Scene and Task Annotations), a multimodal egocentric benchmark for goal-oriented assistance for blind and low-vision users. VISTA contains $\sim$1,000 egocentric recordings captured with Project Aria glasses; each recording provides synchronized sensor streams, six of which serve as model inputs, and three different types of annotation, covering ten task categories (TC), from object localization to hazard detection. We evaluate 14 representative vision-language models zero-shot on VISTA and design a lightweight multimodal model baseline that integrates all six modalities. Experimental results show that, although existing models can often describe egocentric scenes, they remain limited in generating goal-directed assistive guidance. VISTA thus provides a benchmark for evaluating multimodal models that could improve everyday accessibility.

0.25

EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input

cs.CV cs.AI ·2026-08-16 ·Matteo Stoiber, Niels Buus Lassen
published similarity 0.71
Abstract · 16 pages. Accepted at the WearableAI Workshop, ECCV 2026 (Archival Track)

The use of multimodal LLMs (MLLMs) for egocentric video understanding with wearable devices is constrained by the token budget. Memory and compute cost scale with the number of visual tokens, and high-resolution video quickly becomes expensive to transmit and process at scale. Prior work (GazeLLM) addresses this by cropping the video around the camera wearer's gaze. This reduces the number of visual tokens by about tenfold while maintaining or improving the quality of full-resolution descriptions. However, this compression strategy depends on dedicated eye-tracking hardware, which is unavailable on consumer smart glasses. Building a software-only substitute poses a joint constraint: the predictor must be accurate enough to preserve downstream description quality, yet light enough to run on-device, within the power and compute budget of a smartphone. We address this with EgoGazeLite, a lightweight dual-process gaze predictor for egocentric video. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops show no significant difference from ground-truth-gaze crops. Equivalence is confirmed in all ten cases. EgoGazeLite achieves this at 15.7M parameters, 6.71 GFLOPs, and runs the full gaze-and-crop pipeline end-to-end in real time (21.6 ms/frame) on consumer accelerator hardware. Together, these results remove the need for eye-tracking hardware for token-efficient, gaze-conditioned egocentric video understanding with MLLMs.

0.26

From Seeing to Acting: Smart Glasses as First-Person Intelligence Platforms

cs.CV ·2026-08-25 ·Jiangning Zhang, Haojun Chen, Yong Liu
surveycode/data similarity 0.69
Abstract · Project at https://github.com/zhangzjn/awesome-smart-glasses

Smart glasses are evolving from capture and display accessories into first-person intelligence platforms that connect human perception, persistent context, and digital or physical action. Their on-body viewpoint aligns with the wearer's vision, audition, motion, and hand-object interaction, but must operate under tight energy, thermal, privacy, and feedback constraints. Despite rapid progress in augmented reality, egocentric vision, multimodal models, human-computer interaction, and embodied intelligence, the literature remains fragmented across devices, tasks, and benchmarks. \textit{The key challenge is not whether a model can recognize, answer, remember, or act in isolation, but whether a complete system can sustain a reliable, temporally valid, correctable, and governable perception-state-interaction-action loop.} This survey is \textit{the \textbf{first} to systematically study smart glasses through such a unified framework}. We formalize first-person data flow and constrained task utility, characterize devices along eight verifiable hardware capability axes, organize the literature around seven interdependent foundational capabilities, and introduce an L0-L5 framework spanning capture, reactive perception, contextual assistance, persistent state, governed action, and embodied coupling. Across nine application scenes, we connect tasks with datasets, systems, products, stakeholders, failure consequences, and evidence gaps. We further present a nine-dimensional deployment framework, a claim-conditioned evaluation protocol, and an evidence ladder from controlled measurement to longitudinal field validation and audit. Together, these elements make smart glasses more comparable, deployable, and reproducibly evaluated, while outlining a roadmap toward trustworthy first-person intelligence.

0.27

SurgGaze: Implicit Calibration for Accurate Gaze Analysis in Operating Rooms with Wearable Eyetrackers

cs.HC ·2026-09-22 ·Jingying Wang, Rosiana Natalie, Keyuan Hu, Wenqian Xu, Brian George +3 more
similarity 0.69
Abstract

Accurate gaze tracking is essential for understanding surgeons' visual attention and cognitive processes during laparoscopic surgery, yet wearable eye trackers produce large errors systematically correlated with ground-truth gaze locations, as demonstrated in Study 1. We introduce SurgGaze, an implicit calibration method that corrects these errors using high-confidence surgical moments. Building on evidence that surgeons' gaze converges near the tool-tissue contact point (TTCP) during dissection, SurgGaze uses TTCP as a surrogate for true gaze to construct training pairs. We evaluate SurgGaze in a simulated operating room trial and an authentic operating room case study. In simulation, SurgGaze reduced gaze estimation error by 40.6%, significantly outperforming conventional 9-point explicit calibration. The case study showed that these moments provide reliable training data and that calibrated gaze improves interpretation of surgeons' attention beyond numeric error reduction. These findings demonstrate that structured behavioral signals can enable implicit calibration for gaze tracking in complex real-world settings.

0.06

Drift Calibration in Geometric Eye Tracking Systems

cs.CV ·2026-08-30 ·Jiaqi Liu, Zixuan Wang, Yuhong Zhang, Dingkang Liang, Jane Hanqi Li +2 more
benchmarkdataset similarity 0.69
Abstract · 11 pages, 6 figures, 3 tables

Geometric eye trackers can provide the spatial accuracy required for gaze-based interaction and multimodal studies, but their measurements remain sensitive to residual session-specific calibration error. Research on correcting this error is difficult to compare because methods are typically evaluated with different devices, target layouts, and error definitions. We present a calibration-focused dataset containing 163 trials from 12 participants, with separate 18-point fitting and 32-point test grids, and use it to evaluate global, local, and composite correction functions under a common spatial-extrapolation protocol. We further introduce a lightweight neural refiner that combines ranked predictions from complementary calibrators. On this controlled dataset, post-vendor correction reduces the mean angular error from $1.53^\circ$ to $1.03^\circ$ with the strongest classical composite and to $0.96^\circ$ with the refiner. In a closed-loop gaze task, lower residual error is associated with higher performance across four online correction conditions. These results provide a reproducible data-quality benchmark for using gaze as a behavioral signal in interactive modeling.

0.21

EyeTAG: Eye Trajectory-Aware Gaze Estimation

cs.CV cs.LG ·2026-10-01 ·Jungmin Lee, Niamat Ullah, Yoseob Han
publishedcode/data similarity 0.68
Abstract · Accepted to BMVC 2026

Gaze estimation under natural head-eye motion underpins applications from driver monitoring to human-computer interaction. Single-frame methods predict each frame independently, so consecutive outputs fluctuate as jitter. Multi-frame methods reduce this, but they learn motion implicitly inside appearance features, so the gaze trajectory is never an explicit variable. We propose EyeTAG (Eye Trajectory-Aware Gaze Estimation), a causal multi-frame framework built around an explicit first-order gaze prior: at each step it differentiates its own recent predictions and feeds the resulting trajectory back as a compact kinematic token. Because differencing is translation-invariant in gaze space, this token carries subject-invariant motion rather than personal gaze offsets. Face and eye streams supply visual evidence, fused by cross-attention and a causal Transformer decoder. EyeTAG reduces the mean angular error by about 1.0$^\circ$ on Gaze360 and performs on par with the strongest baseline on EVE (2.56$^\circ$ vs. 2.58$^\circ$). Within-model ablations, which keep the encoder and the rest of the architecture fixed and vary only the gaze history, show that the differential formulation, rather than temporal context alone, removes the systematic saccade bias that persists even with an absolute gaze-history prior. Our code is available at https://github.com/peter8366/EyeTAG.

0.38

An Eye-Tracking Dataset for Viewing Distance Categories in Real-World Scenarios

cs.HC ·2026-08-29 ·Dohwa Kim, Yejin Choi, Seungbok Lee, Chi Yoon Jeong, Eunji Park
dataset similarity 0.68
Abstract · 22 pages, 6 figures, Manuscript submitted to Scientific Data

Estimating viewing distance from gaze behavior is essential for understanding user intent and enabling distance-aware interactive systems. However, most existing eye-tracking datasets have been collected in constrained settings, such as laboratory environments or static tasks. Consequently, they only partially capture viewing behaviors in real-world situations where viewing distance changes with natural head and body movements. We introduce GazeDepth, an eye-tracking dataset collected from 19 participants using a wearable tracker during tasks reflecting real-world scenarios. GazeDepth includes fixed-distance viewing scenarios with constant observer-target distances at near (33 cm), middle (50 cm), and far (300 cm), as well as variable-distance viewing scenarios in which participants shift gaze among targets at different depths in indoor and outdoor environments. The dataset provides synchronized gaze data, pupil size, 3D eye-vectors, and head-motion signals, along with distance labels. Statistical analyses showed that distance-related gaze features, such as vergence angle and estimated viewing distance, differed consistently across viewing-distance categories. In addition, classification models trained on GazeDepth further demonstrated that the dataset captures gaze characteristics that distinguish the three viewing-distance categories, supporting gaze-based distance inference and distance-aware interaction in realistic scenarios.

0.26