CS Paper Radar

Camera-Free Object Discovery and Interaction with GazeTags

Vimal Mollyn, Chris Harrison

cs.HC cs.CVcs.GRcs.MM ·submitted 2026-11-02 06:24 UTC · v1
top venue

Abstract

We present GazeTags, a camera- and computer-vision-free approach for worn head-gaze object awareness. Our small electronic tags cost roughly \$2 each and can operate for more than a year on a coin cell battery. They work in concert with a user-worn device (smartglasses, hearables, etc.) that periodically emits a focused infrared signal aligned with the user's head direction. Only GazeTags within that narrow head-oriented beam wake and respond, providing their identity and optional payloads such as object state or live sensor readings. By avoiding cameras and computer vision pipelines, our approach is very energy efficient, able to operate continuously even in battery-constrained wearables, and sidesteps non-trivial privacy issues. We describe our hardware design, characterize our system performance and operating tradeoffs, and demonstrate how GazeTags can enable practical gaze-aware environments.

Comment
Published at UIST 2026
DOI
10.1145/3830398.3830713

Closest papers

Nearest neighbours by meaning across the whole index.

EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input

cs.CV cs.AI ·2026-08-16 ·Matteo Stoiber, Niels Buus Lassen
published similarity 0.65
Abstract · 16 pages. Accepted at the WearableAI Workshop, ECCV 2026 (Archival Track)

The use of multimodal LLMs (MLLMs) for egocentric video understanding with wearable devices is constrained by the token budget. Memory and compute cost scale with the number of visual tokens, and high-resolution video quickly becomes expensive to transmit and process at scale. Prior work (GazeLLM) addresses this by cropping the video around the camera wearer's gaze. This reduces the number of visual tokens by about tenfold while maintaining or improving the quality of full-resolution descriptions. However, this compression strategy depends on dedicated eye-tracking hardware, which is unavailable on consumer smart glasses. Building a software-only substitute poses a joint constraint: the predictor must be accurate enough to preserve downstream description quality, yet light enough to run on-device, within the power and compute budget of a smartphone. We address this with EgoGazeLite, a lightweight dual-process gaze predictor for egocentric video. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops show no significant difference from ground-truth-gaze crops. Equivalence is confirmed in all ten cases. EgoGazeLite achieves this at 15.7M parameters, 6.71 GFLOPs, and runs the full gaze-and-crop pipeline end-to-end in real time (21.6 ms/frame) on consumer accelerator hardware. Together, these results remove the need for eye-tracking hardware for token-efficient, gaze-conditioned egocentric video understanding with MLLMs.

0.26

GAZEleak: Passcode Inference Against Eye-tracking XR Devices Through External Observation

cs.CR ·2026-09-28 ·Hwanjo Heo, Junhee Lee, Jinwoo Kim
similarity 0.62
Abstract · The vendor requests more time to review our responsible disclosure

Mixed-reality headsets such as Apple Vision Pro replace the touch screen with gaze-as-pointer interaction: the wearer looks at a target and confirms with an air pinch. Because the display is inside the headset and the eye tracker is walled off from third-party software, such input is widely assumed to be unobservable to bystanders---a built-in defense against the shoulder-surfing that plagues phones and laptops. We present GAZEleak, a side-channel attack that recovers gaze-driven input from external video of head motion alone, under a strictly local, physical-observer threat model: the adversary only films the wearer from across the room and installs no software on the device. The attack exploits the centrally coupled eye-head motor program: gaze shifts recruit small, target-dependent head reorientations that project into sub-degree pose changes recoverable from commodity video. GAZEleak implements a measurement-based, sparse-optical-flow inference pipeline for users' 6-digit device passcodes. On a preliminary front-view dataset from three author-subjects who were aware of the attack hypothesis, GAZEleak places the true code within the top ten guesses for 56% (10 of 18) of test codes under a cross-person protocol with no labeled victim data, and for every code of the most exposed subject. The performance is subject-dependent: no passcode from the least exposed subject reaches the top ten, although its median guessed passcode rank is 12,786 rather than 500,000 expected from an uninformative ordering. These results provide preliminary evidence that gaze-coupled head motion can expose passcode information under controlled conditions, while motivating broader evaluation across users, behaviors, and capture settings.

0.26

Smart Glasses for Ubiquitous Computing: A Layered Survey and the Road Ahead

cs.HC ·2026-11-17 ·Linlu Gao, Yueyuan Sui, Junxi Xia, Xin Liu, Stephen Xia +1 more
survey similarity 0.62
Abstract

We conduct a systematic survey of smart-glasses research for ubiquitous computing. Our search covers work published from 2020 to October 2026, with selected earlier work included for historical context. After screening and title-level deduplication, we identify 163 smart-glasses papers and separately compile related resources, including 14 representative hardware platforms, their associated software tools and pretrained models, and 10 datasets. We organize this literature through four complementary layers. Research Resources covers hardware platforms, software tools and models, and datasets that support development and evaluation. Sensing & Perception examines how glasses measure and interpret information about users and their surroundings. Interaction & Assistance addresses how users express intentions and receive feedback and assistance. Applications situates these contributions in tasks and use contexts. Energy, latency, wearability, reliability, privacy, security, and social acceptability are treated as deployment constraints across all four layers. This framework provides a unified view of current research and a vision for how advances across the four layers could come together to support smart glasses as a ubiquitous computing platform. Building on this synthesis, we outline research directions for extending capabilities across tasks and over time, supporting assistance under human control, and achieving reliable, efficient, and sustained everyday use.

0.15

AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalization

cs.CV ·2026-09-17 ·Xu Yuan, Yi Wang, Zhuohang Jiang, Haohao Qu, Yujuan Ding +6 more
survey similarity 0.61
Abstract

Recent advances in artificial intelligence (AI) are reshaping smart glasses from egocentric capture and display devices into platforms for wearable intelligence. Smart glasses increasingly serve as wearable AI systems that connect first-person observation with real-time assistance under strict form-factor constraints. We frame this transition through the lens of \emph{AI smart glasses} and define them as a system-level concept in which egocentric sensing, resource-aware computing, intelligent reasoning, multimodal interaction, and real-world application constraints are co-designed for personalized assistance in the physical world. To systematically study this perspective, we organize the survey around four connected dimensions. First, we examine the hardware foundation that bounds sensing, computation, feedback delivery, and sustained deployment. Second, we study wearable intelligence, where egocentric signals are transformed into perceptual, contextual, and agentic capabilities. Third, we discuss interaction design, through which users request, receive, correct, and regulate assistance during ongoing activity. Fourth, we analyze application scenarios across healthcare, accessibility, situated learning, daily life assistance, cultural tourism, and industrial support, showing how domain requirements reshape system design and evaluation. We further identify five cross-cutting research challenges for future AI smart glasses: next-generation hardware, trustworthy egocentric intelligence, lifelong personalized memory, proactive intelligence, and embodied foundation models. By centering smart glasses as wearable-intelligence platforms, this survey provides a unified framework for organizing technologies, applications, and open challenges in this emerging area.

0.30

From Seeing to Acting: Smart Glasses as First-Person Intelligence Platforms

cs.CV ·2026-08-25 ·Jiangning Zhang, Haojun Chen, Yong Liu
surveycode/data similarity 0.60
Abstract · Project at https://github.com/zhangzjn/awesome-smart-glasses

Smart glasses are evolving from capture and display accessories into first-person intelligence platforms that connect human perception, persistent context, and digital or physical action. Their on-body viewpoint aligns with the wearer's vision, audition, motion, and hand-object interaction, but must operate under tight energy, thermal, privacy, and feedback constraints. Despite rapid progress in augmented reality, egocentric vision, multimodal models, human-computer interaction, and embodied intelligence, the literature remains fragmented across devices, tasks, and benchmarks. \textit{The key challenge is not whether a model can recognize, answer, remember, or act in isolation, but whether a complete system can sustain a reliable, temporally valid, correctable, and governable perception-state-interaction-action loop.} This survey is \textit{the \textbf{first} to systematically study smart glasses through such a unified framework}. We formalize first-person data flow and constrained task utility, characterize devices along eight verifiable hardware capability axes, organize the literature around seven interdependent foundational capabilities, and introduce an L0-L5 framework spanning capture, reactive perception, contextual assistance, persistent state, governed action, and embodied coupling. Across nine application scenes, we connect tasks with datasets, systems, products, stakeholders, failure consequences, and evidence gaps. We further present a nine-dimensional deployment framework, a claim-conditioned evaluation protocol, and an evidence ladder from controlled measurement to longitudinal field validation and audit. Together, these elements make smart glasses more comparable, deployable, and reproducibly evaluated, while outlining a roadmap toward trustworthy first-person intelligence.

0.27

GlanceAR: Repurposing Phone Sensors via Low-Cost Reflectors for Always-On Obstacle Awareness

cs.HC ·2026-10-23 ·Boyang Zhou, Xizhe Hao, Minbeom Kim, Jaewook Lee, Jon E. Froehlich
similarity 0.60
Abstract

Smartphone use while walking reduces environmental awareness and can lead to accidents and injury, yet existing obstacle-awareness systems remain unadopted because they require users to hold the phone upright and aimed forward, conflicting with how people actually hold their devices. We introduce GlanceAR, an always-on mobile obstacle-awareness system designed for natural phone-holding postures. GlanceAR utilizes a low-cost passive mirror attachment to redirect the phone's rear LiDAR and RGB camera forward without additional electronic components. In a within-subjects study with N=11 participants navigating indoor and outdoor obstacle courses while engaged in a primary on-phone task, we evaluated three sensing configurations: GlanceAR, a LiDAR-only variant for privacy, and an attachment-free ultra-wide camera baseline. Our findings reveal trade-offs among sensing fidelity, obstacle cue design, and social acceptability, and surface insights around AI complacency and the tension between on-screen task focus and physical safety.

0.13

ItemCompass: Real-Time Edge-AI Wearable Vision for Tag-Free Domestic Object Retrieval

cs.HC ·2026-10-28 ·Tsz Lok Cheung, Jeff Kai-Tai Tang, Anthony Kong, Zackary P. T. Sin
published similarity 0.59
Abstract · Accepted and presented at the ACM UbiComp / ISWC 2026 Workshop on Wearable Design in Games and Entertainment (WDGE), Shanghai, China

Misplacing household objects disrupts daily routines, especially for elderly users. ItemCompass addresses this with a real-time, tag-free wearable AR system that builds a persistent digital twin of the home for natural language object retrieval. A custom eyewear node streams visual and inertial data to an edge server, where YOLOv11n detection, Siamese re-identification, and a Pedestrian Dead Reckoning engine log object locations as Spatial Anchors in InfluxDB. Voice queries are resolved by a local Faster-Whisper and Qwen2 LLM pipeline, delivering concise spoken and AR guidance on a heads-up display. Evaluation shows 0.40 m localization drift, 97.5% step precision, and mAP@0.5 of 0.874, with users rating Accuracy and Trust at 4.6/5.0. The system highlights always-on sensing, conversational spatial memory, and privacy-by-design for wearable computing.

0.36

From Laboratory to Road: Evaluating Wearable Gaze Accuracy for Driving

cs.CV ·2026-10-06 ·William Engel, Fabian Flohr
similarity 0.58
Abstract · Peer-reviewed and accepted as an Extended Abstract at the German Conference on Pattern Recognition (GCPR 2026). Presented as a poster at GCP…

Bird's-eye-view (BEV) representations have become a widely used interface between perception and planning in autonomous driving, but they encode what is in a scene, not what is behaviorally relevant to a human driver. Gaze offers a compelling behavioral signal for this gap, yet wearable eye trackers are routinely deployed as if their spatial output were ground truth, despite known sensitivity to head motion, illumination, and calibration drift. We present, to our knowledge, the first unified framework for quantifying wearable gaze accuracy under real driving conditions. Our on-road study contains 41 validated scenes in which one driver fixated a vehicle's license plate. Gaze error is measured as the angular difference between the plate center and the gaze direction estimated by the glasses. Separate indoor studies with the same driver and device systematically analyze how distance, illumination, head motion, target motion, and gaze eccentricity affect both systematic bias and gaze precision. The mean on-road error was 4.58 degrees. Applying an offset estimated from the indoor recordings reduced it to 1.10 degrees and improved all 41 scenes. Because this offset varied between sessions, reliable BEV supervision may require online recalibration and condition-dependent estimates of gaze uncertainty.

0.15