CS Paper Radar

Apple Vision Pro: The good, the bad, the ugly

Santiago Berrezueta-Guzman, Andrei Koshelev, Habiba-Loai Khalil, Stefan Wagner

cs.HC ·submitted 2026-10-29 14:09 UTC · v1
survey

Abstract

Apple Vision Pro (AVP), the headset Apple introduced as its first ``spatial computer,'' built by a single, vertically integrated vendor, entered the market in February 2024 and has since accumulated a fast-growing body of primary research spanning surgery, education, accessibility, design, and human-computer interaction. In this paper, we adopt the tripartite structure of Leone's 1966 film to organize a systematic review of this literature: we cataloged the validated, working applications of AVP (the good), the persistent technical and ergonomic limitations of a still-early-generation headset (the bad), and the ethical concerns and structural harms that follow from deliberate design, marketing, and deployment choices (the ugly). The review is built from a Scopus export of 218 records mentioning AVP, which we narrowed to 104 papers (2024--2026) with a primary focus on AVP, using explicit inclusion/exclusion criteria that we report in full, alongside a PRISMA-informed flow diagram of the screening process. We find the corpus heavily skewed toward validating benefits (medicine and surgery, education, and software tooling account for over half of the included papers) and comparatively thin on documented risks or misuse, a pattern we treat as a finding in its own right rather than a limitation to be explained away. We further show that several recurring issues, namely price, simulated social presence, and the gaze-tracking privacy architecture, resist a clean separation into ``bad'' or ``ugly'' using the published evidence alone, since both readings remain compatible with the same data. We close by identifying where the evidence base most needs to grow: independent risk and ethics research, bystander- and equity-focused study designs, and disclosure that would let researchers distinguish an engineering backlog from a structural choice.

Comment
Pre-print submitted in Elsevier

Closest papers

Nearest neighbours by meaning across the whole index.

From Human Factors to Human-Technology Factors: An HCI Perspective on Technology in Avalanche Safety

cs.HC ·2026-09-07 ·Björn Hartmann, Jason Smith
similarity 0.55
Abstract · Preprint of ISSW 2026 paper

The avalanche safety literature has identified human factors that contribute to accidents, yet researchers note a persistent gap between these insights and effective product design. Meanwhile, interactive technologies are shaping backcountry decision making with or without grounding in theory and research. We broaden the discussion of human factors into ``human-technology factors," examining how technology can both support and undermine judgment in avalanche terrain, from a human-computer interaction (HCI) perspective. We summarize relevant HCI concepts along four dimensions - attention, cognition, trust, and social interaction - and use them to revisit McCammon's FACETS framework, cataloging ways in which specific technologies may mitigate or exacerbate classic heuristic traps, grounded in accident reports and literature where possible. We identify recurring patterns, including technologies with dual-sided effects and a pervasive ``digital expert halo."

0.33

Video-Based Markerless Motion Capture for Clinical and Rehabilitation Biomechanics: A PRISMA-ScR Scoping Review of Validated Architectures, Clinical Readiness, and Emerging Methods

cs.CV physics.med-ph ·2026-09-16 ·Florian Delaplace, Elodie Piche, Frédéric Chorin, Raphael Zory
similarity 0.50
Abstract

Background.. Video-based markerless motion capture promises movement analysis without the cost, space and skin-marker constraints of optoelectronic systems, with particular potential for clinical and rehabilitation settings. Whether validated pipelines yet deliver clinically acceptable biomechanics, and how they relate to the underlying computer-vision research, remains unclear. Methods. We conducted a scoping review following the PRISMA extension for Scoping Reviews, with a registered protocol and searches of PubMed, Scopus and IEEE Xplore (January 2015 to February 2026; the computer-vision scan was updated to July 2026). A dual-tier design paired a primary corpus of validated biomechanical studies with a complementary, curated and deliberately non-exhaustive corpus of emerging computer-vision work, used qualitatively. We charted study characteristics, pipeline architecture, validation methods and joint-angle accuracy. Results. We included 117 studies, most published from 2024 onward and conducted on healthy adults walking in a laboratory. Pipelines formed five architectural families across monocular and multi-camera modalities; most reported raw joint angles without biomechanical refinement. Sagittal lower-limb agreement clustered around 5 to 6{\textdegree}, generally short of clinical acceptability, while out-of-plane kinematics, kinetics, and pathological or older populations were rarely validated. Emerging computer-vision building blocks (foundation-model mesh recovery, differentiable inverse kinematics, video-based kinetics) were almost absent from validated studies. Conclusions. Video-based markerless capture is not yet interchangeable with marker-based systems for clinical joint kinematics, and it remains barely validated where rehabilitation needs it most: older and pathological populations, out-of-plane kinematics, and kinetics. Mapping this evidence gap onto emerging computer-vision advances, we propose hypothesis-generating design guidelines, not a validated method, to steer the next generation of pipelines toward accessible, clinically meaningful movement analysis.

0.33

GAZEleak: Passcode Inference Against Eye-tracking XR Devices Through External Observation

cs.CR ·2026-09-28 ·Hwanjo Heo, Junhee Lee, Jinwoo Kim
similarity 0.50
Abstract · The vendor requests more time to review our responsible disclosure

Mixed-reality headsets such as Apple Vision Pro replace the touch screen with gaze-as-pointer interaction: the wearer looks at a target and confirms with an air pinch. Because the display is inside the headset and the eye tracker is walled off from third-party software, such input is widely assumed to be unobservable to bystanders---a built-in defense against the shoulder-surfing that plagues phones and laptops. We present GAZEleak, a side-channel attack that recovers gaze-driven input from external video of head motion alone, under a strictly local, physical-observer threat model: the adversary only films the wearer from across the room and installs no software on the device. The attack exploits the centrally coupled eye-head motor program: gaze shifts recruit small, target-dependent head reorientations that project into sub-degree pose changes recoverable from commodity video. GAZEleak implements a measurement-based, sparse-optical-flow inference pipeline for users' 6-digit device passcodes. On a preliminary front-view dataset from three author-subjects who were aware of the attack hypothesis, GAZEleak places the true code within the top ten guesses for 56% (10 of 18) of test codes under a cross-person protocol with no labeled victim data, and for every code of the most exposed subject. The performance is subject-dependent: no passcode from the least exposed subject reaches the top ten, although its median guessed passcode rank is 12,786 rather than 500,000 expected from an uninformative ordering. These results provide preliminary evidence that gaze-coupled head motion can expose passcode information under controlled conditions, while motivating broader evaluation across users, behaviors, and capture settings.

0.26

AUV-Bench: Aesthetic Understanding and Generation Evaluation for User Interfaces

cs.AI ·2026-09-28 ·Zhijie Deng, Ling Li, Junhao Ji, Siwei Lyu, Zhipeng Xu +5 more
similarity 0.50
Abstract

Multimodal foundation models are increasingly used for evaluating and generating user interfaces (UIs), often producing seemingly reasonable aesthetic judgments and visually plausible pages. However, under professional design scrutiny, their behavior can differ substantially from that of human designers. In professional design practice, designers rely on a systematic set of aesthetic principles that consistently guide judgment, diagnosis, repair, and creation. A coherent aesthetic capability should therefore connect aesthetic judgment with design actions. Existing evaluations, however, typically assess these abilities in isolation, making it difficult to determine whether task-level success reflects a shared aesthetic understanding or merely fragmented task-specific competence. To address this gap, we introduce AUV-Bench, developed in collaboration with professional UI designers around 1,395 executable web interfaces and four tasks: aesthetic scoring, diagnosis, repair, and text-to-UI generation. The tasks share a pool of UIs and aesthetic principles, with diagnosis and repair further aligned on 660 controlled-degradation instances to enable instance-level analysis of judgment and action. Evaluation of 12 models reveals a capability imbalance: models show moderate agreement with professional designers in holistic aesthetic scoring, yet exact diagnosis-chain success peaks at only 24.7%. On the aligned diagnosis-repair cases, correct judgments and successful repairs do not consistently coincide, exposing a Judgment-Action Gap between identifying aesthetic problems and successfully acting on them. In open-ended generation, even leading models achieve only moderate aesthetic quality under human-calibrated evaluation. Overall, current models exhibit partial aesthetic competence, but still lack the fine-grained understanding and judgment-action coherence required for reliable UI design.

0.13

MRxD: A Usage-Pattern Framework for Mixed-Reality Cross-Device Interaction

cs.HC ·2026-11-12 ·Yang Zhan, Yuxuan Huang, Yuchen Cai, Yichen Andy Yu, Xiaoran Yang +2 more
similarity 0.49
Abstract · 22 pages, 11 figures

Mixed reality (MR) cross-device experiences often combine MR headsets with non-immersive devices such as laptops, tablets, and phones. Yet such cross-device design remains challenging because users must coordinate workflows across heterogeneous devices with different roles. This motivates our focus on usage patterns: recurrent ways in which people distribute and coordinate work across devices. Through a systematic analysis of 238 MR cross-device papers, we developed MRxD, a usage-pattern framework organized around three dimensions: user attention, task relatedness, and cognitive symmetry. Using these dimensions, we identified six usage patterns and synthesized the design strategies commonly used within each pattern. We further investigated the framework's application in design practice through a workshop (n=6) and the results suggest that MRxD can guide pattern-based design exploration and identify blind spots. We further discuss how usage patterns may shift or combine, and outline research gaps and design implications for future MR cross-device systems.

0.30

AdapTest: Supporting Designers in Authoring and Evaluating Environmentally Responsive User Interfaces for Extended Reality

cs.HC ·2026-10-15 ·Jungwoo Rhee, Guhn Lee, Yumin Jung, Mustafa Doga Dogan, Kongpyung +2 more
published similarity 0.49
Abstract · Accepted to PACM HCI ISS 2026

Extended Reality (XR) headsets increasingly host familiar 2D user interfaces as windows floating in physical environments. However, their usability depends on environmental conditions that designers cannot fully control. Current design workflows often separate how a variant is designed from how it actually looks in the target environment. We present AdapTest, an in-situ authoring and testing tool that lets designers select interface elements, define when and how they change, compare the resulting variations, and test their activation without coding. Based on prior research and a formative study with 12 interface designers, the system targets three key environmental variables: distance, lighting, and occlusion. We evaluated AdapTest against a desktop design and in-headset preview workflow in a within-subjects study with 12 designers. Compared with the baseline, designers accounted for a wider range of environmental factors, addressed more overlapping conditions, and repeatedly tested and revised their designs in place; there were no significant differences in workload or usability despite longer sessions. Our work contributes an integrated, designer-centered approach to adapting, comparing, and validating environmentally responsive XR user interfaces.

0.30

PhoneValue-Bench: A Multimodal Benchmark and Datasets for Consumer Smartphone Recommerce Valuation

cs.CV cs.AI ·2026-10-31 ·Zihan Deng, Jun Gao
benchmark similarity 0.47
Abstract

As consumer smartphone recommerce scales, buyers must quote binding repurchase prices from seller-uploaded multi-view photos and diagnostic screenshots before devices reach the warehouse. Accurate remote valuation requires fusing noisy cosmetic evidence with identity, battery, lock, and channel risk from screenshots and staff forms, since an error in any modality changes both grade and payout. However, existing multimodal resources mainly target general VLM competence, screen reading, or factory defect detection, and largely overlook staff-aligned trade-in fields, dual appearance/staff ground truth, and executable valuation scorecards. This gap stems from two challenges: (1) the lack of large-scale real trade-in data with audited dual annotations and a compact ranking benchmark kept separate from training labels, and (2) the need to jointly reason over unordered multi-view cosmetics and diagnostic screens under operational deduction rules. To address these challenges, we propose PhoneValue-Bench, a dual-GT evaluation suite with tier-gated scorecards, together with PhoneValue-Corpus, a matched weakly labeled training set of over twenty thousand orders and hundreds of thousands of captioned images. We further present ValueAgent, a staged perception-fusion-valuation reference pipeline aligned with production inspection, plus an executable public scorer and submission schema. Experiments show that ValueAgent reaches a measured 84.1/100 on Bench-Full, with 97.0 appearance fill and 54.1 staff-field fill, establishing a practical dual-GT benchmark for multimodal recommerce valuation.

0.32

Passthrough Rigidity: The Behavioral and Visuomotor Costs of Mediated Perception

cs.HC ·2026-09-21 ·Markus D. Solbach, Mohit Goyal, Sakar Khattar, Jayant Varma, Bjorn Vlaskamp +2 more
publishedcode/data similarity 0.46
Abstract · 10 pages, 16 Figures, ISMAR

Broad public adoption of head-mounted displays using video passthrough remains elusive despite significant market investment. A precise understanding of why users experience persistent discomfort even as hardware factors such as resolution and latency have dramatically improved remains an open issue. This paper investigates the impact of viewing the world through video passthrough systems on human behavioral and physiological patterns through a large-scale multimodal study. We developed a novel protocol to capture synchronized oculomotor, kinematic, and physiological data during a block assembly task requiring complex hand-eye coordination. Using a within-subject design (N=110), we evaluated both natural and passthrough viewing conditions. Our results reveal a four-fold suppression of rotational head velocity and a pronounced decoupling of head-gaze coordination. This suggests motor caution being employed as an adaptive strategy - which we term "Passthrough Rigidity". This phenomenon appears to shift the information-gathering burden to the oculomotor system, resulting in significantly longer fixation durations and restricted visual search patterns. These kinematic shifts directly correlate with poorer task performance and measurable physiological cost, evidenced by a significant reduction in blink duration and increased reports of ocular strain and cognitive load. We conclude that current passthrough implementations induce a measurable shift from flexible exploration to motor caution, where task performance is preserved at the cost of user comfort and biomechanical efficiency. These findings provide a novel quantitative framework for evaluating and improving future XR devices, establishing that resolving "comfort" for passthrough requires addressing the deep-seated biomechanical compensations caused by mediated perception. Data available at https://data.nvision.eecs.yorku.ca/Passthrough

0.29