CS Paper Radar

MRxD: A Usage-Pattern Framework for Mixed-Reality Cross-Device Interaction

Yang Zhan, Yuxuan Huang, Yuchen Cai, Yichen Andy Yu, Xiaoran Yang, Ye Yuan, Qiao Jin

cs.HC ·submitted 2026-11-12 02:00 UTC · v1

Abstract

Mixed reality (MR) cross-device experiences often combine MR headsets with non-immersive devices such as laptops, tablets, and phones. Yet such cross-device design remains challenging because users must coordinate workflows across heterogeneous devices with different roles. This motivates our focus on usage patterns: recurrent ways in which people distribute and coordinate work across devices. Through a systematic analysis of 238 MR cross-device papers, we developed MRxD, a usage-pattern framework organized around three dimensions: user attention, task relatedness, and cognitive symmetry. Using these dimensions, we identified six usage patterns and synthesized the design strategies commonly used within each pattern. We further investigated the framework's application in design practice through a workshop (n=6) and the results suggest that MRxD can guide pattern-based design exploration and identify blind spots. We further discuss how usage patterns may shift or combine, and outline research gaps and design implications for future MR cross-device systems.

Comment
22 pages, 11 figures

Closest papers

Nearest neighbours by meaning across the whole index.

AdapTest: Supporting Designers in Authoring and Evaluating Environmentally Responsive User Interfaces for Extended Reality

cs.HC ·2026-10-15 ·Jungwoo Rhee, Guhn Lee, Yumin Jung, Mustafa Doga Dogan, Kongpyung +2 more
published similarity 0.64
Abstract · Accepted to PACM HCI ISS 2026

Extended Reality (XR) headsets increasingly host familiar 2D user interfaces as windows floating in physical environments. However, their usability depends on environmental conditions that designers cannot fully control. Current design workflows often separate how a variant is designed from how it actually looks in the target environment. We present AdapTest, an in-situ authoring and testing tool that lets designers select interface elements, define when and how they change, compare the resulting variations, and test their activation without coding. Based on prior research and a formative study with 12 interface designers, the system targets three key environmental variables: distance, lighting, and occlusion. We evaluated AdapTest against a desktop design and in-headset preview workflow in a within-subjects study with 12 designers. Compared with the baseline, designers accounted for a wider range of environmental factors, addressed more overlapping conditions, and repeatedly tested and revised their designs in place; there were no significant differences in workload or usability despite longer sessions. Our work contributes an integrated, designer-centered approach to adapting, comparing, and validating environmentally responsive XR user interfaces.

0.30

User Preferences for UI Anchoring in MR: Effects of Task Mobility and Interface Properties

cs.HC ·2026-08-28 ·João Belo, Sina Elahimanesh, Anna Maria Feit
top venue similarity 0.62
Abstract · Accepted to IEEE International Symposium on Mixed and Augmented Reality 2026

Anchoring - the choice of frame of reference for mixed reality (MR) interface elements - is a critical design decision involving trade-offs between accessibility, interaction comfort, and visual interference. Despite its importance, user preferences for anchoring across different mobility contexts and interface properties remain poorly understood, as prior work has largely focused on specific tasks or fixed interface configurations. We address this through a mixed-methods user study in which participants configure anchoring strategies across different mobility conditions and interface types. Combining behavioral analysis with structured qualitative inquiry, we analyze how participants select and reason about anchoring modes. Our results show a clear transition from world-anchored interfaces in stationary contexts to body-anchored interfaces during locomotion. However, no single body anchor consistently dominates, highlighting the personal nature of anchoring strategies. Our qualitative analysis reveals the factors users consider in their anchoring decision, including interface accessibility, stability during interaction, visual clutter, and individual mental models. These findings inform the design of adaptive and controllable MR interfaces and highlight the importance of supporting user customization.

0.53

Immersive Commons: Designing Ephemeral Spatial Dialogue for Co-Located Mixed Reality Environments

cs.HC cs.MM ·2026-10-26 ·Wuwu Ran, Mohammed Faraj Al-Suwaidi
published similarity 0.60
Abstract · 14 pages, 8 figures. Accepted at SIGraDi 2026, the Conference of the Iberoamerican Society of Digital Graphics

Head-mounted mixed reality (MR) often prioritises individual immersion, potentially drawing attention away from others sharing the room. This paper asks how MR can support co-located spatial dialogue through embodied interaction and activity visible to non-headset observers. We introduce Immersive Commons, a design framework, and Atmos Messages, a prototype combining gesture-based, ephemeral spatial messages with voice transcription, preset text, and a public floor-plan overview. Following a Research through Design approach, fifteen participants created 236 messages across two sessions, including 48 voice-authored messages. Questionnaire responses highlighted playfulness and social awareness, while observations suggested opportunities for exchange grounded in the architectural setting. Interaction clustered near the public display, although this was also where participants put on their headsets. These findings suggest the potential of ephemeral spatial messaging to support co-located exchange, with playfulness, shared visibility, and architectural context informing the design of more social and situated MR experiences.

0.22

Passthrough Rigidity: The Behavioral and Visuomotor Costs of Mediated Perception

cs.HC ·2026-09-21 ·Markus D. Solbach, Mohit Goyal, Sakar Khattar, Jayant Varma, Bjorn Vlaskamp +2 more
publishedcode/data similarity 0.58
Abstract · 10 pages, 16 Figures, ISMAR

Broad public adoption of head-mounted displays using video passthrough remains elusive despite significant market investment. A precise understanding of why users experience persistent discomfort even as hardware factors such as resolution and latency have dramatically improved remains an open issue. This paper investigates the impact of viewing the world through video passthrough systems on human behavioral and physiological patterns through a large-scale multimodal study. We developed a novel protocol to capture synchronized oculomotor, kinematic, and physiological data during a block assembly task requiring complex hand-eye coordination. Using a within-subject design (N=110), we evaluated both natural and passthrough viewing conditions. Our results reveal a four-fold suppression of rotational head velocity and a pronounced decoupling of head-gaze coordination. This suggests motor caution being employed as an adaptive strategy - which we term "Passthrough Rigidity". This phenomenon appears to shift the information-gathering burden to the oculomotor system, resulting in significantly longer fixation durations and restricted visual search patterns. These kinematic shifts directly correlate with poorer task performance and measurable physiological cost, evidenced by a significant reduction in blink duration and increased reports of ocular strain and cognitive load. We conclude that current passthrough implementations induce a measurable shift from flexible exploration to motor caution, where task performance is preserved at the cost of user comfort and biomechanical efficiency. These findings provide a novel quantitative framework for evaluating and improving future XR devices, establishing that resolving "comfort" for passthrough requires addressing the deep-seated biomechanical compensations caused by mediated perception. Data available at https://data.nvision.eecs.yorku.ca/Passthrough

0.29

Code Black: Desktop-Mediated Co-Design of AR-HMD Microinteractions for Emergency Department Teamwork

cs.HC ·2026-09-03 ·Jonathan Segal, Jalynn Nicoly, Francisco Ortega, Angelique Taylor
published similarity 0.55
Abstract · 12 pages, 5 figures, 2 tables. Accepted to ACM Symposium on Spatial User Interaction (SUI 2026)

Emergency Department (ED) teams coordinate shifting roles, medication decisions, and time-critical interventions under uncertainty. Augmented reality head-mounted displays (AR-HMDs) have shown potential to spatially anchor information during care, creating opportunities to examine how spatial interfaces might support teamwork. We conducted a speculative co-design study with 12 healthcare workers (HCWs) using an editable, desktop-mediated Unity-based 3D design probe to visualize and refine work-as-imagined AR-HMD interfaces for role-based notifications, task-specific timers, and dosage verification. Guided by microinteraction rules, participants identified future spatial user interfaces (SUI) requirements such as how they appear, update, or are dismissed in relation to clinical practice, safety concerns, and existing tools. Five returning participants and 26 additional HCWs subsequently provided follow-up feedback on derived visual interface alternatives. Findings show that desktop-mediated spatial co-design elicited formative specifications for role visibility, task-linked timing, and verification-oriented dosage assistance, while revealing tensions involving clutter, shared awareness, communication, privacy, and reliability. Rather than evaluating a functional AR-HMD system or team-based clinical performance, this study contributes the Speculative Co-Design Framework for AR-HMD Teamwork (SCF-HMD) and a visual design catalog for translating expert critique of work-as-imagined (WAI) concepts into situated goals for future AR-HMD systems.

0.46

Smart Glasses for Ubiquitous Computing: A Layered Survey and the Road Ahead

cs.HC ·2026-11-17 ·Linlu Gao, Yueyuan Sui, Junxi Xia, Xin Liu, Stephen Xia +1 more
survey similarity 0.54
Abstract

We conduct a systematic survey of smart-glasses research for ubiquitous computing. Our search covers work published from 2020 to October 2026, with selected earlier work included for historical context. After screening and title-level deduplication, we identify 163 smart-glasses papers and separately compile related resources, including 14 representative hardware platforms, their associated software tools and pretrained models, and 10 datasets. We organize this literature through four complementary layers. Research Resources covers hardware platforms, software tools and models, and datasets that support development and evaluation. Sensing & Perception examines how glasses measure and interpret information about users and their surroundings. Interaction & Assistance addresses how users express intentions and receive feedback and assistance. Applications situates these contributions in tasks and use contexts. Energy, latency, wearability, reliability, privacy, security, and social acceptability are treated as deployment constraints across all four layers. This framework provides a unified view of current research and a vision for how advances across the four layers could come together to support smart glasses as a ubiquitous computing platform. Building on this synthesis, we outline research directions for extending capabilities across tasks and over time, supporting assistance under human control, and achieving reliable, efficient, and sustained everyday use.

0.10

Exploring Object Recall in an Object-Rich AR Office Scene

cs.HC ·2026-10-13 ·Sangita Kunapuli, Shane Dirksen, Radha Kumaran, Yanxiu Jin, You-Jin Kim +2 more
published similarity 0.54
Abstract · 11 pages, 9 figures. Accepted to IEEE ISMAR 2026 and published in IEEE Transactions on Visualization and Computer Graphics (TVCG), Vol. 32, …

Augmented reality (AR) changes how users perceive and attend to their surroundings. Prior work suggests AR can reduce attention to the physical world, but it remains unclear whether this holds in environments dense with both virtual and physical objects, and which factors most affect object recall. This study examines recall in a mixed-reality office scene after an object-selection task, testing object-level factors (virtuality, size), context-level factors (semantic congruency, saliency), and user-level factors (cueing, dual tasking). Cueing informed participants of a later recall test, while dual tasking added cognitive load through a secondary audio task. Virtual objects were recalled more accurately than physical ones. Cueing produced a modest improvement in recall of present objects. Using playback software to reconstruct each participant's view throughout the trials, we computed object-level saliency and found a weak positive correlation with recall, slightly stronger than the effects of virtuality, but not reaching statistical significance. These findings have implications for AR interface design and for building mixed-reality environments that better support cognition during real-world tasks.

0.30

ControlAnything: Generative UI and On-Demand Device Integration for Mixed-Reality Hand Interaction by a Coding Agent

cs.HC ·2026-11-05 ·Bektur Ryskeldiev
code/data similarity 0.54
Abstract · 10 pages, 2 figures, 1 table. Code: https://github.com/Apolotary/controlanything

Connecting hand input in mixed reality (MR) to a nearby physical device typically requires an integration written in advance for that device, its transport and the mapping. ControlAnything defers this integration to the moment of use and assigns it to a coding agent. On a Meta Quest 3, the wearer selects an object outlined by SAM 3.1 and states what their hands should control. An agent built on Meta's Muse Spark model then writes a connector program on the wearer's laptop, tests it against the live device through hand-written MIDI, Bluetooth and key-press transports, and returns augmented-reality (AR) controls, a per-frame hand program, or both. These outputs are a form of generative UI, interfaces created by an AI model; here, the agent also generates and tests the code that binds the interface to a physical device. In a single-session technical evaluation with the author as the only wearer, nine of ten requests addressing five targets reached a ready state, and the wearer stopped one. The median time to ready was 32.5 s (range 24.9-57.8 s, excluding speech transcription); sequential model calls accounted for a median of 89.9% of this time, and 91.7% of input tokens came from the prompt cache. The system's evidence of success extends to transport acknowledgements but not to physical effects. These findings point to connector reuse and the observation of effects through the headset's own sensors as next steps for on-demand device integration in MR. A user study with first-time wearers is planned.

0.48