CS Paper Radar

AdaptType: Accessible and Customizable Multimodal Text Entry in Virtual Reality for People with Upper-Body Motor Impairments

Junhan Kong, Mingyuan Zhong, Sanjana Pavani, Jacob O. Wobbrock

cs.HC ·submitted 2026-11-14 02:16 UTC · v1

Abstract

Text entry in virtual reality (VR) is cumbersome, especially for people with upper-body motor impairments. Existing techniques often demand both fine and gross motor control, yet users' motor abilities vary widely. To address this range of needs, we present AdaptType, a multimodal VR text entry system that enables users to configure input techniques that best suit their abilities, and scaffolds customization through ability-based recommendations. Through a study with 10 participants, we found that customized multimodal techniques improved text entry speed compared to gaze- or hand-only techniques, and complemented voice for error correction, decreasing failure rate from 15.1% to 2.0%. We found that AdaptType yielded high text entry speed (36.26 WPM) and accuracy (0.36% error rate), with low task load and high usability ratings. We also found ability-based recommendations effectively scaffolded customization. Our work contributes more accessible and personalized VR text entry solutions for people with upper-body motor impairments.

Closest papers

Nearest neighbours by meaning across the whole index.

Comparing Hand and Controller Avatars with Hand Tracking and Controller-Based Interaction

cs.HC ·2026-09-18 ·Natalia Ocampo, J. Felipe Gonzalez, Robert J. Teather
top venuehighly cited similarity 0.65
Abstract

Previous research suggests that the congruency between common VR input devices - such as controllers or hand tracking - and their visual representations (e.g., hand or controller avatars) influences user experience and performance. However, the specific effects of input-avatar combinations remain underexplored. We study the effects of common input devices (hand tracking and controllers) and visual representations (hand and controller avatars) on performance and perceived success in target acquisition tasks. We included both grasping and pinching gestures across 16 combinations of input, avatar, and target size. Results indicate that hand tracking benefits from any form of visual representation - even when mismatched - achieving up to 5.8% greater accuracy compared to having no avatar, likely due to its reliance on visual feedback in the absence of a physical prop. Controllers were generally preferred and offered faster task completion. However, mismatched avatars had a stronger negative effect with controllers, particularly when the virtual gesture did not align with the physical action, leading to a 5.6% drop in accuracy compared to the matched condition - suggesting that inaccurate feedback can be more disruptive than having no avatar feedback at all.

0.65

Selfrionette: A Fingertip Force-Input Controller for Continuous Full-Body Avatar Manipulation and Diverse Haptic Interactions

cs.HC ·2026-11-02 ·Takeru Hashimoto, Yutaro Hirao
top venuehighly cited similarity 0.64
Abstract · 14 pages, 12 figures. Author-prepared version of the UIST 2024 paper. Both authors contributed equally

We propose Selfrionette, a controller that uses fingertip force input to drive avatar movements in virtual reality (VR). This system enables users to interact with virtual objects and walk in VR using only fingertip force, overcoming physical and spatial constraints. Additionally, by fixing users' fingers, it provides users with counterforces equivalent to the applied force, allowing for diverse and wide dynamic range haptic feedback by adjusting the relationship between force input and virtual movement. To evaluate the effectiveness of the proposed method, this paper focuses on hand interaction as a first step. In User Study 1, we measured usability and embodiment during reaching tasks under Selfrionette, body tracking, and finger tracking conditions. In User Study 2, we investigated whether users could perceive haptic properties such as weight, friction, and compliance under the same conditions as User Study 1. Selfrionette was found to be comparable to body tracking in realism of haptic interaction, enabling embodied avatar experiences even in limited spatial conditions.

0.61

Ageing, Digital Literacy, and Interaction Modality in Immer-sive Virtual Reality: Psychomotor Performance, Cognitive Flexibility, and Their Processing-Speed Association

cs.HC cs.CY ·2026-09-17 ·Panagiotis Kourtesis, Katerina Denaxa, Lydia Asimakopoulou, Petros Roussos, Maria Roussou
published similarity 0.62
Abstract · 27 pages, 5 figures, 6 Tables

Extended reality (XR) increasingly supports training and cognitive assessment, yet the age sensitivity of its interaction techniques is unclear. This study examined age, digital literacy, and interaction modality as correlates of psychomotor and cognitive-flexibility performance in immersive virtual reality (VR). Two hundred and two adults (19-90 years) completed a five-mode Fitts' law task (eye-gaze, head-gaze, controller ray-casting, virtual finger, and controller direct touch), the Trail Making Test in VR (TMT-VR), and a digital-literacy questionnaire. Age was associated with slower task times across modes, but controller direct touch carried the steepest relative age gradient yet remained among the fastest in absolute terms; technique altered relative age sensitivity without determining absolute efficiency. Higher digital literacy was associated with faster TMT-VR completion but not Fitts task times. A Fitts-derived speed score predicted TMT-VR performance beyond age and partly accounted for its age association, consistent with shared processing-speed variance; an exploratory full-battery extension confined the Fitts-score association to completion-time and error-adjusted-time indices and the digital-literacy association to error-adjusted-time indices, not wrong-target errors, mean selection distance, or relative Part B indices. Age-inclusive XR assessment should standardise interaction modality, evaluate rather than exclude mid-air direct selection, and interpret cognitive scores alongside digital literacy and psychomotor speed.

0.50

InterView-C: A Synchronized Multimodal Corpus of VR Avatar-Mediated Survey Interviews

cs.CL ·2026-10-07 ·Patrick Schrottenbacher, Leon Hammerla, Lydia Kleine, Doris Stingl, Alexander Mehler
similarity 0.61
Abstract

We present InterView-C, a German multimodal corpus of 27 survey interviews conducted entirely in virtual reality, with both interlocutors represented by avatars. The corpus aligns spoken interaction with synchronized behavioral data, including gaze, head and body movement, facial behavior, hand and finger tracking. Its reference transcripts and linguistic annotations provide a reliable interface between this multimodal spoken interaction and predominantly text-based NLP methods. This interface is important because automatically transcribing speech can distort linguistically relevant information, while downstream models trained on existing resources may additionally face transfer challenges when applied to transcribed spoken data. InterView-C therefore provides word-timed and manually post-edited verbatim transcripts for all 54 recordings, interview-item timings, questionnaire responses and negation cue and scope annotations for 1,422 sentences, 1,398 of them doubly annotated (α=0.87 for cues; α=0.81 for scopes). We demonstrate both challenges empirically: nine open-weight ASR systems disproportionately misrecognize short closed answers and number words, while negation models trained on existing corpora show lower and highly variable performance on our transcribed interviews than a model trained on the InterView-C annotations. InterView-C thus enables linguistic analyses of spoken interaction while retaining their alignment with rich multimodal behavior.

0.07

Comparing Haptic Feedback Across Hand Tracking and Controllers in VR Object Interaction Tasks

cs.HC ·2026-09-18 ·Natalia Ocampo, J. Felipe Gonzalez, Robert J. Teather, Kiyoshi Kiyokawa
published similarity 0.60
Abstract

Hand tracking offers natural VR interaction but lacks controllers' inherent tactile feedback, while haptic feedback across input methods remains understudied. We conducted a participant study comparing vibration, impulse, and no feedback in hand- and controller-based VR interaction across grasp-, pinch-, and tap-like gestures assessing performance, workload, and preference. A custom glove and controller-mounted device provided both feedback types, aiming for consistency across input modalities. Controllers were faster for grasp and pinch, whereas hand tracking was more accurate for pinch and preferred overall. Haptics had limited performance effects, although impulse reduced grasp accuracy with controllers. Participants preferred vibration and impulse over no haptic feedback, favouring vibration overall. Our findings reveal a more nuanced relationship between haptic feedback, input device, and interaction context than suggested by previous work.

0.36

Exploring Avatar-Based Representations of Desktop Analytical Workflows for Asymmetric Collaborative Visual Analytics

cs.HC ·2026-09-14 ·Tiansu Chen, Yalong Yang, Wai Tong
similarity 0.60
Abstract

Collaborative visual analytics increasingly occurs across asymmetric desktop-VR settings, with desktop analysis offering precision and efficiency and VR providing spatial and embodied affordances. However, maintaining workspace awareness remains challenging because desktop collaborators are often represented in VR only through indirect cues such as perspective sharing, shared visualization state, or lightweight cursor traces, which do not convey their ongoing analytical activity in a VR-native manner. To address this gap, we present Desk2Avatar, which explores avatar-based representations of desktop analytical workflows in VR. We introduce two representation strategies, Distanced and Embodied, inspired by remote pointing and direct manipulation. We conducted a within-subject study with 18 participants comparing these strategies against a depth-adaptive cursor baseline. Our findings show that avatar-based representations improved collaborator awareness and attentional guidance over cursor cues, while excessive embodiment introduced occlusion, distraction, and additional workload. Finally, we discuss design implications for future desktop-to-VR representations, focusing on balancing collaborator presence, attentional guidance, and workspace readability.

0.24

A Browser-Based Gesture-Driven Avatar Interaction Framework for Metaverse Onboarding Environments

cs.HC cs.GR ·2026-08-12 ·Deepti Parachuri, Chhayank Sahu, Sameer Singh Choudhary
similarity 0.59
Abstract · 6 pages, 4 figures, 2 tables

Avatar interaction shapes how engaging and immersive a metaverse experience feels, and for that interaction to feel natural, avatars need to respond to users without forcing them through a controller-based interface first. This paper describes a gesture-driven interaction layer built for a browser-based metaverse onboarding environment, where users explore a set of virtual rooms as an avatar and interact with embedded video, document, and quiz content using hand, arm, and head gestures instead of a keyboard or controller. The system combines real-time gesture recognition (Google MediaPipe) with two alternative locomotion techniques - hand-raise navigation and in-place walking - so users can trade off precision against physical immersion depending on the task. The contribution is the integration, deployment, and evaluation of these techniques as a single lightweight, web-deployable, controller-free interaction model, assessed through a structured internal onboarding session with five participants. We report what worked, what didn't, and the design trade-offs that came out of combining these techniques in one deployed system.

0.07

C-LISTEN: Cognitive Load Impacts of Sensory-Triggered Environmental Navigation in Virtual Reality

cs.HC ·2026-09-29 ·Md. Monowar Hossain, M. Rasel Mahmud
top venue similarity 0.59
Abstract · 6 pages. Published in AIVR 2026, The Third International Conference on Artificial Intelligence and Immersive Virtual Reality, Lisbon, Portug…

Nowadays, Virtual Reality (VR) systems are more helpful for making decisions, training, and rehabilitation. Multimodal settings in these systems impose a significant balance and cognitive difficulties. Additionally, decreased performance, higher cognitive load, user overwhelm, and limited accessibility of VR technology can be caused by excessive auditory, visual, and sensory stimulation. In our pilot study, we examine the strategic design of directional auditory cues to improve cognitive load and enhance user experience in virtual environments. This research also enhances fundamental knowledge that auditory feedback reduces cognitive load in VR, with statistical analysis confirming this improvement (p = .0011). This study investigates how postural balance and pupil diameter correlate with cognitive load while the users navigate. Moreover, this study provides realistic overview design guidelines for accessibility, which will make VR experiences safer and more useful for everyone, including people with cognitive impairments.

0.49