CS Paper Radar

DOBI: Dynamic Opportunistic Body Input via Spare Joint Recruitment for Hands-Free XR

Rachel Kim, Xun Qian, Sang Ho Yoon

cs.HC ·submitted 2026-08-31 06:56 UTC · v2
top venue

Abstract

Extended Reality (XR) systems are often most useful when users are engaged in ongoing physical tasks, yet current interaction techniques still largely assume the hands are available. We present opportunistic body input, an interaction paradigm that redirects continuous XR control to whichever available body region remains free in the moment. To investigate how users naturally coordinate these spare-body movements, we conducted an elicitation study across six hand-busy scenarios. We found that while users' preferred spare body regions shift dynamically based on physical constraints, the resulting spontaneous movements share a consistent, low-dimensional kinematic structure organized around a dominant principal axis. Building on these findings, we present DOBI (Dynamic Opportunistic Body Input), a real-time XR technique that uses gaze to target a UI element, a brief trigger gesture to identify the recruited spare body region, and the region's subsequent motion to drive continuous 1D control. A 1D Fitts' law study establishes the baseline motor performance of this paradigm across four distinct body regions, achieving throughputs up to 2.62 bits/s with an overall 5.0% error rate, and a dual-task usability study shows that DOBI supports reliable, low-effort control (SUS = 84.2) while users remain engaged in realistic hand-busy activities.

Comment
14 pages, 12 figures. To appear in the 39th Annual ACM Symposium on User Interface Software and Technology (UIST '26), November 2-5, 2026, Detroit, MI, USA

Closest papers

Nearest neighbours by meaning across the whole index.

UniScale: Exploring Unimanual Gesture Mapping Strategies for Gaze+Pinch-based Scaling Interaction

cs.HC ·2026-09-01 ·Kyoungwhan Mheen, Jinwook Kim, Sang Ho Yoon
published similarity 0.69
Abstract · 11 pages, 7 Figures, Accepted to ISMAR'26, TVCG'26

Object scaling serves as a fundamental spatial manipulation that enables complex and productive tasks in XR environments. This paper investigates unimanual scaling techniques for XR using gaze and hand interactions. We propose UniScale, a set of unimanual alternatives to the standard bimanual pinch, allowing users to scale objects while preserving hand availability for concurrent spatial manipulations. We design five distinct mapping strategies based on physical metaphors, exploring unimanual control that varies depth, angle, micro-gestures, and finger-distance input. We then compare these techniques against a standard bimanual baseline, in which users adjust the inter-hand distance via a bimanual pinch gesture. In a user study, we evaluate their effectiveness in a 3D object scaling task under both clutching and clutching-free conditions. The results indicate that while bimanual scaling relies on clutching for stable control, unimanual techniques excel in clutching-free conditions, significantly reducing physical hand movement. From the results, we derive valuable design implications for developing efficient 3D multimodal interactions in XR.

0.34

Point&Spawn: Mid-Air Reference-Free Object Instantiation Using Gaze and Hand Gestures in Extended Reality

cs.HC ·2026-09-03 ·Jihyeon Lee, Ken Pfeuffer, Jinwook Kim, Jeongmi Lee
top venue similarity 0.69
Abstract · 11 pages, 6 figures, Accepted to ISMAR'26

Mid-air object instantiation in XR requires users to specify a 3D position without spatial references, such as surfaces or existing objects. We present Point&Spawn, a staged pipeline for pre-instantiation position specification through Direction Setting, Depth Setting, and Position Refinement within a continuous gesture flow. We evaluated six controller-free techniques combining Gaze or Non-Dominant Hand (NDH) direction setting with Ray Intersection, Relative Gain, or Drag&Hold depth setting in a user study (N=24) across Near and Far spawn depths. Relative Gain and Drag&Hold yielded faster and more accurate spawning, lower workload, higher usability, and greater preference than Ray Intersection. The shoulder-referenced NDH ray improved speed and coarse accuracy, whereas the viewpoint-based Gaze ray reduced hand movement with comparable final accuracy. Farther spawn depth imposed greater temporal costs as well as Gaze and accuracy costs with Ray Intersection. These findings offer empirical guidance for designing direction and depth control in spawning in XR.

0.44

Body-Grounded Replanning for Physically Adaptive Manipulation

cs.RO ·2026-09-24 ·Namiko Saito, Hiroshi Kera
similarity 0.67
Abstract

Manipulation requires not only reasoning about the external environment, but also about the robot's physical condition. A strategy may remain geometrically feasible while becoming physically unsuitable due to increased joint load or limited mobility, yet internal physical state is typically used only for low-level control. We propose body-grounded high-level replanning, which uses internal physical state to adapt manipulation strategies during execution. Body-state events trigger strategy replanning, and an LLM interprets the underlying joint-level state, recent execution statistics, and execution history to select a context-dependent alternative, while leaving the task objective and low-level controller unchanged. We evaluate the framework on a reaching task under controlled load and asymmetric mobility constraints in simulation and on a real robot. Our experiments show that body-grounded replanning maintains high task success while reducing physical effort and enabling more efficient strategy adaptation. Additional contact-rich manipulation experiments demonstrate the applicability of the same replanning interface beyond reaching. These results show that internal physical state can inform not only low-level control, but also high-level decisions about how a manipulation task should be performed.

0.27

Selfrionette: A Fingertip Force-Input Controller for Continuous Full-Body Avatar Manipulation and Diverse Haptic Interactions

cs.HC ·2026-11-02 ·Takeru Hashimoto, Yutaro Hirao
top venuehighly cited similarity 0.66
Abstract · 14 pages, 12 figures. Author-prepared version of the UIST 2024 paper. Both authors contributed equally

We propose Selfrionette, a controller that uses fingertip force input to drive avatar movements in virtual reality (VR). This system enables users to interact with virtual objects and walk in VR using only fingertip force, overcoming physical and spatial constraints. Additionally, by fixing users' fingers, it provides users with counterforces equivalent to the applied force, allowing for diverse and wide dynamic range haptic feedback by adjusting the relationship between force input and virtual movement. To evaluate the effectiveness of the proposed method, this paper focuses on hand interaction as a first step. In User Study 1, we measured usability and embodiment during reaching tasks under Selfrionette, body tracking, and finger tracking conditions. In User Study 2, we investigated whether users could perceive haptic properties such as weight, friction, and compliance under the same conditions as User Study 1. Selfrionette was found to be comparable to body tracking in realism of haptic interaction, enabling embodied avatar experiences even in limited spatial conditions.

0.61

PolyMorphWBC: Whole-Body Control from End-Effector Trajectories for Quadrupedal Mobile Manipulation across Arm Morphologies with Extended Reach

cs.RO ·2026-11-11 ·Heng Li, Yang Li, Yifan Duan, Ren Fang, Matao Duan +4 more
similarity 0.65
Abstract · 8 pages, 4 figures

Quadrupedal mobile manipulators can extend their manipulation workspace by coordinating the mounted arm with the mobile base. At the task level, short-horizon end-effector (EE) trajectories provide an interface for specifying the desired manipulation motion without prescribing how it should be realized by the arm and base. However, executing such trajectories remains challenging because the controller must determine when and how the base should move, and this decision depends on the mounted-arm morphology. We therefore present PolyMorphWBC, a hierarchical controller that takes an EE trajectory as the sole task-level motion command and generates coordinated arm--base motions across four families of 6-DoF mounted arms, with extended-reach trajectories beyond the initial fixed-base reach. Within PolyMorphWBC, a trajectory-conditioned coordinator determines the base motion from previewed EE targets, while arm-specific Inverse Kinematics (IK) tracks the current target, and a frozen locomotion backend executes the resulting motion. Cross-arm reuse is enabled by a Unified Robot Description Format (URDF)-derived morphology conditioning, together with online residual-wrench feedback. We train PolyMorphWBC on procedurally generated arms and structured trajectories within and beyond fixed-base reach. In IsaacLab, it achieves 93.95% completion with 2.30cm position error and 4.81° orientation error on held-out trajectories and arm morphologies. In MuJoCo evaluation against UMI-on-Legs, PolyMorphWBC reduces EE position error by 69--72% and orientation error by 91--97% across four within- and extended-reach trajectories. On a physical Go2--ARX5 system, we demonstrate language-conditioned extended-reach pick-and-place using EE trajectories from pi_0 without vision-language-action (VLA)-generated base commands.

0.13

Praxis: Distilling Physical Interaction Priors from Egocentric Videos for Generalizable Whole-Body Manipulation

cs.RO ·2026-09-25 ·Shuliang He, Ruiyan Xu, Bo Yue, Hengming Zhang, Huayi Zhou +3 more
similarity 0.65
Abstract

Mobile humanoid manipulation requires both reaching a usable workspace and preserving precise hand-object interactions as object poses and contact conditions change. Learning these behaviors from limited task-specific data remains challenging. To bridge this gap, we introduce Praxis, a whole-body manipulation framework that combines physical interaction priors from one-shot egocentric video demonstrations with closed-loop posture calibration and online perception. The framework coordinates three stages: vision-language-guided navigation toward target objects, closed-loop posture calibration to align the arm-hand workspace, and dexterous manipulation with synchronized upper- and lower-body control. Online visual feedback re-grounds demonstrated interaction geometry under new object poses and scene configurations, while tactile feedback adapts hand motions to actual contact conditions. Each manipulation skill is specified by one human demonstration, without task-specific manipulation-policy retraining. Experiments across five long-horizon manipulation tasks demonstrate spatial, visual, and cross-object generalization, as well as recovery from external physical disturbances across all three stages.

0.20

Ctrl-F: Force as an Interface for Humanoid Loco-Manipulation

cs.RO cs.LG ·2026-11-03 ·Mengjie Zhao, Yuxin Chen, Yiyang Shao, Chuye Hong, Jiakang Chen +2 more
positioncode/data similarity 0.64
Abstract · Project page: https://forcectrl.github.io/

Humanoid motion tracking has scaled to general policies that reproduce diverse human motion, but the resulting behavior is in free-space: the tracker knows where the body should be, not how it should interact with the environment. Other work makes the object part of the objective, yet abstracts that interaction to contact or geometry, which leaves the demanding cases under-explored. We argue that force is a natural command channel between motion imitation and object interaction, one that specifies not just where to make contact but how hard. We present Ctrl-F, a hierarchical motion-force framework that augments motion tracking with an explicit force command, a 3-D force at each designated contact body such as a hand or a foot. The hierarchical framework comprises: (1) A unified motion-force controller tracks the reference motion while applying the commanded force to the world. (2) A high-level force planner that issues a force command online based on object state and proprioception observations. In simulation and on hardware, Ctrl-F completes loco-manipulation that demands large, sustained interaction force and adjusts its command online as the load changes, both out of reach of a force-blind tracker. On hardware, Ctrl-F demonstrates hand-based dragging, pushing, and lifting, as well as foot-based kicking, successfully manipulating boxes weighing up to 20kg in pushing and kicking tasks where the force-blind baseline fails, while adapting online to additional loads introduced mid-task.

0.20

GazeTune: Facilitating Precise Gaze-Driven Interactions with Cascaded Touch Input

cs.HC ·2026-09-01 ·Jina Kim, Eric J. Gonzalez, Yang Zhang, Sang Ho Yoon
top venuehot direction similarity 0.64
Abstract · 13 pages, 8 Figures, Accepted to UIST'26

Eye gaze has become an essential input for spatial computing, but its coarse targeting and saccadic nature limit precision and complicate continuous interactions such as dragging, especially under user motion. Gaze+pinch has also become standard in XR for its convenience, yet mid-air gestures remain imprecise, fatiguing, and socially unacceptable. These limitations underscore the need for an approach that preserves the speed of gaze while enabling stable, fine control. We present GazeTune, a cascaded multimodal interaction technique combining gaze and touch to refine gaze-based selection and manipulation. Touch serves as a refinement channel within gaze pointing, allowing precise cursor and target control. Our work investigates how gaze-and-touch enhances dragging and mitigates Motion-Induced instability. In a study (N=20), we compared GazeTune against gaze-only and gaze-pinch methods in 2D dragging. Results show that GazeTune achieves significantly lower error with comparable execution time, validating its effectiveness and balanced trade-off between time and accuracy.

0.47