CS Paper Radar

Cross-Modal Guidance for Out-of-View Object Search in Simulated Prosthetic Vision

Adyah Rastogi, Apurv Varshney, Tobias Höllerer, Michael Beyeler

cs.HC ·submitted 2026-09-01 15:45 UTC · v1

Abstract

Out-of-view guidance is well established in virtual and augmented reality, but its effectiveness may depend on the visual bandwidth available to the user. We test this under simulated prosthetic vision (SPV), where visual guidance must share the same sparse representation used to inspect the scene. Nineteen participants performed object search under two SPV conditions differing in electrode density and phosphene spread (10x10 and 20x20) and four guidance conditions (no guidance, visual, haptic, audio) all driven by the same horizontal target-offset variable. All three modalities reduced search time and head movement. The tested auditory and haptic cues produced approximately 25% faster overall search and 11-13% faster target acquisition than the visual cue, despite similarly direct orienting trajectories. The tested haptic and auditory cues also shortened post-acquisition search. Final head-target angular offset was reduced substantially more in the 10x10 SPV condition; there, all three cues also reduced vertical localization error by approximately 45-58% despite providing no elevation information. Under severe visual constraints, guidance performance depended on cue implementation and search stage.

Closest papers

Nearest neighbours by meaning across the whole index.

Tactile Search: Enhancing Targeting in 3D Space

cs.HC ·2026-09-05 ·Amber Maimon, Iddo Yehoshua Wald, Jonas Keppel, Eunhee Chang, Yoshifumi Kitamura +3 more
top venue similarity 0.63
Abstract · Accepted to the 2026 IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2026)

Visual search is crucial in daily life, from scanning for relevant information to spotting signs of danger. When sensory channels are overloaded or degraded, cognitive tasks can be supported by crossmodal information representations through vibrotactile cues. We introduce Tactile Search, an approach that uses modulation of frequency and amplitude of vibrations to the hands, for guiding attention to the location of objects in 3D space. We evaluated this approach in a competitive VR game where participants searched for targets using both vision and touch. Across two studies -- an in-the-wild demonstration (n=55) and a controlled laboratory experiment (n=28) -- we found that vibrotactile feedback significantly improved performance and increased user confidence. In the combined haptic condition, performance did not differ across target heights. We further analyzed participants' subjective experiences and search strategies highlighting the benefits of the tactile cues. Our findings suggest that Tactile Search can enhance interaction and provide design considerations for integrating haptic search into interactive systems.

0.24

TACIT: Tactile Contact Supervision for Spatial Attention in Dexterous Manipulation

cs.RO ·2026-09-21 ·Yanhou Lai, Fucai Zhu, Ruiqiang Wang, Koichi Hashimoto
benchmark similarity 0.60
Abstract

Visuomotor policies trained from a few demonstrations may reproduce demonstrated trajectories without reliably following changes in object position. Existing approaches with explicit attention typically obtain spatial priors from human annotation or visual models. We introduce TACIT (tactile contact informs attention), which uses measured tactile contacts from teleoperated demonstrations to supervise spatial attention without additional point annotation. Gaussian targets over preceding camera point clouds supervise an attention head whose pooled output conditions a visuotactile diffusion policy. Targets are used only during training; tactile observations remain inputs at inference. In the primary real-robot benchmark, with ten demonstrations per task and five demonstrated placement regions, TACIT achieves 66.7% success on ball placement and 73.3% on peg insertion, compared with 10.0% and 20.0% for input-matched 3D visuotactile fusion and 20.0% and 43.3% for vision-only DP3. TACIT enters the 150 mm palm-to-object approach region within 12 seconds in all 30 trials per task; all remaining failures occur after arrival. Across three training seeds on real ball and simulated peg, TACIT outperforms input-matched fusion and an architecture-matched control without explicit attention supervision, supporting the contribution of supervision beyond branch capacity. Pre-contact and contact-time supervision show no consistent ordering. These results demonstrate that measured tactile contact provides effective spatial supervision for approach behavior from few demonstrations within the evaluated workspace.

0.21

Smart Navigation for Visual Prostheses in Virtual Reality: An End-to-End Framework for Priority-Based Scene Translation and Path Guidance

cs.HC ·2026-10-05 ·Mohamed H. Abdellatif, Fatma S. Elsharkawy, Nouran H. Qassem, Talal M. Emara, Nada K. Kotb +2 more
similarity 0.60
Abstract

Visual prosthetics provide a promising direction for partial restoration of functional vision for people with total retinal blindness. However, existing systems face significant challenges in translating complex visual scenes into meaningful perceptions due to limited spatial resolution, leading to difficulties in scene understanding. Furthermore, existing solutions don't adequately account for user requirements and concerns, and this creates a significant gap between user expectations and the developed solutions. To address these gaps, we conducted interviews with 10 blind human subjects. These interviews essentially indicated that the main key challenge for these people is outdoor navigation. In this paper, we present an end-to-end smart navigation system for visual prostheses users. Our approach employs a three-stage framework. First, an object detector is implemented to identify and localize points of interest for navigation tasks in pedestrian environments, and simultaneously generate the safest paths available to the users. Second, the detected objects are abstracted into simple geometric shapes suitable for low-spatial-resolution vision. The detected objects are filtered based on a multi-criteria priority scoring function. Finally, this information is encoded into optimized stimulation parameters, which are fed into the visual prosthesis implants to generate enhanced phosphene representations for obstacle avoidance and path planning. This whole system is validated with sighted participants using virtual reality to simulate outdoor navigation. Our smart navigation system improves user independence while taking into account the limitations of visual prosthetics.

0.12

Assessing the Marginal Utility of Exocentric Head Views in UMI-Style Large-FoV Wrist-Camera Manipulation

cs.RO ·2026-10-08 ·Yuhui Duan, Shi Jin, Yuntian Wang, Jin Wang, Siao Liu +3 more
similarity 0.60
Abstract

Multi-view perception is often treated as a safe design choice for robot manipulation: if one camera is useful, more cameras should be better. We revisit that assumption in tabletop bimanual manipulation by comparing policies trained with two large-field-of-view (FoV) wrist cameras against policies that additionally receive a head view, using controlled-variable experiments on a real bimanual platform. Across three tabletop tasks, adding a head view does not consistently improve task completion or execution quality: when the wrist cameras already observe the objects, grippers, and goal regions, wrist-only policies form a strong baseline. Restricting the wrist-camera FoV increases the potential value of a head view, but performance improves most reliably when the added view supplies concentrated, task-relevant information: region-of-interest (ROI) cropping yields more consistent head-view gains than the uncropped image, which is dominated by background and task-irrelevant regions. The data-scaling results further suggest an interaction between FoV and demonstration count: wide observations improve coverage but may require more data to stabilize, and they may reach a higher performance ceiling once enough demonstrations are available. These results suggest that camera design should be framed as a task-relevance and information-density problem rather than as a simple count of input views.

0.12

Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments

cs.HC cs.AI ·2026-09-18 ·George Xi Wang, Xiangyu Li, Shaoyue Wen, Jiaqian Hu, Junan Xie +6 more
similarity 0.59
Abstract · 12 pages, including figures and references

Blind and low-vision users often face challenges when locating and physically acquiring objects in unfamiliar indoor environments. Existing vision-language-model-based assistants can provide semantic descriptions but may introduce latency, hallucinations, and guidance that is poorly aligned with embodied action. We present Touvigation, a hands-free object acquisition system that combines vision-language understanding with persistent local spatial modeling to provide low-latency, body-relative guidance. Drawing on formative interviews with eight blind and low-vision participants, we design a multi-stage guidance framework that adapts spatial references as users transition from orienting, to walking, to reaching and tactile verification. We evaluated Touvigation with 12 blind and low-vision participants against a multimodal large-language-model assistant and unassisted search. Touvigation achieved 100% task success, compared with 58% for the multimodal assistant and 85% for unassisted search, while reducing completion time and cognitive workload. Our findings demonstrate how persistent spatial grounding and adaptive embodied guidance can improve object acquisition for blind and low-vision users.

0.27

Looking Around by Looking Around: Omnidirectional Gaze-based VR Viewport Control

cs.HC ·2026-08-30 ·Hock Siang Lee, Jinghui Hu, Florian Weidner, Haopeng Wang, Hans Gellersen
published similarity 0.59
Abstract

Traditional VR viewport control primarily relies on head and torso movement, which can be effortful and limiting in both constrained and extended-use settings. We introduce Looking Around by Looking Around (LALA), a gaze-based VR pitch-and-yaw viewport control technique designed for natural and effortless omnidirectional exploration via eye movements, without requiring or obstructing movement of the head, hand, or body, offering a low-effort and highly accessible interaction method. Because gaze is primarily used for perception and exhibits oculomotor and perceptual asymmetries, using it directly for control is difficult. To address this, we designed an asymmetric omnidirectional control profile for the eye, then built on it to exploit tendencies for eyes to stay within comfortable regions for viewport control. We evaluated LALA in a user study (N=18) featuring two contrasting tasks: alignment towards known directions and open-ended visual search towards unknown directions. LALA was strongly preferred over the traditional baseline, achieving competitive performance while enabling fully hands-free interaction with minimal physical movement.

0.36

Visual Cue Interactions in AR-Guided Needle Insertion: A Prostate Biopsy-Inspired Phantom Study

cs.GR ·2026-08-27 ·Xinrui Zou, Mingxu Liu, Thomas T. Jones, Braden Millan, Sandeep Gurram +3 more
published similarity 0.58
Abstract · Accepted at IEEE ISMAR 2026. The definitive version will be available in IEEE Xplore

Despite the apparent simplicity of the motor action involved during percutaneous needle procedures, manipulating the tool's direction becomes challenging when clinicians cannot directly visualize internal anatomy and must rely on ultrasound images, which increase cognitive demand. Augmented reality (AR) offers the promise to assist with these tasks by providing pertinent visual information in the clinician's field of view. However, simply providing visual information that ignores meaningful visual cues can complicate depth perception and spatial understanding. In this work, we introduce and evaluate three visualization techniques for needle alignment developed during design sessions with medical experts: a localized focus-and-context window, a color-based proximity encoding, and an explicit trajectory overlay. These techniques were evaluated in a user study (n=26) including clinical experts (n=7) using a prostate-biopsy-inspired phantom. Results from this study suggest that cue effects depended on the surrounding cue configuration and user expertise. For novices, explicit trajectory overlay improved targeting accuracy and reduced retreat behavior, but its effect on completion time varied across cue configurations, with slower performance when the overlay was presented alone. For experts, the focus-and-context window reduced completion time and retreat events, while color-based proximity overlay improved completion time. Subjectively, color cues were often perceived as helpful even when their effects on accuracy were not consistent. These results suggest that AR guidance strategies for percutaneous interventions should consider user expertise and visual context.

0.41

Effects of Locomotion and Viewpoint Control on Immersive Robot Telepresence for Navigation and Inspection

cs.RO cs.HC ·2026-11-18 ·Ammaar Zaman, Neven ElSayed, Oliver Prentner, Mohamed Ellethy, Mostafa M. Lotfy +2 more
similarity 0.58
Abstract

In many robot telepresence interfaces, movement and viewing are managed through handheld inputs. This can divide attention between steering, looking, and inspection. We compared joystick versus omnidirectional treadmill locomotion and joystick-based versus head-guided camera control in a within-subject 2x2 study in which 17 participants used a virtual reality headset to guide a quadruped robot through a physical maze while locating marked wall symbols and capturing them as visual evidence. Mixed-effects analyses showed that head-guided camera control was associated with faster completion, lower wall-contact cost, lower workload, and more favorable presence, usability, technology acceptance, and flow than joystick-based camera control. The strongest overall objective performance was observed in the joystick locomotion plus head-guided camera condition, whereas the omnidirectional treadmill plus head-guided camera condition emerged as the strongest experiential compromise. Exploratory camera analyses showed that head-guided camera control reduced switching, increased reliance on the pan-tilt view, and supported a more stable viewing strategy across the maze. These strategy differences were consistent with the performance advantage, especially for wall-contact duration. Targets captured remained near ceiling and simulator sickness remained low across all conditions. Overall, the results indicate that viewpoint control was the stronger differentiator in this combined navigation and inspection task, while locomotion still shaped the objective and experiential tradeoff.

0.29