CS Paper Radar

Navigation Explainability for Object Avoidance with LLMs for Assistive Robots

Courtney Young, Yi Lian, Taery Kim, Sehoon Ha

cs.RO ·submitted 2026-10-20 20:54 UTC · v1
published

Abstract

Blind and visually impaired individuals face uncertainty when navigating unfamiliar areas. Whether a cane or a guide dog is used for navigation, this may prove to be challenging. To mitigate this challenge, we present the Context-Aware Navigation Explanation framework that explains guide dog movements for obstacle avoidance and navigation using a large language model. This system integrates perceptual context with planned future trajectories to generate natural-language explanations of robot movement. We deploy the system on a Unitree Go2 platform in indoor navigation scenarios and evaluate how different explanation context configurations affect user trust and explanation satisfaction. Results show a strong positive relationship between explanation satisfaction and trust, and suggest that combining obstacle information with future trajectory leads to the most effective explanations. These findings highlight the potential of context-aware language explanations to improve transparency and trust in assistive robot navigation.

Closest papers

Nearest neighbours by meaning across the whole index.

A Multi-Layer Navigation System for Smart Wheelchairs with a Limited Input Interface

cs.RO ·2026-10-28 ·Piero Simonetto, Anna Polato, Francesco Pasti, Muhammad Rameez ur Rahman, Sebastiano Vascon +4 more
published similarity 0.66
Abstract · 8 pages, 6 figures, 3 tables. Presented at IROS 2026 (IEEE/RSJ International Conference on Intelligent Robots and Systems); to appear in the…

Smart wheelchairs are robotic devices that gather information from the surroundings and allow people to safely drive through various environments. They usually rely on the interpretation of high-level user's intentions, typically decoded through human-robot interface (HRI) or brain-machine interface (BMI). However, navigating in an unknown environment and accurately understanding user commands remain significant challenges. Misinterpretation of commands or the environment can lead to user frustration, distrust of the system, or even unsafe situations. In this paper, we present a novel Multi-Layer Navigation System (MLNS) for a smart wheelchair that enables navigation in unknown environments while interpreting and safely contextualizing user's intentions even in the case of limited and faulty commands. The stack relies on a Virtual Sensors-Driven Potential Fields (ViSPF) method for obstacle detection that accounts for the wheelchair's actual size, a Model Predictive Control (MPC) module for trajectory generation, and an open-vocabulary computer vision algorithm for target identification. Finally, a system based on Priority Queue (PQ) was implemented for target selection. We validated our system on a real smart wheelchair with experiments involving 10 participants without disabilities, including scenarios with faulty input interfaces. The results demonstrate the robustness and effectiveness of our system in real-world environments, proving its potential for practical applications in assistive technologies.

0.36

Explaining Navigation Decisions: Stage-Aware Visual Attribution for Vision-Language Navigation

cs.CV ·2026-10-25 ·Yiyuan Yao, Xinpeng Ding, Minglei Chen, Fei Gao, Lihuo He +4 more
similarity 0.65
Abstract

As Vision-Language Models (VLMs) increasingly enable robots to follow language instructions, understanding their navigation decisions becomes important for reliable deployment. Existing explainability methods overlook a key challenge in Vision-Language Navigation (VLN): evidence for the current action may be distributed across historical and current observations and associated with different instruction stages. We identify instruction-stage insensitivity, where existing methods produce similar explanations despite changes across different navigation stages. To address this issue, we propose a stage-aware attribution framework for VLN. We first construct action-conditioned temporal instruction--visual attributions across observations. These attributions are projected into instruction space to estimate stage associations, which guide the aggregation of relevant historical evidence for refining the current explanation. An instruction-guided spatial filter further suppresses isolated responses while preserving compact, instruction-relevant regions. Experiments across multiple VLN policies and benchmarks show improved attribution quality over existing methods. We further demonstrate that our method acts as a diagnostic tool for analyzing failure cases, and validate its applicability to real-world navigation on a Unitree G1 robot.

0.11

Risk-Aware Semantic Grounding for Trustworthy LLM-Based Robot Planning

cs.RO cs.AI ·2026-09-29 ·Łukasz Sobczak, Nur Keleşoğlu, Sławomir Piotr Nowak
benchmark similarity 0.64
Abstract

Large language models (LLMs) are increasingly used as high-level planners in robot navigation, but their outputs may become unreliable when instructions are ambiguous, unsupported by the environment, or semantically inconsistent. This paper presents a Risk-Aware Semantic Grounding framework for trustworthy LLM-based robot planning. Unlike existing LLM-based planners that primarily optimize plan generation, we formulate semantic grounding reliability as a multi-dimensional risk estimation problem. The proposed architecture explicitly models grounding uncertainty through ambiguity, hallucination and semantic-conflict risks before planning occurs, enabling the system to decide whether to execute the instruction, request clarification, or reject it. To evaluate the approach, we introduce TRUST-NAV, a benchmark containing both standard navigation tasks and risk-inducing instruction scenarios. Experimental results show that while conventional LLM planners achieve strong performance on valid navigation tasks, the proposed framework substantially improves ambiguity detection and semantic conflict rejection. These findings suggest that trustworthy robot planning should be evaluated not only by task completion, but also by the ability to recognize when execution should not occur.

0.42

From Wizard-of-Oz Human-Robot Dialogue Collection to a Taxonomy of Robot Response Decisions: A Retrospective Analysis of Assistive Pilot Interactions

cs.RO ·2026-09-16 ·Guangping Liu, Nicholas Hawkins, Tipu Sultan, Flavio Esposito, Madi Dian
published similarity 0.63
Abstract · Accepted at the Human-Robot Dialogue (HRD) Workshop, held by the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (I…

Robots that follow natural-language instructions in everyday indoor environments must act on incomplete human utterances. Instructions often omit essential information, such as the identity of an out-of-view object, an intended destination, or the user's goal. Existing datasets contain little real-world situated dialogue and provide few practice-grounded criteria for deciding when a robot should act, confirm, clarify, or refuse. We retrospectively analyze a pilot Wizard-of-Oz study in which five participants performed everyday indoor tasks, including door opening, drawer opening, feeding, drinking, and cleaning, with a wheelchair-mounted mobile manipulator while the wizard responded without a formal communication policy. This preserved authentic user behavior but produced inconsistent robot-side decisions, motivating an explicit decision scheme. From 40 episodes, we derived a hierarchical taxonomy of six response modes (ANSWER, REPORT_DONE, REFUSE, CONFIRM, CLARIFY, ACT) and four ambiguity types (intent, referential, spatial, intelligibility). Two human annotators and an AI annotator applied the scheme to the pilot data. Clean-label rates were 91% and 89%, and Cohen's ranged from 0.72 to 0.95 across decision-point, mode, and ambiguity levels for both human-human and human-AI comparisons. Fine-tuning LLaVA-1.6-7B on taxonomy-derived labels for ACT and CLARIFY indicates the feasibility of training vision-language models using annotations from our taxonomy. Remaining boundary cases in decision-point identification and REPORT_DONE motivate a constrained protocol for more consistent dialogue collection.

0.34

Time-Aware Assistive Navigation

cs.CV cs.RO ·2026-09-04 ·Masaki Kuribayashi, Zhongkai Shangguan, Eshed Ohn-Bar
top venuebenchmarkcode/data similarity 0.63
Abstract · ICRA 2026

Can interactive vision-and-language agents learn not just what to say but also \textbf{\textit{when}} to say it? Current language models rarely plan over whether and when to realize a real-time response to a user. However, providing accurate and timely support for human decision-making, such as when guiding visually impaired individuals through urban environments, requires careful real-time responsiveness--poorly timed responses can distract users or add unnecessary cognitive load. As a machine intelligence challenge for Multimodal Large Language Model (MLLM)-based agents, we introduce a large-scale multimodal benchmark for an egocentric, assistive navigation task in complex outdoor environments. Using this benchmark, we uncover a fundamental limitation of off-the-shelf MLLMs in delivering safe and time-sensitive navigation instructions, even with model fine-tuning on substantial amounts of data. We then demonstrate that a simple yet effective modification of the model, including direct supervision to predict the underlying reason for each instruction, yields significant performance gains across open-loop, closed-loop, and sim-to-real generalization settings. However, our analysis highlights persistent challenges in temporal reasoning, safety-critical object awareness, and relational and distance understanding. To advance the development of scalable assistive agents, we will release our simulation, benchmark, and code (available at the project website: https://timeli-icra.github.io/).

0.46

Lightweight and Resource-Efficient Perception for Robotic Guide Dogs

cs.DC cs.CV ·2026-10-02 ·Jinse Kwon, Yoojin Lim, Choonghan Lee, Yongseung Yu, Yongin Kwon +1 more
publishedbenchmark similarity 0.63
Abstract · accepted in ACCV 2026

Robotic guide dogs should understand their surroundings, objects, and potential risks. Prior research has focused on raw sensor data from cameras and 2D or 3D LiDAR, which precisely measure distance points rather than provide a semantic understanding of the scene. While these physical measurements are effective for robot-centric collision avoidance and robot safety, they are not suitable for human-centric guidance. The system should recognize the type and relevance of obstacles and explain them, clearly and actionably, in terms of their spatial relation to the user. We present complete on-device perception modules that fuse a 360 camera and a 2D LiDAR for reliable collision avoidance, with moving-object detection and tracking for human-centric guidance. Finally, in walking-impossible situations, a vision--language model delivers pathway explanations as a safety mechanism to reduce user anxiety. In experiments, verification of fused 360 camera--LiDAR depth shows reliable near-range perception but inherent mid-range bias, while the system as a whole sustained real-time performance under 55 W. On the real-world egocentric GuideDogQA benchmark, our system achieved 83.8\% accuracy, compared with 67.1\% for GPT-4o. These results demonstrate that practical human-centric guidance with real-time on-device inference is feasible even on quadrupeds.

0.21

APPROVE: Visual End-User-in-the-Loop Robot Programming with LLMs

cs.RO cs.HC ·2026-08-19 ·Bijan Kavousian, Miray Özakkas, Josefine Monnet, Oliver Petrovic, Christian Brecher
published similarity 0.62
Abstract · Accepted for publication in Procedia CIRP, Proceedings of the 20th CIRP Conference on Intelligent Computation in Manufacturing Engineering (…

Programming robots remains challenging for non-experts, as traditional methods require expert knowledge and even block-based interfaces often lack flexibility. Recent work has explored Large Language Models (LLMs) to automatically generate robot programs from natural language, but these systems remain limited by a lack of transparency, missing mechanisms to ensure alignment with user intent, and little support for reuse. We present APPROVE (AI-Powered Programming for Robots with Visual End-User Feedback), an LLM-based multi-modal end-user programming framework that integrates natural language input with a block-based interface and an explicit user confirmation step. Generated programs are visualized using a block-based interface in Blockly, allowing users to confirm, modify, or reject them before execution. Confirmed functions are stored in a library for reuse, gradually building a set of reliable program components. Our approach contributes a human-centered design for LLM-based robot programming that emphasizes user trust, intent alignment, and reusability.

0.37

InSight: A Mobile Assistant Helping Visually Impaired People Navigate Unfamiliar Environment

cs.HC ·2026-10-13 ·Junwei Zheng, Jiale Wei, Di Wen, Yufan Chen, Ruiping Liu +4 more
similarity 0.62
Abstract

Navigating unfamiliar outdoor environments is one of the most demanding everyday tasks for people with visual impairments (PVI). However, existing navigation systems cannot simultaneously satisfy the needs of informative perception and precise localization during navigation. This paper explores the core navigation demands of PVI through a preliminary study. Based on the results of the pre-study, we present the design of InSight, a mobile assistant that combines scene descriptions with precise localization to support PVI navigating unfamiliar outdoor environments. The offline mode of InSight runs entirely on device for real-time navigation, while the online mode streams camera frames to a self-hosted server for arbitrary object detection and accurate localization. Finally, a user study with 21 visually impaired participants validates that InSight could assist users in navigating unfamiliar outdoor environments with increased success rate and confidence compared to mainstream navigation applications like Apple Maps.

0.18