CS Paper Radar

Adaptive Question Generation for Ambiguous Robot Instructions Based on Information Gain and Human Cognitive Characteristics

Ryo Terashima, Yuga Yano, Hakaru Tamukoh

cs.RO ·submitted 2026-09-19 12:36 UTC · v1
published

Abstract

In this study, we propose an adaptive question generation mechanism that addresses uncertainty arising from ambiguous instructions. Natural-language instructions for service robots often contain ambiguities. In scenes with multiple similar objects, the robot must identify the target object by asking questions. However, existing question generation methods focus on probabilistic discrimination efficiency and do not sufficiently account for user ease of response. Asking questions without considering the user's ease of response may cause incorrect or uncertain answers. Such incorrect or uncertain answers lead to additional clarification turns and degraded dialogue efficiency. The proposed mechanism integrates probabilistic discrimination efficiency and human cognitive characteristics to generate questions that users can intuitively and easily answer. First, the proposed mechanism extracts visual attributes of candidate objects using a vision-language model (VLM). Next, the proposed mechanism computes the information gain (IG) of each attribute and selects the visual attribute (e.g., color, shape) that most reduces uncertainty in the candidate set. When multiple attributes have the same IG, the proposed mechanism applies a visual attribute filter based on cognitive characteristics to select the visual attribute that users can most intuitively perceive. The proposed mechanism then prompts the VLM with the selected attribute and the corresponding attribute values of each candidate object to generate a concise natural-language question. This process is repeated until the target object is uniquely identified. The proposed mechanism achieved a target identification success rate of 88.5% in the evaluation using a dataset of 120 scenes. Furthermore, the proposed mechanism received high ratings in subjective evaluations of response ease.

Comment
8 pages, accepted as a conference paper for 2026 35th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN 2026)

Closest papers

Nearest neighbours by meaning across the whole index.

CLARA: A Cognitively-Driven Clarification Framework for Grounding Ambiguous Requests

cs.RO cs.AI ·2026-11-03 ·Dimitrios Bekris, Marios Glytsos, Panagiotis P. Filntisis, Niki Efthymiou, George Retsinas +1 more
similarity 0.77
Abstract · Work conducted in early 2025

Robotics and AI systems are increasingly deploying large vision-language models (LVLMs) to follow high-level instructions, from routine household chores to more complex organizational tasks. Yet, broad directives like "clean the room" or "organize the desk" frequently omit crucial details about user preferences and specific arrangement of objects that influence the desired outcome. Without any further guidance, such models tend to overestimate their grasp of ambiguous instructions and frequently assume they have sufficient understanding despite missing important details. In this paper, we propose CLARA (Conversational cLArification for Robotic Arrangement), a cognitively-driven clarification framework that proactively elicits key information from users, in a similar fashion as humans' clarification seeking behavior. Through targeted questioning, CLARA refines ambiguous requests into structured, detailed instructions describing precisely how objects should be organized or rearranged, according to the user's preferences. These clarifications then guide a diffusion-based image generation pipeline, producing realistic visual renderings of the final, desired scene. This final rendering also serves as a visual confirmation of the system's understanding from the user. Experimental results and user studies demonstrate that CLARA leads to more accurate and consistent outcomes compared to another State-of-the-Art LVLM system, highlighting the importance of interactive clarification for bridging the gap between vague directives and fully grounded task specifications.

0.11

AmbiSteer: Selectively Eliciting Ambiguity-Aware Responses in Vision-Language Models

cs.CV cs.CL ·2026-11-13 ·Monica Riedler, Samuele Antonelli, Barbara Plank, Raffaella Bernardi
similarity 0.70
Abstract · 23 pages, 23 figures

Vision-language models (VLMs) often exhibit overconfidence, providing firm answers even when visual evidence is insufficient. A prominent case is referential ambiguity in Visual Question Answering (VQA), where multiple entities in an image are plausible referents of a question. Addressing such cases requires recognizing when ambiguity is present: ambiguous questions can be handled by asking for clarification or enumerating plausible referents, while unambiguous questions should receive a direct answer. Current VLMs struggle with this distinction, often defaulting to a single interpretation. To investigate whether activation steering can serve as an inference-time mechanism to elicit such behaviors, we introduce AmbiSteer, a lightweight framework to selectively elicit ambiguity-aware behavior. Across three VLMs, we find that steering alone has limited and model-dependent effects, while prompting is generally more effective. Combining prompting and steering substantially increases clarification and enumeration behaviors, but also elicits them on unambiguous cases. AmbiSteer addresses this using a linear classifier to predict input ambiguity and condition the steering direction accordingly, increasing the proportion of inputs handled appropriately from 59.0% to 77.0% on average across models. Our work sets the ground for lightweight approaches to reduce model overconfidence in uncertain multimodal settings.

0.15

Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language

cs.RO cs.AI ·2026-09-24 ·Zachary Ravichandran, Jonathan Diller, Fernando Cladera, Varun Murali, George J. Pappas +1 more
published similarity 0.65
Abstract · Accepted to the International Symposium of Robotics Research (ISRR) 2026

Foundation models provide robots with the ability to interpret natural language and reason about environmental context, yet most language-conditioned policies assume that goals are well-specified and that task-relevant information is provided upfront via a prior map. Operating in unfamiliar environments with underspecified tasks entails high contextual uncertainty: the robot must jointly infer what constitutes task success, what constitutes relevant information, and where (or whether) that information exists. We address these limitations via CLUE (Closed-Loop contextual Uncertainty rEsolution), a framework for actively resolving contextual uncertainty given underspecified tasks in natural language. CLUE uses an LLM-derived policy to hypothesize task-relevant concepts and potential plans. It then uses a language-embedded map, which is constructed online, to ground these hypotheses into actions. The policy sequentially evaluates hypotheses via closed-loop environment interaction and refines its plans as it gathers new information. We deploy CLUE on a Boston Dynamics Spot across three real indoor and outdoor environments spanning 15 tasks that require object disambiguation, functional inference, and occlusion reasoning. CLUE achieves a success rate within 7 percentage points of an oracle policy and outperforms an LLM-enabled planner without closed-loop feedback by a 4x margin. Supporting experiments demonstrate that simply building and then querying a language-enriched map is insufficient to resolve complex contextual planning tasks; these approaches achieve roughly one third the success rate of CLUE while requiring over 10x more VLM tokens. We provide additional information at https://zacravichandran.github.io/CLUE.

0.56

Evaluating VQA in Vision Language Models using Cooperative Principles

cs.CL ·2026-10-02 ·Monika Shah, Sudarshan Balaji, Somdeb Sarkhel, Sanorita Dey, Deepak Venugopal
similarity 0.65
Abstract

We evaluate the performance of Vision Language Models in Visual Question Answering (VQA) when questions violate Grice's maxims. To do this, we use VLMs to generate question modifiers that add non-essential, ambiguous or false information and show that in the presence of such violations, the VLMs that we evaluate (ChatGPT, Claude, Gemini and Llava) show diminished performance. Further, we empirically show the difference between how humans reason pragmatically compared to VLMs, and the difference in VLM reasoning when it resolves violations that are human-induced compared to those that are AI-generated. Finally, we show that human cognitive effort (measured through time-on-task in an experiment) is lower for resolving VLM-induced violations, but VLMs themselves perform less accurately in such cases.

0.18

Learning to Clarify Underspecified Intents Under Limited Interaction

cs.AI ·2026-10-03 ·Pranav M R, Manuel Cherep, Pattie Maes, Nikhil Singh
similarity 0.65
Abstract · 43 pages, 15 figures

AI assistants receive requests that leave out information needed for a good outcome, for example about users' preferences or goals. They must then either speculate or ask for more information before proceeding. We reconceptualize this as a value-of-information problem: the assistant should acquire information whose absence causes the greatest avoidable loss in user utility. This is rarely known ex ante; rather, assistants must predict it in order to optimally allocate limited user interactions. We instantiate this problem in image generation and derive a reinforcement learning framework using multi-turn simulated users to maximize utility recovery under uncertainty. In a preregistered study with 456 interactive sessions across 76 human participants, this helped users significantly better match reference images with significantly fewer questions, less total interaction time, and lower cost. This points toward a simple and scalable framework for training language model assistants to better disambiguate user intent by asking more informative questions.

0.31

Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards

cs.RO cs.AIcs.HCcs.LG ·2026-10-15 ·Helena Merker, Nick Walker, Andreea Bobu
published similarity 0.65
Abstract · Accepted to the 10th Conference on Robot Learning (CoRL 2026), Austin, TX. 18 pages, 8 figures, 1 table

Reward learning typically assumes the human's input covers everything that matters for the task. In practice, humans can only attend to so much at once, so their teaching naturally captures some features---or aspects of the task---better than others. Prior work has shown how the robot can identify which features were poorly supervised directly from its teaching data. But the current solution is to ask the human to try again, putting the burden on someone who already struggled to convey those features. We propose shifting this burden to the robot. Our key insight is that the same information that tells the robot what it doesn't know can help it ask better questions. Preference queries offer a low-effort channel for resolving this uncertainty, but only if the robot knows where to direct them and the human knows what to attend to when answering. We act on this insight in two ways: we build an informed prior that targets preference queries at the features that need resolving, and we accompany each query with an explanation that focuses the human's attention on exactly those features. In simulation, the informed prior beats the standard active preference learning baseline by up to $30.8\%$ in the low-query regime, where reducing human effort matters most. In a user study, the informed prior and explanations each significantly improve reward recovery and amplify each other.

0.28

SAIN: Structure-Aware Interactive Navigation with Active Dialogue Grounding for Mobile Robot

cs.RO ·2026-08-10 ·Yuhao Cao, Xiao Liu, Yang Xie, Lu Liu, Haoyao Chen
benchmark similarity 0.64
Abstract · The authors have decided to withdraw this preprint due to unresolved internal disagreements regarding its public release at this stage

Most existing vision-language navigation tasks assume that instructions are complete and unambiguous. However, real-world robots often encounter natural human instructions that are ambiguous, underspecified, or incomplete, requiring them to resolve such uncertainties through active questioning. Interactive Instance Goal Navigation (IIGN) requires an embodied agent to find the specific instance under an ambiguous category-level instruction through active dialogue. However, existing dialogue-enabled methods often consume oracle answers as transient textual context for immediate decisions, rather than persistent spatial or object-centric structured state. We present SAIN, a zero-shot framework that turns active dialogue into persistent navigation state. Instead of consuming oracle answers as one-step text hints, SAIN compiles them into target evidence, route-level corridor memory, and object-candidate labels. These states are stored in structured value, room, graph, and object memories, then consumed by a unified policy for frontier ranking and final target approach. On the VL-LN IIGN benchmark, SAIN improves SR from 20.2 to 25.4 and SPL from 13.07 to 14.17 over the strongest reported dialogue-enabled baseline, while requiring no task-specific policy training. The results support dialogue-to-state conversion as an effective zero-shot mechanism for long-horizon interactive instance navigation. Project website: https://zorattc.github.io/SAIN/

0.17

Active Perception for Embodied Disambiguation

cs.AI cs.RO ·2026-08-11 ·Yiwei Liu, Luwei Yang
similarity 0.64
Abstract

Natural language provides robots with a flexible task interface, but target ambiguity in embodied environments arises not only from user intent; it can also result from missing taskrelevant physical evidence in the current observation. Existing interactive disambiguation methods primarily obtain additional information by asking the user, whereas occlusion, restricted viewpoints, unreadable text, and unobserved targets require the robot to actively change its observation. We propose an active-perception framework for embodied target disambiguation that uses active observation as the backbone for information acquisition and uses a vision-language model to decide, on the basis of accumulated visual evidence and interaction information, whether to continue observing, request clarification, or complete target selection. Active observation can both directly recover missing discriminative evidence and reveal object names, labels, and semantic attributes, thereby improving user clarification when it remains necessary. Real-robot experiments show that the framework combines physical information acquisition and userintent clarification within a unified embodied disambiguation process.

0.04