CS Paper Radar

Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards

Helena Merker, Nick Walker, Andreea Bobu

cs.RO cs.AIcs.HCcs.LG ·submitted 2026-10-15 17:41 UTC · v1
published

Abstract

Reward learning typically assumes the human's input covers everything that matters for the task. In practice, humans can only attend to so much at once, so their teaching naturally captures some features---or aspects of the task---better than others. Prior work has shown how the robot can identify which features were poorly supervised directly from its teaching data. But the current solution is to ask the human to try again, putting the burden on someone who already struggled to convey those features. We propose shifting this burden to the robot. Our key insight is that the same information that tells the robot what it doesn't know can help it ask better questions. Preference queries offer a low-effort channel for resolving this uncertainty, but only if the robot knows where to direct them and the human knows what to attend to when answering. We act on this insight in two ways: we build an informed prior that targets preference queries at the features that need resolving, and we accompany each query with an explanation that focuses the human's attention on exactly those features. In simulation, the informed prior beats the standard active preference learning baseline by up to $30.8\%$ in the low-query regime, where reducing human effort matters most. In a user study, the informed prior and explanations each significantly improve reward recovery and amplify each other.

Comment
Accepted to the 10th Conference on Robot Learning (CoRL 2026), Austin, TX. 18 pages, 8 figures, 1 table

Closest papers

Nearest neighbours by meaning across the whole index.

SPEAR: Selective Preference Elicitation with Adaptive Rewards for VLM-Guided Robotic Manipulation

cs.RO ·2026-10-15 ·Zewen Wu, Ruizhe Liu, Pei Zhou, Feng Chen, Yanchao Yang
publishedcode/data similarity 0.67
Abstract · 8 pages, 4 figures. Accepted at IROS 2026. Project page: https://zevinwu.github.io/spear-project

Recent Vision-Language Models (VLMs) enable automated preference labeling for robotic manipulation, yet their deployment remains impractical since each query incurs substantial inference cost, while VLMs frequently abstain on visually challenging comparisons. We propose SPEAR, a framework that transforms VLMs into efficient preference oracles via two synergistic mechanisms. First, we design distinguishability-aware query selection to reduce abstention rates by targeting only informative and distinguishable comparisons. Second, we deploy an adaptive reward transition that bootstraps exploration with VLM-synthesized guidance before preference-based refinement. Experiments on the manipulation benchmarks demonstrate that SPEAR achieves 88.9% average success, outperforming baselines by 23.3 percentage points, while significantly reducing abstention and wall-clock training time on contact-rich tasks. SPEAR learns three Mobile ALOHA manipulation tasks fully online on real hardware, demonstrating end-to-end real-world applicability. By addressing query efficiency and exploration challenges, SPEAR establishes VLMs as practical preference oracles for scalable robot learning without manual reward engineering.

0.45

What Does It Take For A Generalist Robot Policy To Adapt Using Minimal Data?

cs.RO cs.LG ·2026-08-11 ·Shreyas Kowshik, Sreyas Venkataraman, Leo Wang, Niharika Pant, Max Simchowitz +1 more
similarity 0.66
Abstract

A central goal in robot learning is to move beyond task-specific human data collection toward robots that improve through autonomous interaction. Yet fully autonomous learning remains difficult with current policies: sparse rewards and weak zero-shot exploration often prevent online learning from discovering successful behavior. As a first step towards fully autonomous learning, we propose the minimal-data adaptation regime, in which a pretrained robot policy must learn a new task from as little as one demonstration followed by autonomous online interaction. We establish that, although pretrained policies fail zero-shot on many new tasks, a single demonstration suffices to direct their behavioral prior to coarse task-following behavior, albeit with unreliable performance. From this insight, we isolate the key design principles required to convert coarse task-following into reliable task success. These include (1) the surprising efficacy of pretrained VLA representations to support sample-efficient value learning from online interaction (2) the sufficiency of residual policy learning to learn corrective behavior and (3) the need for careful data-balancing to prevent exploration collapse. As a representative of these principles, we propose MiDAS, a simple offline-to-online RL recipe that combines few-demonstration behavior cloning with value-based residual learning over frozen VLA representations. Across LIBERO and RoboCasa tasks, MiDAS substantially improves performance from single-demonstration initializations, including policies that succeed only rarely, validating the principles highlighted above. On a bimanual YAM platform, starting from a brittle initial policy, MiDAS improves task success after approximately six hours of online interaction.

0.34

Adaptive Question Generation for Ambiguous Robot Instructions Based on Information Gain and Human Cognitive Characteristics

cs.RO ·2026-09-19 ·Ryo Terashima, Yuga Yano, Hakaru Tamukoh
published similarity 0.65
Abstract · 8 pages, accepted as a conference paper for 2026 35th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN 202…

In this study, we propose an adaptive question generation mechanism that addresses uncertainty arising from ambiguous instructions. Natural-language instructions for service robots often contain ambiguities. In scenes with multiple similar objects, the robot must identify the target object by asking questions. However, existing question generation methods focus on probabilistic discrimination efficiency and do not sufficiently account for user ease of response. Asking questions without considering the user's ease of response may cause incorrect or uncertain answers. Such incorrect or uncertain answers lead to additional clarification turns and degraded dialogue efficiency. The proposed mechanism integrates probabilistic discrimination efficiency and human cognitive characteristics to generate questions that users can intuitively and easily answer. First, the proposed mechanism extracts visual attributes of candidate objects using a vision-language model (VLM). Next, the proposed mechanism computes the information gain (IG) of each attribute and selects the visual attribute (e.g., color, shape) that most reduces uncertainty in the candidate set. When multiple attributes have the same IG, the proposed mechanism applies a visual attribute filter based on cognitive characteristics to select the visual attribute that users can most intuitively perceive. The proposed mechanism then prompts the VLM with the selected attribute and the corresponding attribute values of each candidate object to generate a concise natural-language question. This process is repeated until the target object is uniquely identified. The proposed mechanism achieved a target identification success rate of 88.5% in the evaluation using a dataset of 120 scenes. Furthermore, the proposed mechanism received high ratings in subjective evaluations of response ease.

0.37

Learning to Clarify Underspecified Intents Under Limited Interaction

cs.AI ·2026-10-03 ·Pranav M R, Manuel Cherep, Pattie Maes, Nikhil Singh
similarity 0.62
Abstract · 43 pages, 15 figures

AI assistants receive requests that leave out information needed for a good outcome, for example about users' preferences or goals. They must then either speculate or ask for more information before proceeding. We reconceptualize this as a value-of-information problem: the assistant should acquire information whose absence causes the greatest avoidable loss in user utility. This is rarely known ex ante; rather, assistants must predict it in order to optimally allocate limited user interactions. We instantiate this problem in image generation and derive a reinforcement learning framework using multi-turn simulated users to maximize utility recovery under uncertainty. In a preregistered study with 456 interactive sessions across 76 human participants, this helped users significantly better match reference images with significantly fewer questions, less total interaction time, and lower cost. This points toward a simple and scalable framework for training language model assistants to better disambiguate user intent by asking more informative questions.

0.31

Bayesian Active Learning for Intent Disambiguation in Interactive Robot Planning

cs.RO cs.AIcs.HC ·2026-09-28 ·Huao Li, Carson Sobolewski, Augustinos Saravanos, William Tan, John Karigiannis +1 more
published similarity 0.62
Abstract · Accepted by CoRL 2026 as Spotlight, also presented at IROS 2026 Human-Robot Dialogue workshop

Interactive robot planning requires robots to infer and execute human intentions from natural language instructions that are often ambiguous, incomplete, or underspecified. Although large language models (LLMs) provide a powerful interface for clarification, relying on the generative model to drive an multi-turn conversation can introduce systematic failures. We propose a Bayesian framework that treats clarification as an active learning problem over grounded Signal Temporal Logic (STL) task specifications. Our method uses LLMs to initialize candidate formal specifications and translate informative contrasts into natural-language clarification questions, while Bayesian optimization maintains uncertainty estimation over user intent and selects queries that maximize information gain. After convergence, the inferred STL specification is passed to a formal planner to synthesize a verifiable robot trajectory. Across four simulated and real-world task domains, our approach generally achieves higher task satisfaction and requires fewer clarification rounds than LLM baselines, while helping smaller models close the performance gap against larger reasoning models.

0.28

Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers

cs.AI cs.CLcs.LG ·2026-09-01 ·Giovanni Bonetta, Matteo Merler, Davide Zago, Rossella Cancelliere, Bernardo Magnini
top venue similarity 0.62
Abstract · 9 pages, 3 figures, 4 tables in the main text, 27 pages, 4 figures, 9 tables including Appendix

Vision-Language Models (VLMs) provide useful priors for interactive decision-making, but using them directly as policies is expensive and brittle: they must be queried at every step, do not improve from environment interaction, and can repeat systematic errors. We study how to learn a cheap autonomous policy from an online, expensive, and imperfect but informative VLM teacher. We propose SAGE (Selective Agent Guidance via Entropy), a framework that queries a VLM only when the learner is uncertain, executes the suggested action during training, and distills guidance into a lightweight Reinforcement Learning (RL) policy. Because VLM advice is not always reliable, SAGE can weight teacher-action distillation using environment-derived advantages rather than treating all suggestions as equally useful. Across sparse-reward visual reasoning and navigation tasks, SAGE learns policies that act without VLM guidance at evaluation time and improves over unguided RL in several environments, including settings where the learned policy exceeds its VLM teacher. The results show that selective guidance is most beneficial when the VLM can help the agent discover high-reward trajectories, and less useful when unguided exploration already succeeds or teacher actions do not lead to informative experience. SAGE also reduces VLM usage by prompting the teacher only on a fraction of training steps and requiring no VLM calls at deployment. Overall, our results suggest that VLMs don't need to be used as fixed policies to be useful; they can instead act as temporary, imperfect sources of guidance whose value is tested and internalized through interaction.

0.42

LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation

cs.RO cs.AIcs.LG ·2026-10-15 ·Zhixu Li, Keqian Tang, Litian Gong, Jingyu Yao, Wenqian Zhang +5 more
publishedbenchmark similarity 0.61
Abstract · Conference on Neural Information Processing Systems (NeurIPS 2026); Project Website: https://libero-perm.github.io/

Personalization is essential for general-purpose robots to transition into household environments and achieve true practicality. While general robotic manipulation has been extensively studied, personalized robotic manipulation (PeRM) lacks a controlled benchmark for separating task completion from preference satisfaction. We introduce LIBERO-PeRM, a benchmark for preference learning in embodied agents. LIBERO-PeRM studies five questions: 1) how policies follow preferences in zero-shot settings; 2) how they learn single preferences from demonstrations; 3) how efficiently they acquire preferences as demonstrations increase; 4) how learned preferences generalize across layouts and tasks; and 5) how policies compose preferences and behave under conflicts. We propose a controlled formulation and taxonomy of observable preferences in simulation, covering motion, skill, and user-profile choices. This formulation is restricted to language-expressible preferences that can be instantiated in LIBERO without changing the task goal, evaluated automatically, and reused across tasks. Building on LIBERO, our procedural pipeline generates paired demonstrations with and without preferences. We create five task suites (312 tasks) with high-quality demonstrations, rich metadata, and preference-satisfaction metrics. Experiments show that current VLA policies struggle with zero-shot personalization, while SFT makes preferences learnable. Mixed training achieves better no-preference retention than sequential adaptation and improves performance with contextual prompts compared with preference-only training, but can weaken motion-preference learning; preference generalization, decomposition, and conflict handling remain difficult. The code, datasets, and demos can be found on the project website: https://libero-perm.github.io/.

0.39

Agentic-Priors: Using Large Language Models to Address the Cold Start Problem in Preference Elicitation

cs.MA cs.AI ·2026-10-15 ·Bruno Arcanjo, Zhaoxing Li, Sebastian Stein
similarity 0.61
Abstract

An agent that learns a user's preferences from their choices faces a cold start problem: before any choices are observed, it must rely solely on a prior. Informative priors are usually fitted to historical choice data, which new tasks and deployments lack. To address this, we present Agentic-Priors, which lets agentic systems use the implicit knowledge of large language models (LLMs) instead. Specifically, an LLM states a new user's trade-offs in interpretable terms, such as willingness to pay or value of time, which we map to a prior over multinomial logit coefficients. This prior is then refined by a Bayesian learner as choices are observed. Across seven choice datasets, Agentic-Priors improves cold start accuracy over an uninformed prior by 5-40 percentage points. It also outperforms AutoElicit, a state-of-the-art LLM prior-elicitation method, on three datasets and loses on one. It is up to 18 points more accurate than in-context learning, and never significantly worse. Against hierarchical Bayes models fitted to surveys of increasing size, our approach is worth roughly 6-40 surveyed users, in terms of log-loss, so its value is concentrated in a deployment's first users.

0.25