CS Paper Radar

PhysVLMM: Interaction-Conditioned Physical Belief Maps for Articulated Objects in Robot Scene Maps

Dibyendu Ghosh

cs.RO ·submitted 2026-10-29 17:35 UTC · v1

Abstract

Mobile robots must open doors and drawers, yet their scene maps record what each object is and whether it has moved, not how it will respond to a pull. This paper proposes PhysVLMM, a method that maintains a physical belief for every articulated part of a Vision-Language-Motion Map and answers a planner's queries about it, with two algorithms: a map update and a query. Watching a part move provably cannot identify its dynamics, and a measured pull identifies only the pulled part, up to one direction. PhysVLMM therefore (i) keeps that direction as a posterior instead of a point estimate, (ii) passes the evidence of each pull in closed form to the untouched look-alike parts of the map, and (iii) answers whether an action will succeed at a stated risk, or abstains. On real doors and drawers in 18 homes, it identifies the joint type in 86% of held-out pulls (48% for a learned single-view method) and predicts the response to a pull with 41% lower error than differentiable simulation (CRPS 0.323 vs. 0.544), four orders of magnitude faster (13 ms vs. 230 s per case). The posterior lowers the final-position error from 0.460 to 0.270, sharing lowers the force error of untouched parts from 56% to 42%, and at every stated risk from 0.02 to 0.30 the false-safe answers stay within it. Pooling all measured parts of a home does as well as the look-alike groups, and simpler variants predict where a pulled part stops at least as well.

Closest papers

Nearest neighbours by meaning across the whole index.

Motion Belief Action: From a Visual Scene Map to Physical Decisions

cs.RO ·2026-11-09 ·Dibyendu Ghosh
similarity 0.79
Abstract

A robot that works among people must judge what it cannot see: whether an object is still where it was last seen, and whether a pull on a door or drawer is worth its risk. Maps that track object change hold nothing about force, maps of physical properties answer a risk question at a fixed risk level, and vision-language-action models keep no explicit estimate of hidden physics. We propose Motion Belief Action (MBA), a decision layer on the scene map. From how often people move an object and how much force it needs, how much load it holds and whether it is blocked, MBA chooses under a force limit to look again, nudge, open, brace or decline, with its risk set by the task's costs. On real activity in 9 kitchens, it predicts whether an object is still in place better than FreMEn and Perpetua, and decides when to look again at lower cost than these and the persistence filter, because about a quarter of placements end in a pick-up within 30 s. On real pulls of doors and drawers in 17 locations, made by a person with a force-sensing gripper, the map's force estimates decide better than a part-type average. Setting the risk from the costs also beats a published risk rule at its fixed level. In simulation, with the force limit at the training parts' median force, MBA raises SmolVLA's and Octo's scores by declining pulls; most declined door pulls would fail, but on drawers more would have opened, and a part-type average gets about half of this gain. In both, a published risk rule set to the risk these costs imply is MBA's rule for a single pull and decides as well in the main settings. A link between opening and holding force, a probing nudge and an action prior for camera tracking do not help in the main settings, and over the real-pull sweep the nudge costs a little.

0.22

Open-World Law-to-Action: Vision-Language Priors and Touch for Parts with Hidden Physics

cs.RO ·2026-10-24 ·Mohamed A. Mabrok
theory similarity 0.70
Abstract

A robot in a home it has never seen must open doors, drawers, lids and taps whose physics a picture does not show. Instructions such as "open the drawer halfway, gently" depend on that physics: how far a part travels, how it resists, and what locks it. We present Open-World Law-to-Action (L2A). A vision-language model (VLM) proposes, for each movable part, its motions, the terms of its law of motion, and what may lock it. Each proposal becomes a prior with a floor, so that touch can always overturn it. The robot identifies each part's law online by Bayesian linear regression with closed-form evidence, for parts that move in one way or in two, and keeps a hypothesis that an unseen part or motion locks the target. We prove that touch alone cannot rule this hypothesis out, derive when another look is worth its cost, and show that a constant load and friction cannot be told apart while a part moves one way. All experiments are in simulation, developed on some scenes and run once on held-out ones. In new kitchens the robot completes 80% of tasks from the VLM's proposals, against 98% when every part is known. Proposed physics raises success on objects that need new terms from 353 to 378-383 of 400 episodes, and a second look raises success in mixed scenes from 284 to 329 of 366 tasks. A mobile robot that takes nothing from the simulator completes 70 of 100 requests in new scenes, against 62 when the simulator says which part each box is (after one change made between two held-out tests).

0.39

AWAM: Active Trajectory-Conditioned View Imagination for Robust Robot Manipulation

cs.RO cs.AI ·2026-10-16 ·Jiawei Geng, Zhenyang Liu, Yuzheng Wu, Yanwei Fu
similarity 0.67
Abstract

Rapid progress in embodied intelligence has enabled increasingly capable robots to follow language instructions and manipulate diverse objects. However, strong results in structured environments do not automatically transfer to open-world conditions, where clutter, occlusion, and changing viewpoints can obscure the spatial relationships required for action. To address this perceptual gap, a virtual active-perception paradigm is introduced. AWAM, an Active World-Action Model actively plans where to observe and imagines what will be seen along the planned trajectory before acting. Task-related virtual camera motion is predicted and used as a structured query into a geometry-conditioned world model, producing a reusable spatial prior for manipulation, which is later compressed into a fixed memory and fused with live observations through cross-attention for diffusion-based action prediction. AWAM achieves competitive success rates with 75.9% on LIBERO-Occ, 62.4% on LIBERO-Plus, 76.5% on RLBench-OG Occlusion1, and 94.4% in real-robot trials. These results support improved robustness to the evaluated occlusions and distribution shifts, addressing part of the gap between structured manipulation and less controlled environments.

0.16

MS-MEM: Multi-Skill Manipulation-Enhanced Mapping via Uncertainty- and Disturbance-Aware Action Selection

cs.RO ·2026-09-02 ·Yitian Shi, Jesper Mücke, Nils Dengler, Sicong Pan, Rania Rayyes +1 more
similarity 0.66
Abstract · under review

Accurate scene understanding in confined, cluttered spaces such as shelves is essential for service robots, as many everyday tasks require them to locate and retrieve objects reliably. Yet, it remains challenging due to severe occlusions, restricted accessibility, and the need to avoid excessive scene changes. In this paper, we propose Multi-Skill Manipulation-Enhanced Mapping (MS-MEM), an evidential framework for uncertainty-aware mapping that integrates active viewpoint selection, object pushing, and grasping. MS-MEM combines scene-level metric-semantic evidential belief estimators with an uncertainty-aware grasp representation. This representation is learned using a novel full-evidential grasp estimator that models both grasp affordance and orientation uncertainty. In our framework, candidate perception and manipulation actions are evaluated within a unified action selection pipeline using a common information gain criterion. For manipulation actions, we further introduce a collateral disturbance constraint (CDC) that discourages excessive changes to confident regions of the scene belief. This enables MS-MEM to select actions that effectively reduce map uncertainty while limiting collateral scene changes. Experimental results show that, compared with single-skill and unconstrained baselines that ignore scene disturbance, MS-MEM achieves higher mapping accuracy while substantially reducing scene disturbance, highlighting the synergistic effects of active viewpoint selection, push, and grasp actions.

0.24

Learning to Explore Hidden Kinematics for Articulated Object Manipulation

cs.RO ·2026-09-29 ·Ruiyao Liu, Boshu Lei, Zhuoyang Pan, Kostas Daniilidis
benchmarkdataset similarity 0.65
Abstract

The kinematics of an articulated object is often ambiguous from vision alone. Interaction resolves the ambiguity, and active perception methods exploit this by searching for the single action that most sharpens a belief over the kinematic parameters at each step. Such greedy search cannot be extended over a horizon without forward models of the contact and inertial dynamics, which are themselves unknown. We instead amortize action selection into training. We maintain a belief distribution over joint type and parameters, initialized from a generative prior and updated by Bayesian filtering on the observed part motion. To condition the policy on this belief, we render it as a per-point articulation flow field, the motion that the current posterior predicts for every point on the object. Carrying the inductive bias of articulated motion, this representation generalizes better than a latent encoding of the belief or flow tracked from observation. We train the policy with reinforcement learning, rewarding the entropy that each interaction removes from the posterior, so that informative exploration becomes learned behavior rather than a search at every step. Our method outperforms previous approaches across door and drawer manipulation on the PartManip benchmark, and reaches 61.7% success on ArticuRiddle, a new dataset of objects whose appearance implies the wrong articulation, against 44.4% for the best previous method. Project Website: https://hiddenkinematics.github.io/

0.33

Visibility-Aware Task and Motion Planning with Task-Centric Perception

cs.RO ·2026-11-12 ·Sahit Chintalapudi, Xiaoyan Wang, Aryan Zoroufi, Aditya Agarwal, Nancy Kanwisher +2 more
similarity 0.65
Abstract

Long-horizon manipulation under uncertainty remains a challenging problem in robotics. We present \ourmethod{}, an algorithm for Visibility-Aware Task and Motion Planning. \ourmethod{} extends existing deterministic task and motion planning solvers to partially observable object rearrangement tasks, while still being able to take advantage of the efficiency and compositional generalization over long horizons of the base solver. This extension interleaves sensing actions into a plan skeleton that assumed full observability, and uses a novel persistent object-based memory to fuse observations into object mesh hypotheses and an observed occupancy map. We evaluate \ourmethod{} on real-world tabletop rearrangement tasks which feature heavy occlusion and partial visibility and demonstrate that it outperforms baselines that do not perform active perception or have persistent memory.

0.18

Acting from Belief, Looking When Needed: A Bayesian Spatial World Model for Navigation under Intermittent Perception

cs.RO ·2026-10-08 ·Feihong Yang, Xiang Long, Jincheng Yu, Jianfei Zhang, Guangjun Ge +2 more
similarity 0.63
Abstract

Robot navigation commonly uses wide-coverage, high-frequency sensing to reduce partial observability; this reliance becomes restrictive when another task temporarily redirects a shared sensor from navigation, interrupting navigation-relevant observations. We study navigation under intermittent perception: acting from an internal spatial belief and looking again only when execution needs a new observation, potentially freeing the shared sensor for other tasks between navigation observations. ALONE, a Bayesian spatial world model, propagates a structured spatial belief using executed actions and corrects it with selectively acquired observations; learned priors over common geometric structures infer unobserved structure from available observation history. It decodes the belief into a spatial estimate for the motion-planning module and predicts a reliability map expressing confidence in the estimate's accuracy. ALONE requests an observation only if insufficient reliability hinders navigation and new evidence should make relevant-region spatial information more reliable; otherwise, it continues acting from the propagated belief. We instantiate ALONE for drone navigation with intermittent single-camera depth images. Across two simulated scene families, it achieves 98% and 97% closed-loop success at a 10 Hz decision rate. Among successful trials, median fractions of decision steps requiring a new depth observation are only 0.9% and 1.3%, respectively, demonstrating high navigation success with substantially reduced observation demand. Real-world indoor flight experiments further validate navigation under intermittent depth observations, with all 10 trials successful.

0.19

Part Grounding, Not Action Knowledge: Locating the Bottleneck in VLM Affordance Prediction

cs.CV cs.LGcs.RO ·2026-09-01 ·Sarthak Sattigeri
similarity 0.63
Abstract

Benchmarks agree that vision-language models reason poorly about low-level manipulation, but an aggregate accuracy score does not say which step fails. We separate two steps that affordance questions conflate: identifying which part of an object to act on, and knowing what action that part requires. Across 19 articulated objects we asked eight models, spanning three developers, what motion a robot should apply. Under an open prompt, push was produced once in 64 evaluations where it was correct, despite being correct for 8 of 19 objects and appearing in the offered label set every time. Inspecting the outputs showed why: models described a different part than the one being scored, e.g. explaining how to pick up a camera rather than press its button. Naming the target part raises action accuracy by 0.32 to 0.63 for every model, from 0.158-0.474 to 0.684-0.947, and push recall from 0-1/8 to 7-8/8. No model beats a constant answer that ignores the image under the open prompt; once the part is named, all eight do. Asked to describe the same part in free prose with no label set, models produce pressing language for 6 to 8 of 8. These results are hard to reconcile with missing action knowledge, and instead point to part grounding as the dominant bottleneck, a pattern that holds across all three model families and does not diminish with capability. Naming the part supplies the grounding variable, so this bounds what a perfect part detector would offer rather than demonstrating a general model of mechanics. Two supporting results agree: on real photographs only three of eight models localize grasp points better than a constant baseline, and on rendered objects none do. We also document two measurement errors of our own, a threshold that let a constant baseline score 0.929 and a labelling rule wrong on 4 of 19 objects, both caught only by testing our numbers against trivial alternatives.

0.23