CS Paper Radar

AD-WM: Action-Discriminative World Models for Counterfactual Model Predictive Control

Jiabin Qiu, Zixuan Chen, Hongye Cao, Jieqi Shi, Jing Huo, Yang Gao

cs.AI cs.RO ·submitted 2026-09-24 17:59 UTC · v2
code/data

Abstract

Latent world models are typically trained to predict factual transitions, whereas model predictive control (MPC) must compare alternative actions from the same state. A model can therefore achieve low factual prediction error yet poorly distinguish candidate actions. We introduce AD-WM, an action-discriminative joint-embedding world model for counterfactual MPC. AD-WM combines residual latent dynamics with predictor-level action-recovery regularization, using inverse dynamics and a normalized recovery objective motivated by conditional mutual information. Both objectives encourage planning transitions to preserve action information; their auxiliary heads are discarded at test time, leaving MPC unchanged. On OGBench-Cube, AD-WM improves hard-start success from 3.7% to 52.0% over a matched LeWM baseline and improves mean success over the reproduced baseline in four of five simulation environments. Planning diagnostics show that factual prediction error and whole-bank action ranking do not follow the closed-loop success ordering, whereas CEM-aligned elite regret tracks success more closely. With a frozen V-JEPA 2 encoder and matched DROID post-training, AD-WM also improves zero-shot transfer to our Franka setup, increasing basic pick-and-place success from 42.2% to 71.1% without lab-specific adaptation. These results suggest that world models for planning should preserve action-dependent differences needed for counterfactual selection, rather than optimize factual prediction accuracy alone. More videos and code are available at https://ad-wm.github.io/.

Comment
9 pages, 5 figures, 4 tables. Project page: https://ad-wm.github.io/

Closest papers

Nearest neighbours by meaning across the whole index.

Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning

cs.RO cs.AIcs.LG ·2026-09-03 ·Muyuan Liu, Yue Huang, Zheng Liang, Xiang Gao
publishedbenchmarkhighly cited similarity 0.77
Abstract · 5 pages, 4 figures, 2 tables. Accepted to the IROS 2026 Workshop on Physical World Models for Scaling Embodied AI (PWMS 2026)

Action-conditioned JEPA world models enable planning toward visually specified goals without reconstructing future pixels, yet latent prediction alone does not explicitly encourage the learned representations to retain information relevant to robotic control. We introduce an end-to-end JEPA world model that augments latent prediction with inverse dynamics (IDM) and state alignment (SA). While inverse dynamics discourages latent collapse and makes latent transitions informative of the actions that produced them, state alignment grounds consecutive representations in their associated physical configuration and motion. Across four benchmark tasks, our model attains the highest success rates on TwoRoom (100%), PushT (98%), and OGBench-Cube (87%), while performing comparably to LeWorldModel on Reacher. Our ablation further shows that adding state alignment consistently improves planning success over IDM alone across all four tasks. Although LeWorldModel, our primary baseline, attains higher average straightening on OGBench-Cube, transition-subspace analysis shows that its transition energy is concentrated in a substantially lower-dimensional subspace. Our state-aligned model exhibits a higher effective transition dimension than LeWorldModel and improves planning over IDM alone, supporting state alignment as an effective complement to inverse dynamics for robotic planning.

0.49

WA-JEPA: Goal-Aware Predictive Feedback Action Refinement for a World-Action Model

cs.CV ·2026-11-14 ·Tao An, Weihang Liang, Tianqi Wang, Chuqing Cao, Wenming Tan +5 more
code/data similarity 0.75
Abstract · 16 pages, 5 figures, including supplementary material. Code available at https://github.com/Dovisya/WA-JEPA

JEPA-based world models predict action consequences in latent space, but representative planning approaches rely on candidate-action search, requiring repeated evaluation of action sequences and incurring substantial online computational cost. Direct action generation avoids this search, yet how to leverage latent predictions to guide action generation given task goals remains a key challenge in translating learned physical dynamics into effective decisions. We introduce WA-JEPA, a world-action model that couples action generation, action-conditioned latent prediction, and goal-aware feedback. The model first generates an action proposal and predicts the resulting future scenes and proprioceptive states, then refines the proposal through two complementary feedback paths, using the task goal as a reference. Goal-discrepancy residual feedback (GDRF) uses differences between the latent representations of predicted scenes and the task goal to guide chunk-wise action refinement. Future-scene context feedback (FSCF) preserves fine-grained predicted scene representations to guide subsequent action refinement. Jointly learned from offline demonstrations, this process iteratively updates the same plan before execution, making future predictions directly participate in action generation without candidate-action search. We evaluate short-horizon goal reaching, long-horizon tasks, and full-task execution. The results show that goal-aware predictive feedback supports competitive task performance while maintaining a compact model and low online computational cost. Code is available at https://github.com/Dovisya/WA-JEPA.

0.20

EmbodiedSkills: World Action Model-augmented Cascaded Skills for Vision-Language-Action Agent

cs.RO cs.AI ·2026-09-01 ·Wei Wang, Wenqiao Zhang, Yutong Lin, Yuqian Yuan, Tianwei Lin +12 more
similarity 0.73
Abstract

Vision-language-action (VLA) policies generate executable robot actions, but plausible proposals can have different consequences for the task. We present EmbodiedSkills, a skill-based agent framework that augments frozen VLA policies with world action models for consequence-aware action selection. The framework defines reusable skills for scene observation, context encoding, candidate generation, consequence prediction, outcome evaluation, candidate comparison, and action execution. Five stages organize their input-output dependencies in a shared feedback loop, accommodating both continuous action chunks and pick-and-place primitives. At each decision point, independently trained world action models predict the latent consequences of a common set of policy proposals. Selection combines their outcome scores and prediction disagreement with a lightweight comparator that estimates the benefit of an alternative from agreement in its predicted effects. The predictive and comparison modules learn from matched-state candidate branches, while the action policy and pretrained observation encoders remain frozen. On MetaWorld MT50 and CLIPort, EmbodiedSkills improves success rates from 76.28% to 78.64% and from 33.94% to 36.83%, respectively. An additional CALVIN evaluation with the frozen FLOWER policy increases five-task-chain success from 77.10% to 77.90%. These results demonstrate the applicability of consequence-aware selection across different policy backends and action representations.

0.38

FACT: Failure-Aware Causal Training for World-Action Models

cs.RO cs.AIcs.LG ·2026-08-10 ·Quanquan Peng, Yutong Liang, Rui Yan, Nicklas Hansen, Xiaolong Wang
highly cited similarity 0.73
Abstract

Recent world-action models (WAMs) show that co-training policies with future prediction can provide physical priors for action generation. Building on the future-prediction ability of video models, many WAMs generate future videos and recover actions with inverse-dynamics models, or use these predicted videos as goal conditions for action generation. In both cases, the world model is trained mostly on successful demonstrations and has little reason to predict the consequences of bad actions. We introduce FACT, a causal World-Action Model that predicts future video and task progress conditioned on the executed action. This action-conditioned interface allows failure rollouts to supervise action consequences, turning bad actions into valid future targets rather than being discarded. Failure-aware training makes the progress predictor aware of both successful and failed action outcomes, which can optionally be used to score sampled action candidates at inference. Extensive experiments on simulation and real-world bimanual manipulation tasks show that FACT outperforms many existing baselines, improves as failure data are incorporated into training, and reduces success-biased future hallucination under bad actions. See more details at https://fact-wam.github.io/

0.56

Diffusion-Steering-Based MPC with a JEPA World Action Model

cs.RO cs.AIcs.CV ·2026-10-08 ·Shashank Hegde, Alexander Popov, Elie Aljalbout, Nikolai Smolyanskiy
similarity 0.73
Abstract · 14 pages, 6 figures

World action models (WAMs) predict actions and future observations, typically from a reconstruction-based representation that carries noisy, redundant information which can complicate downstream predictions. We introduce LeWAM, a bidirectional transformer for forward, backward, inverse dynamics and policy prediction, on a decoder-free JEPA latent trained end-to-end through all four modes. We see the following benefits: 1) Alignment: linear probes read robot and object state from LeWAM's latent better than from a regular Le World Model (a forward-only JEPA world model), while the latent ignores visual distractors as well as LeWM does and far better than a reconstruction-based WAM. 2) Acting: Closed-loop evaluations of LeWAM match a regular flow-matching policy trained on the same encoder at matched size, while also providing a world model. 3) Planning: Sampling raw actions when planning with WAMs lets MPC exploit dynamics-model inaccuracies; planning in the noise space of the policy head instead improves the closed-loop performance of these WAMs.

0.45

Action-Consequence Alignment for Reliable Planning and Self-Improving in Latent World Models

cs.AI ·2026-10-03 ·Jinping Wang1, Zhiqiang Gao, Xiantong Zhen, Ling Shao
similarity 0.72
Abstract

Latent world models learn to predict observed transitions, yet low prediction error alone does not guarantee reliable planning. Inspired by self tickling experiments in neuroscience showing that disrupting motor sensory correspondence increases prediction mismatch, we examine whether learned world models preserve an analogous action consequence correspondence.The results show nearby alternatives can receive lower prediction errors despite producing physical outcomes farther from the recorded target. With that future treated as a goal, this reveals a concrete prediction planning mismatch: the model assigns a lower cost to an action that achieves the target less accurately. To mitigate this gap, we introduce Action Consequence Alignment (ACA), a training objective that complements forward prediction by penalizing the prediction error advantage of locally searched alternatives over factual actions without additional model components or environment interactions during training. The same principle can also guide additional data collection for self improvement. We demonstrate that across diverse environments and evaluation settings, ACA improves planning performance and reduces real goal error, while ACA guided data collection outperforms random local sampling. These results support action consequence alignment as a practical principle for bridging predictive learning and reliable planning.

0.40

WM-Quant: Rollout-Aware Post-Training Quantization for JEPA-Style World Models

cs.LG ·2026-11-16 ·William Hammer, Andrew Heuer, Nikhil Krishna, Matthew Grenier, Ramtin Zand
similarity 0.72
Abstract · 11 pages, 5 figures, and 7 tables

We present WM-Quant, to our knowledge, the first post-training quantization (PTQ) framework designed specifically for JEPA-style world models. Although world models typically feature relatively few parameters, action planning algorithms often require thousands of inferences, which can quickly become computationally expensive. PTQ is a natural way to reduce this cost. However, we show that in world models using autoregressive model predictive control (MPC) rollouts, quantization error can compound with each predictor step, resulting in action rankings from the quantized model becoming near-random. WM-Quant addresses this with three components that build on existing PTQ methods: early-encoder precision, control-path preservation, and rollout-aware scaling. Early-encoder precision protects accurate latent state and goal information to mitigate long-term error propagation. Control-path preservation allows the world model predictor to distinguish between fine-grained actions, significantly improving action rankings and execution. Rollout-aware scaling jointly calibrates encoder and per-step predictor activation ranges with respect to an entire rollout rather than a single prediction, reducing the cumulative rollout error. Across multiple world models, environments, and PTQ methods, WM-Quant substantially improves planning success at low bit-widths, approaching full-precision performance at W4A4.

0.29

$R^2$-WAM: Repair-and-Reject Post-Training for World Action Models

cs.RO ·2026-10-04 ·Ruiyan Xu, Haisheng Su, Sixu Lin, Zhaokun Yue, Chengming Hu +2 more
code/data similarity 0.72
Abstract · 24 pages, 7 figures

World Action Models (WAMs) emerge as a promising foundation for policy refinement by predicting the consequences of sampled actions. However, visually plausible predictions can mislead policy refinement if they fail to reflect the input actions. To address this mismatch, we introduce $R^2$-WAM, a two-stage repair-and-reject post-training framework that first improves the consistency of predicted futures with input actions, then uses these futures to select inferior action samples for negative fine-tuning. The repair stage grounds imagination in observed robot behavior through a kinematic alignment score that measures agreement between predicted and demonstrated motion, enabling the predicted video to faithfully reflect its input actions. Using the repaired video model, the rejection stage compares imagined outcomes of sampled and demonstrated actions, selectively applying negative fine-tuning to samples whose predicted task progress falls below the demonstrated reference by a prescribed margin. Together, the two stages extend video prediction from representation learning to consequence-based policy refinement without additional environment interaction or changes to the inference procedure. $R^2$-WAM achieves 93.8% average success on RoboTwin 2.0 across clean and randomized settings. On the long-horizon real-world Fold Shirt task, it achieves 87.5% average success, compared with 0% for Fast-WAM. Our project page is available at https://r2-wam.github.io/.

0.34