CS Paper Radar

MHE-Former: Multi-Hypothesis Transformers via Entropy Maximization for 3D Mesh Recovery

Boshu Jia, Rongyu Chen, Linlin Yang, Zihao Liu, Yingjie Chen, Zhongqun Zhang, Zhulin Tao, Shaohui Lin, Xiaoyu Wu, Libiao Jin, Baochang Zhang, Angela Yao

cs.CV ·submitted 2026-09-09 18:39 UTC · v1

Abstract

Monocular 3D hand and body mesh recovery often suffers from severe occlusion and ambiguity. Traditional deterministic methods typically regress a single optimal solution, leading to overconfident predictions. In this paper, we introduce an exploration--exploitation paradigm for ambiguous mesh recovery with multi-hypothesis learning and selection. Specifically, during exploration, based on our probabilistic formulation and entropy maximization, we propose a novel multi-hypothesis method referred to as MHE-Former. It is a Transformer-based multi-hypothesis framework, ensuring high training efficiency and label friendliness while generating plausible and diverse hypotheses. During exploitation, we propose Hypothesis Selection, a context-aware process for multiple predictions. Especially leveraging VLM's powerful visual understanding and reasoning capabilities, it allows users to choose the most plausible and desired estimate with additional evidence and natural language intent. Extensive experiments demonstrate that our framework achieves state-of-the-art performance in accuracy and diversity across multiple datasets. The user preference study further shows the practicality of our hypothesis selection process.

Comment
14 pages, 11 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Matisse: Evidence-Space Reasoning for Active 3D Reconstruction

cs.CV cs.RO ·2026-09-30 ·Xihang Yu, Kaichen Zhou, Lorenzo Shaikewitz, Clément Jambon, Xiao Zhan +2 more
code/data similarity 0.70
Abstract · 20 pages, 7 figures, 7 tables. Project page: https://xihangyu630.github.io/matisse/

How can a 3D reconstruction system acquire and retain useful information to understand the geometry of a scene from partial views under a limited computation budget? Existing active view acquisition methods typically estimate uncertainty over observed or instantiated geometry, limiting their ability to reason about unseen structure, while long-horizon reconstruction methods often retain redundant observations. We introduce Matisse, a training-free framework that unifies active reconstruction and keyframe selection by leveraging evidence provided by a pretrained generative 3D model. Matisse estimates Evidential Uncertainty from cross-attention evidence associated with 3D latent tokens and derives an Evidential Information Gain to guide both view acquisition and keyframe selection based on the expected reduction in posterior entropy. Matisse supports multi-object scenes through occlusion-aware, object-balanced aggregation and propagates uncertainty through intermediate latents to avoid full reconstruction during planning. Matisse reduces Chamfer distance by 12.7%, 3.8%, and 9.2% on GSO30, YCB-V, and Replica, respectively, relative to the best baseline on each dataset, and achieves a $1.50\times$ end-to-end speedup over the best active reconstruction baseline on GSO30 with the same reconstruction backend. In the GSO30 keyframe selection experiment for long-horizon reconstruction, Matisse achieves comparable Chamfer distance using 14% of the input views compared with Stream3D.

0.31

Modeling Uncertainty Propagation in 3D Morphable Models

cs.CV cs.LG ·2026-10-17 ·Megh Shukla, Edo Collins, Prashanth Chandran, Karel Petránek, Sergio Orts-Escolano +2 more
code/data similarity 0.67
Abstract

We study the problem of mapping parameter-space variance of a 3D Morphable Model (3DMM) to per-vertex variance. Traditionally, per-vertex variance has been used to visually interpret the learnt parameter uncertainty. However, mapping parameter-space variance to per-vertex using sampling or error propagation is computationally expensive. We address this limitation with Closed-form Local Uncertainty Estimation (CLUE), which transforms the parameter-space distribution of 3DMMs to per-vertex through a closed-form approximation. CLUE emerges from applying the law of total variance to the 3DMM formulation and is modular, computationally inexpensive, and accurate. Subsequently, we study integrating CLUE for in-the-wild human mesh recovery training and inference. We note that CLUE uses the geometry of the mesh as an inductive bias when predicting the variance, leading to improved accuracy. Through extensive qualitative and quantitative experiments, we show that when used in training, CLUE (1) outpeforms probabilistic and deterministic baselines on metrics evaluating joints, mesh and uncertainty, and (2) has the highest uncertainty-to-error spearman correlation, with applications including data mining and interpretable uncertainty. Finally, we share SAM 3D Body (CLUE) which augments the regressor with high resolution uncertainty maps. We make our code available at https://github.com/vita-epfl/CLUE

0.27

Beyond Single Object: Learning 3D Relations with Large Language Models

cs.CV cs.AIcs.CL ·2026-08-16 ·Kohsuke Ide, Ryousuke Yamada, Yue Qiu, Xianzheng Ma, Yoshihiro Fukuhara +2 more
top venue similarity 0.65
Abstract · Accepted to CVPR 2026

We address a fundamental gap in 3D-LLMs: existing models focus on single-object/scene description, struggling with detailed, inter-object comparison. We propose a framework for detailed object-level reasoning across multiple objects with three components: (1) MO3D (Multi-Object in 3D), an instruction dataset requiring fine-grained multi-object comparison; (2) Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships while preserving local geometry; (3) Mini-apps, two application-driven benchmarks (Shape Mating, Change Captioning) that probe geometric understanding for practical use. Recent 3D-LLMs and 2D-VLMs perform poorly on these tasks, lacking both comparison-centric design and geometric awareness. In contrast, Multi-3DLLM trained on our mixture data learns geometric reasoning, surpasses all baselines on MO3D, and provides positive transfer to single-object classification.

0.70

ProbeShare: Probing a Few, Sharing Across Many for Fast World-Model Planning

cs.RO ·2026-11-18 ·Zejian Kang, Xiantao Lin, Zekai Chen, Boyang Shen, Mingyu Ma +3 more
similarity 0.65
Abstract

Visual planning with a frozen joint-embedding world model supports robotic reaching and object manipulation by evaluating candidate action sequences against an image goal before execution. Standard cross-entropy method (CEM) planners roll out every candidate independently through a billion-parameter predictor, even when nearby actions induce similar scene changes. This redundant computation grows with population size and search rounds, slowing closed-loop replanning. We observe that candidates within a planning step share an observation history and thus a common action-to-feature map. Based on this insight, we propose ProbeShare, which measures local responses with a few representative probes and reconstructs each candidate's feature grid through scene-conditioned, geometry-aware weighting. When promising candidates move beyond existing probe coverage, ProbeShare evaluates a new probe to rescore nearby candidates. Across four RoboCasa tasks, ProbeShare accelerates episode planning by 43.8--128.9 times over dense RoboJEPA and at least $14.2\times$ over competitive baselines, with greater mean goal progress in a matched-time comparison.

0.27

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

cs.CV cs.CL ·2026-09-29 ·Jaewoo Jung, Hyeonseo Yu, Honggyu An, Jisang Han, Mungyeom Kim +8 more
publishedcode/data similarity 0.65
Abstract · NeurIPS 2026; Project Page: https://cvlab-kaist.github.io/Imagine3D-LLM

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.

0.56

MVDG: Efficient Multi-view 3D Disambiguation on Unconstrained Real-World Images

cs.CV ·2026-10-01 ·Hanyuan Xiao, Gonglin Chen, Haolin Xiong, Wenbin Teng, Haiwei Chen +1 more
similarity 0.64
Abstract

Illusory matches between distinct yet visually similar 3D surfaces--doppelgangers--remain a fundamental obstacle for large-scale, in-the-wild 3D reconstruction and visual localization. Prior work mitigates this issue with pairwise classifiers, but this design limits multi-view contextual reasoning and incurs O(n^2) inference complexity for downstream structure-from-motion (SfM). We present MVDG, a scalable multi-view disambiguation framework built on the 3D foundation model VGGT, which jointly reasons over an arbitrary number of multiview images. By incorporating 3D-aware multi-view features, our method reduces dependence on pairwise comparisons by encoding and decoding views in a single pass. We further observe that direct multi-view fine-tuning of VGGT can be unstable under noisy supervision; motivated by label ambiguity in Doppelgangers, we construct a pseudo-pairwise training set from AerialMegaDepth and show that fine-tuning on sampled subsets yields stable optimization and strong generalization to held-out scenes. Finally, because full SfM evaluation (even with faster pipelines such as GLOMAP) remains expensive, we process a pseudo-pairwise dataset for efficient validation; we derive a predictive relationship between regular SfM metrics and the classification accuracy on this pseudo-pairwise test. Experiments show that our method achieves comparable pairwise accuracy while improving both SfM accuracy and inference speed over baselines.

0.10

Imagining Goals with Geometric Horizon Models for Policy Generalization

cs.LG cs.AI ·2026-10-13 ·Daniel Lawson, Dhruv Sreenivas, Adriana Hugessen, Khimya Khetarpal, Glen Berseth
similarity 0.64
Abstract

Training from offline data has allowed for substantial progress in domains such as robotics, leading to general-purpose policies that can be easily applied zero-shot or efficiently finetuned for downstream tasks. However, these policies can still have poor generalization, both due to the choice of modeling objective and from learning from a static dataset. In this work, we focus on the challenging task of zero-shot goal generalization, where a policy is evaluated on unseen tasks that require combining its existing knowledge (compositional generalization). An avenue for improving a policy's generalization is by generating new experience with world models; however, such generation has proven difficult for longer horizons. Thus, to alleviate this issue, we propose TD-Aug, which samples from a geometric horizon model and allows for directly imagining novel outcomes that can be achieved by composing existing knowledge. We demonstrate that training on these future outcomes as goals for goal-conditioned BC policies significantly improves generalization in stitching-based OGBench tasks.

0.16

MMCoT$^3$: Empowering 3D Scene Understanding with 3D-Text Interleaved Multi-Modal Chain-of-Thought Reasoning

cs.CV ·2026-11-01 ·Yongqi Shan, Yifan Wang, Yunzhi Zhuge, Lijun Wang, Huchuan Lu
top venue similarity 0.63
Abstract · 10 pages, 5 figures, 4 tables. Accepted to ACM Multimedia 2026

Understanding and reasoning over 3D scenes from point clouds is essential for a broad range of AI applications. Recent 3D large language models have shown promising results, yet they still face two fundamental limitations: (i) insufficient capability for grounded multi-step reasoning in complex 3D scenes, and (ii) a text-centric inference paradigm that lacks effective interaction between textual reasoning and visual evidence. As a result, these models often fail to verify intermediate inferences, disambiguate similar object instances, and avoid hallucinations. To address these limitations, we propose MMCoT$^3$, the first 3D-Text Interleaved Chain-of-Thought (CoT) framework for 3D scene understanding. The core idea is to explicitly interleave textual reasoning with task-relevant 3D evidence, enabling the model to consult scene observations throughout the reasoning process. In addition, since a textual mention in 3D scenes may correspond to multiple candidate objects, naive object-mention alignment can inject incorrect evidence and mislead subsequent reasoning. To mitigate this multi-instance ambiguity, we design an ambiguity-aware evidence aggregation mechanism that uses query-guided cross-attention to fuse candidate object tokens and inject both evidence and ambiguity cues into the reasoning process. To better equip the model with multi-step reasoning capability, we further develop PointCoT, a large-scale 3D-specific CoT instruction tuning dataset with approximately 176.2K instances spanning captioning, question answering, dialogue, and planning tasks. Extensive experiments on six benchmarks show that our method consistently improves performance and achieves state-of-the-art results, validating the effectiveness of our proposed ambiguity-aware 3D-text interleaved reasoning for 3D scene understanding.

0.28