CS Paper Radar

From Points to Trajectories: Rethinking Multimodal Alignment for Physical Events

Yiyang Zhu, Guangyuan Liu, Jingdi Lei, Li Wei, Chau Yuen

cs.AI eess.SP ·submitted 2026-11-18 11:26 UTC · v1
theory

Abstract

Advancing physical artificial intelligence (Physical AI) requires moving beyond static object recognition to ground how real-world mechanics (e.g., impact, friction, and force transfer) unfold over time. While multimodal aligners seek to capture these interactions by matching multi-sensor streams, prevailing architectures compress each temporal sequence into a single pooled embedding. We show that this paradigm fails on twin events: distinct physical executions recorded under identical task conditions. Collapsing the temporal dimension averages out transient cross-modal dynamics, effectively reducing physical event alignment to static scene recognition. Formally, we prove that as sequence length increases, any temporally pooled representation converges to the stationary recording condition. Consequently, as windows grow, no choice of loss function or embedding geometry can disambiguate twin events from pooled features alone. Instead, we show that physical events are uniquely identified by the event gap, namely the instantaneous, synchronous co-fluctuations across modalities that are erased by pooling and accumulate only along unpooled trajectories. To recover this signal, we introduce CounterPoint, a lightweight plug-in module that strips out each modality's temporal mean to score frame-level cross-modal agreement. Integrated across four state-of-the-art (SOTA) multimodal aligners, CounterPoint shifts representations from coarse condition matching to fine-grained physical event alignment, consistently outperforming existing temporal alignment methods.

Closest papers

Nearest neighbours by meaning across the whole index.

SapiensID 2.0: Aligning Human Recognition Foundation Models with Human Perception

cs.CV ·2026-08-11 ·Yiyang Su, Jie Zhu, Feng Liu, Anil K. Jain, Xiaoming Liu
similarity 0.63
Abstract

While foundation models have significantly advanced human recognition across diverse modalities, they predominantly rely on static, geometric feature extraction. This approach fundamentally diverges from human perception. Consequently, current models often suffer from "semantic blindness," overfitting to transient noise while failing to leverage invariant soft biometrics, and struggle to capture temporal motion signatures. To bridge this gap, we propose SapiensID 2.0, a human recognition framework enriched with both semantic and temporal awareness. To overcome the lack of soft-biometric annotations, we transfer zero-shot semantic knowledge from Multimodal Large Language Models (MLLMs) into a discriminative embedding space. We resolve the dimensional mismatch between these spaces using Invariant Trait Alignment (ITA) to distill core persistent traits, and Transient Noise Disentanglement (TND) to decouple artifacts like clothing. Furthermore, we design a Kinematic Semantic Attention Head (K-SAH) that extends spatial attention across temporal windows. By tracking semantic patches over time, K-SAH captures rich kinematic signatures without requiring large-scale video datasets. Extensive experiments demonstrate that SapiensID 2.0 achieves state-of-the-art performance across image- and video-based person re-identification and gait recognition, while maintaining robust face recognition capabilities.

0.34

ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction

cs.RO cs.AI ·2026-09-18 ·Tim Engelbracht, René Zurbrügg, Mayank Mittal, Marco Hutter, Marc Pollefeys +2 more
code/data similarity 0.63
Abstract

Manipulating objects requires understanding not only their motion, but also the physical properties that determine it. For articulated objects, these include inertia, friction, and mechanisms such as springs or door closers, whose effects can vary with configuration and velocity. Such properties are not directly observable from appearance: visually identical doors may require very different effort to manipulate. Existing digital-twin pipelines recover primarily kinematics or assign static physical parameters from visual and language priors, which can yield physically implausible estimates. As a result, state-dependent mechanism dynamics remain unidentified and are not represented in standard asset formats. We present ForceTwin, a system for identifying physics-informed digital twins of articulated objects from instrumented human interaction. A person probes an object using a handheld force-sensing gripper, providing synchronized poses and interaction forces from which we estimate the articulation, parametric dynamics including inertia, Coulomb friction, viscous damping, and a structured neural residual capturing state-dependent mechanism forces. ForceTwin nearly halves the inertial-parameter error of a VLM prior. As a feedforward dynamics model for impedance control on a Spot and a Franka FR3, ForceTwin achieves 87% goal completion across nine object-embodiment pairs, compared with 60% using VLM-prior and 57% using kinematics-only twins, with the largest gains on objects whose strong mechanisms cause both baselines to stall. We further use the identified twins to train whole-body door-traversal policies and deploy them in the real world. Project Page: https://timengelbracht.github.io/forcetwin-website/

0.58

FACT: Fidelity-Aware Construction of Articulated Twins

cs.RO cs.AI ·2026-09-29 ·Kuixiang Shao, Chuansen Nie, Yinuo Bai, Jiayuan Gu, Jingyi Yu
similarity 0.63
Abstract

Visually plausible articulated assets may still fail during contact interactions or exhibit inaccurate motion. We present FACT (Fidelity-Aware Construction of Articulated Twins), an agentic framework that progressively constructs articulated twins to improve geometry, contact, and dynamic fidelity. The agent drives an evidence--diagnosis--revision loop on a shared editable representation, selecting measurements and model edits using quantitative feedback, while numerical tools execute and validate the updates. It reconstructs editable articulated geometry from images through feature planning, targeted measurements, and diagnostic refinement. On this reference, it repairs collision proxies through task-aware local repartitioning before fidelity-constrained compression. Finally, it constructs response models from passive-response videos, using simulation residuals to guide model revision and constrained physical parameter fitting. Experiments show that FACT improves geometric reconstruction over baselines, enables more reliable interaction with simpler collision proxies, and better reproduces held-out physical responses than direct parameter inference.

0.27

Beyond Multimodal Alignment: Shared Physical Representations Across Sensors and Action Orders

cs.LG cs.RO ·2026-08-19 ·Kaizhen Tan, Xin Xu, Siru Tao, Yixiao Li, Hanzhe Hong +2 more
similarity 0.62
Abstract

Multimodal world models are often evaluated by whether different sensors produce similar representations. However, similar representations do not necessarily imply that the models make the same physical predictions, or that those representations can be reused when actions are combined in a new order. We study both questions through the physical responses predicted by a model. We first use the Cluster Haptic dataset to ask whether audio and acceleration can independently recover the behavior of the same surface from different observations. Predictions from the two sensors are substantially closer for the same surface than for different surfaces, with a $4.5\times$ gap on average, while both also outperform an average-surface prediction. We then show that this agreement alone does not determine how familiar actions should compose. In a controlled elastoplastic system, shared step dynamics fit observed programs less accurately than a whole-program predictor but generalize better to unseen action orders, with the ranking reversing on both held-out transitions across three independent initializations. Fusing free-decay and hysteresis observations further improves prediction, with diagonal Gaussian beliefs yielding the lowest errors. Together, these results distinguish cross-sensor consistency, multimodal fusion, and generalization to new action orders as separate questions in evaluating multimodal physical representations.

0.28

Mind the Rift: Cross-Scale Coupling Mismatch for AI-Generated Video Detection

cs.CV cs.MM ·2026-09-01 ·Siyu Li, Jin Yang, Weiheng Liang
top venuecode/data similarity 0.61
Abstract · 12 pages, 4 figures, 11 tables. Accepted at ACM Multimedia 2026 (MM '26), Rio de Janeiro, Brazil. This version includes the supplementary ma…

As AI video generators achieve cinematic realism, reliable detection becomes essential for safeguarding digital trust. We identify cross-scale coupling mismatch as a new forensic signal, where scale refers to the level of abstraction (semantic dynamics vs. pixel-level residuals): in natural videos, macro-level temporal dynamics and micro-level residual patterns are intrinsically coupled by the unified imaging physics pipeline, whereas AI generators, whose training objectives do not explicitly preserve this joint distribution, systematically violate this coupling. Detecting such mismatch is challenging because it requires independently extracting information at both scales while simultaneously quantifying their cross-scale relationship. We propose RIFT (Representation Inconsistency Forensics on Trajectories), an orthogonal forensic framework that addresses this through three interlocking components: a macro stream that builds a dynamic baseline of expected temporal evolution via differential geometry and persistent homology on learned manifold trajectories, a micro stream that acts as a sensitive forensic probe via steganalytic filtering and temporal modeling, and a coupling divergence module that measures the conditional dependency between the two streams. Gram-Schmidt orthogonality guarantees the information-theoretic validity of this measurement. Experiments on two benchmarks (VidProM, 120K videos, 7 generators; GenVidBench, 68K videos, 4 generators) demonstrate that RIFT achieves 99.33% and 99.72% F1-score respectively, with 97.87% unseen-generator detection rate in leave-one-out evaluation, while exhibiting encoder agnosticism: scaling from ViT-S/14 (22M) to ViT-L/14 (300M) changes F1 by less than 0.1%, and switching to a different encoder family (DINOv1) reduces F1 by only 0.73 pp. Code is available at https://github.com/Litsay/RIFT

0.51

MiMaAlign: Hierarchical Micro-Macro Alignment for Physically Plausible Video Generation

cs.CV ·2026-10-12 ·Yansong Liu, Litong Gong, Weijie Li, Tiezheng Ge, Ming Zeng
benchmark similarity 0.61
Abstract · 20 pages, 14 figures

Text-to-video (T2V) diffusion models have made remarkable progress in generating high-fidelity visuals, yet their outputs still often exhibit physically implausible phenomena, including discontinuous dynamics and implausible event interactions. Existing representation alignment methods mainly transfer generic video features or short-range motion cues, treating physical plausibility as a single-level representation problem. To address this issue, we propose MiMaAlign, a hierarchical micro-macro alignment framework that transfers complementary dynamics-oriented and event-oriented knowledge into different abstraction depths of a pretrained video diffusion model. Specifically, micro-level continuous dynamics alignment imposes constraints by aligning diffusion features with a continuous-time motion representation learned by a Neural Ordinary Differential Equation (Neural ODE), while macro-level event-commonsense alignment provides representations of object interactions and event semantics through an instruction-prompted vision-language model (VLM). With training-only auxiliary objectives, we align the complementary supervision from the two teachers with shallower and deeper generator features, respectively, enabling the generator to internalize the corresponding physical priors. Empirical evaluations show that, on CogVideoX-2B, MiMaAlign achieves a relative improvement of up to 21.0% in the Physical Commonsense (PC) score on the VideoPhy benchmark over the baseline, and also delivers substantial gains on the highly challenging VideoPhy2 benchmark, while maintaining general video quality on VBench and being supported by blind user comparisons.

0.19

DSAR: Dual-Stream Autoregressive Modeling of Temporal Cloth Dynamics for Photorealistic Animatable Avatars

cs.CV ·2026-08-11 ·Haozhong Xiong, Yao Yu, Yu Zhou, Sidan Du
top venue similarity 0.61
Abstract · Accepted to ECCV 2026. 25 pages, 7 figures, including supplementary material

Creating photorealistic and temporally coherent animatable human avatars from RGB videos remains challenging. Current methods struggle to capture realistic cloth dynamics, producing over-smoothed appearance or severe artifacts on out-of-distribution poses. This limitation stems from a fundamental oversight: existing approaches neglect the temporal causality inherent in cloth physics, where current states emerge from previous states through temporal evolution rather than instantaneous skeletal configurations alone. Without explicit modeling of this causal structure, networks learn pose-appearance correlations instead of motion evolution, leading to poor generalization. We introduce a dual-stream autoregressive framework that explicitly models both observable geometric information and implicit internal state. The geometric stream propagates surface displacement from the previous frame, while the state stream fuses current features with historical states retrieved from a memory bank. Motion-adaptive aggregation handles spatially-varying dynamics, and adaptive regularization balances smoothness with flexibility. Experiments on challenging datasets demonstrate significant improvements in rendering quality, temporal consistency, and generalization to motion patterns beyond training distributions, validating that dual-stream temporal modeling enables realistic cloth dynamics.

0.40

AuRA: From Atoms to Bits through Autonomous Robot Interaction

cs.RO ·2026-10-25 ·Xinchen Yu, Yoonkyo Jung, Haoran Chen, Zikui Cai, Aayush Talreja +3 more
code/data similarity 0.60
Abstract · Project page: https://aura-robot.github.io/

Simulation enables robot-learning experiments at scale, but acquiring physics-informed digital twins often requires human manipulation and manual annotation of recorded interactions. To reduce this reliance on human effort, we present AuRA, a system that automates robot interaction and data preparation for digital-twin reconstruction. AuRA combines execution-aware grasp selection, gripper-motion measurements derived from robot states, and registered pre-interaction geometry and appearance priors. These choices enable autonomous collection, avoid reliance on visual gripper tracking, and recover object information obscured by the robot. Across 13 deformable objects, AuRA reduces active human labor and wall-clock collection time per successfully reconstructed asset by 88.5% and 55.4%, respectively. Relative to the human-collected PhysTwin baseline, AuRA improves seven of nine metrics for future prediction under known gripper motion, including a 28.1% reduction in one-way $L_1$ Chamfer distance. It also achieves similar mean silhouette IoU and Chamfer distance under the evaluated unseen interactions. These results highlight the potential of autonomous robot interaction to support more efficient digital-twin acquisition with comparable reconstruction quality.

0.24