CS Paper Radar

Unified Trajectory and 3D Pose Forecasting for Multi-Agent Sports Analysis

Marc Peral, Guillem Capellera, Luis Ferraz, Antonio Rubio, Stella Graßhof, Dan Witzner Hansen, Antonio Agudo

cs.CV ·submitted 2026-10-21 22:57 UTC · v1
published

Abstract

Forecasting future human motion requires jointly reasoning about an agent's global trajectory and 3D body pose, which are tightly coupled, particularly in team sports where multiple interacting players introduce complex social dynamics. We exploit a socio-temporal encoder that fuses trajectory and pose at the input level and models them jointly through shared factorized attention. The model is further enhanced by a per-joint loss reweighting scheme that emphasizes dynamic joints and prevents degenerate solutions, and a frequency-domain trajectory smoothing term that improves trajectory prediction while indirectly benefiting pose forecasting. To ensure fair comparison with prior joint forecasting methods, we identify and correct evaluation biases arising from overlapping sequence construction, adopting instead a non-overlapping protocol and a deterministic, oracle-free strategy for multi-future prediction. Our approach improves over prior joint forecasting methods on both WorldPose and 3DPW, with the largest gains on pose accuracy, demonstrating that the proposed joint formulation generalizes beyond the densely interactive scenarios that motivate it.

Comment
WACV 2027

Closest papers

Nearest neighbours by meaning across the whole index.

FlowMoP: Stochastic Multi-Person Motion Prediction

cs.CV cs.LG ·2026-11-06 ·Aadya Agrawal, Ho Kei Cheng, Alexander Schwing
top venue similarity 0.76
Abstract · Accepted at NeurIPS 2026. 22 pages, including references and appendix

Predicting the future motion of multiple people is inherently stochastic: the same observed history admits many plausible continuations depending on intent, coordination, and social context. Yet the dominant paradigm in multi-person 3D motion prediction remains deterministic, with existing methods producing a fixed number of outputs per agent. In contrast, we present a conditional flow matching framework that jointly generates diverse future motions for multiple agents, modeling both global trajectories and root-relative poses. A B-spline reparameterization compresses the generative state space while enforcing temporal smoothness, and a dual-stream transformer conditions each agent's predictions on observed neighbor motion for interaction-aware forecasting. Our method achieves state-of-the-art performance on CMU-Mocap (UMPM) and 3DPW, outperforming deterministic baselines on joint position error, pose error, and final displacement error despite producing a full predictive distribution. We additionally establish the first motion prediction baseline on WorldPose, a large-scale professional soccer dataset, where social conditioning demonstrably concentrates predictive uncertainty along interaction-constrained directions.

0.35

From Where to How: Continuous 4D Interaction Forecasting from Egocentric Video

cs.CV cs.AI ·2026-09-08 ·Qiaohui Chu, Haoyu Zhang, Meng Liu, Haoxiang Shi, Dongmei Jiang +1 more
datasetcode/data similarity 0.71
Abstract

Egocentric 4D interaction forecasting aims to anticipate both where future interactions will occur in 3D and how the human body will move to realize them, providing an important capability for assistive robotics and human-computer interaction. Existing methods struggle to translate semantic understanding into precise continuous 3D localization and to balance motion diversity with structural consistency in pose forecasting. More fundamentally, these tasks are often modeled separately, leaving the continuous geometric and temporal correspondence between interaction locations and body motion insufficiently captured. To address these challenges, we introduce Coherent4D, a large-scale egocentric dataset for continuous 4D interaction forecasting, comprising approximately 233K samples across three domains. Each sample pairs a sequence of future 3D interaction locations with corresponding full-body poses, aligned in time and expressed in a shared coordinate system. We also provide evaluation metrics in continuous space. Building on this formulation, we propose HIGFlow, a Hand Interaction Guided Residual Flow framework that models forecasting as a cascaded where-to-how process. HIGFlow first forecasts continuous future interaction locations by combining semantic grounding with short-horizon visual dynamics, and then uses the predicted location sequence to condition a deterministic motion anchor and residual Flow Matching for diverse yet structurally consistent full-body motion forecasting. Extensive experiments across all three domains demonstrate consistent improvements over representative baselines on both location and pose forecasting, while ablations validate the contributions of the proposed components. The project page is available at https://corrineqiu.github.io/from-where-to-how/.

0.29

Adaptive Interaction Network for Human Motion Prediction During Human-Robot Collaboration

cs.CV ·2026-10-12 ·Mengyuan Liu, Yangting Lin, Qiongjie Cui
publishedcode/data similarity 0.68
Abstract · 12 pages, 10 figures, 10 tables. Author accepted manuscript. Published in IEEE Transactions on Image Processing. Code: https://github.com/Ly…

Human motion prediction during human-robot collaboration is critical for achieving safe and efficient interactions in shared environments. Unlike previous studies that have primarily focused on humans and objects, we focus on the robot-aware human motion prediction task, which explicitly models the influence of robots on human motion. This task presents new challenges, such as capturing human-robot heterogeneity and modeling complex spatial interactions. To address these issues, we develop an adaptive interaction network (AINet) model that consists of two branches: a primary branch that predicts future human motion and an auxiliary branch that estimates robot trajectories. The two branches are jointly optimized and coupled via a Local-Global Spatial Interaction (LGSI) Module, which effectively captures fine-grained and global contextual dependencies between human and robot motion sequences. Then, we introduce an Adaptive Weighted Aggregation (AWA) Module to dynamically fuse motion features using context-dependent weights, thereby increasing adaptability across diverse scenarios. Furthermore, we adopt a coarse-to-fine prediction strategy, in which a coarse pseudo-trajectory is first predicted, followed by the refinement of detailed human poses based on that trajectory. Extensive experiments on three challenging datasets reveal that our proposed method achieves superior performance, validating its effectiveness in human-robot collaboration scenarios. Our code is available at: https://github.com/LyTingHub/AINet.

0.47

Throwing4: Phase-Aware Pose Modeling for Throw-Distance Prediction in Elite Athletics

cs.CV ·2026-10-12 ·Bikash Kumar Badatya, Kartike Tiwari, Jyotirmoy Amin, Ravi Hegde
published similarity 0.66
Abstract · Accepted for publication in Sports Engineering

Elite athletics throwing provides a structured biomechanical action sequence together with an objective continuous outcome, namely throw distance, making it a suitable setting for performance-oriented sports analytics. In this work, we investigate whether explicit phase alignment of pose dynamics improves throw-distance prediction under realistic broadcast conditions. To support this study, we curate Throwing4, a dataset of 500 elite throwing clips spanning javelin, discus, hammer, and shot put, with corresponding two-dimensional pose sequences, frame-level phase annotations, and official distance labels. We formulate throw-distance prediction from broadcast-derived pose as a regression problem and compare full-sequence modeling against a phase-conditioned approach that partitions each attempt into four technique stages and fuses phase-specific motion representations. Experiments across multiple graph-based pose encoders show a consistent reduction in prediction error when phase structure is incorporated, indicating that phase-aligned modeling provides a more stable representation than treating the entire motion as an unstructured sequence. These findings suggest that explicit biomechanical phase information is a useful inductive prior for pose-based performance modeling in elite athletics throwing. Beyond score prediction, the dataset and protocol provide a basis for future work on fine-grained technique analysis, interpretable sports analytics, and robust modeling from broadcast video.

0.31

Does 3D Pose Improve Action Valuation in Football? An Empirical Study of Pose Representations and Learning Objectives

cs.CV cs.LG ·2026-10-28 ·Rikuhei Umemoto, Kenjiro Ide, Soham Majumder, Ulf Brefeld, Keisuke Fujii
similarity 0.66
Abstract · Submitted to IEEE Access

Football action valuation estimates players' contributions to future scoring. Existing multi-agent deep reinforcement learning (MADRL) models use 2D positions and velocities, omitting posture information relevant to imminent actions. We investigate whether adding 3D pose to a SARSA-based model improves observed-action values relative to attack returns: goals, conceded goals or final expected possession value. Using UEFA Champions League 2022/23 event, tracking and 3D pose data, we compare six representations (no pose, raw joints, torso and head orientation, and spatial-temporal graph convolutional network (ST-GCN) embeddings) across five learning settings and three seeds. The settings vary temporal-difference targets and action-prediction loss, with Monte Carlo return regression as a reference. Across 50 on-ball and off-ball comparisons, pose robustly improved return prediction in only three, all using torso and head orientation, while worsening it in twelve, mostly with ST-GCN. Training choices had greater effects: removing the action-prediction loss raised on-ball skill of position-only values from 0.003 to 0.040 and their correlation with expected goals from 0.23 to 0.60; the improvement in return prediction persisted at matched training length. Separately, pose improved prediction of the on-ball action 0.2 s ahead (macro AUC 0.914 to 0.934 with raw joints), while the passer's pose during execution improved pass-success prediction (AUC 0.844 to 0.856). Within this training protocol, pose informed imminent actions but rarely improved observed-action values.

0.27

UT-HMR: Uncertainty-Guided Temporal Modeling for Robust 3D Human Mesh Recovery

cs.CV ·2026-10-26 ·Zhibo Wang, Sirisha Rambhatla, Yuhao Chen, Maria Koshkina, David Clausi
published similarity 0.66
Abstract · Accepted at WACV 2027

Monocular 3D human mesh recovery (HMR) remains unreliable under weak observations caused by long camera-to-person distances, occlusion, and motion blur. In challenging scenarios such as sports, these factors make the problem highly ambiguous, as different 3D poses may yield similar 2D projections. This ambiguity is especially severe for one-stage HMR, which recovers all meshes directly from the full image without person crops or external 2D cues. To handle this ambiguity, we present UT-HMR ( Uncertainty-guided Temporal HMR), to our knowledge the first one-stage HMR framework to capture correlated pose uncertainty across joints. Each person query predicts a Gaussian distribution over the full SMPL pose with a compact low-rank-plus-diagonal covariance, which is converted into joint-wise precision weights for heteroscedastic supervision while keeping inference deterministic. Temporal early fusion further injects neighboring-frame evidence before pose decoding, supplying the visual evidence needed to resolve the ambiguity that the uncertainty captures. With a lightweight ViT-B backbone, UT-HMR achieves state-of-the-art results on VIP-HockeyPose-v2 (ice hockey) and WorldPose (soccer), reducing MPJPE, PA-MPJPE, and PVE by up to 4.7%, 3.8%, and 5.6%, and remains competitive on 3DPW against methods with larger backbones and higher input resolutions.

0.29

Capturing Uncertainty in Human Motion for Representation Learning in Soccer

cs.CV ·2026-08-11 ·Yizhou Xu, Lars Bretzner, Tiesheng Wang, Atsuto Maki
similarity 0.66
Abstract

This paper presents a self-supervised representation learning framework for understanding 3D skeleton-based human motion in soccer, using future motion prediction as the learning objective. Since human motion is inherently uncertain, accounting for multiple plausible futures is essential for capturing the underlying motion dynamics and learning effective representations. To this end, we introduce a conditioning module for motion prediction that models a probabilistic distribution over discretized future motions in 3D Euclidean space, learning multimodality with explicit supervision from future trajectories. Experiments on large-scale soccer player tracking data show that our approach substantially improves motion prediction accuracy. Moreover, the learned representations effectively transfer to multiple soccer downstream applications, demonstrating strong cross-task generalization.

0.04

Grasping the Future: Joint 3D Hand-Object Forecasting From Internet Videos

cs.CV ·2026-11-09 ·Rustin Soraki, Matthew Wallingford, Ryan Aby, Brandon Chee, Jason Ren +3 more
similarity 0.65
Abstract

To interact with an object, we need to anticipate both how it will move and how our hands should move with it. Human videos capture these relationships across everyday activities, offering a natural source for learning to predict future interactions. Learning from them requires recovering motion from completed interactions and using it to anticipate what could happen next. We present $\textit{Grasping-the-Future}$, a unified generative model of joint 3D hand-object motion, including wrist motion, finger articulation, and object position and orientation. Through masked spatiotemporal completion and flow matching, we train a single model for several prediction tasks: forecasting future interactions, predicting hand motion given object motion or vice versa, generating both trajectories toward a desired final object pose, and completing partially observed interactions. We explicitly model hand-object relations and incorporate contact constraints to help preserve grasp relationships during motion. To obtain 3D supervision from internet videos, we develop a scalable pipeline that combines pretrained perception models with joint geometric optimization to reconstruct completed interactions. We combine these recovered trajectories with annotated interaction datasets as training data for all tasks. We demonstrate strong performance across all tasks, outperforming released forecasters on datasets with 3D annotations. Moreover, we show geometric and physical plausibility of the predicted trajectories by retargeting them to a robotic embodiment. Videos are available at https://grasping-the-future.github.io.

0.12