CS Paper Radar

MvT: Event-based Multi-view Projection for Multiple Object Tracking

Muxi Zha, Banglei Guan, Minzu Liang, Zibin Liu, Yang Shang, Qifeng Yu, Laurent Kneip

cs.CV ·submitted 2026-10-27 14:26 UTC · v1
publishedcode/data

Abstract

Event cameras are increasingly used for Multiple Object Tracking (MOT), but their asynchronous event output often requires specialized methods. Existing processing methods primarily follow two paradigms, pseudo-frames and event-by-event. The former is the prevailing approach since its data format aligns with images, making image-based techniques applicable. However, it suffers from tracking failures when trajectories overlap or are spatially close on pseudo-frames. Facing this challenge, we propose a multi-view pipeline, Multi-view Tracking (MvT), which preserves the 2D data format to leverage image-based techniques directly while introducing additional spatio-temporal views to resolve tracking ambiguities in a single view. MvT comprises a Multi-view Projection (MvP) module and a Multi-view Fusion (MvF) stage. MvP encodes events into three complementary spatio-temporal views while mitigating the pattern discretization. Within MvF, multi-view results are unified into a 3D coordinate system, and tracklets are associated through an optimization model subject to specific criteria combination. Evaluations on four datasets, including our self-collected Small Objects Dataset (SOD), show that MvT seamlessly integrates image-based methods and outperforms existing non-learning and learning trackers in generalized scenarios, and effectively resolves the single-view tracking ambiguities. Being training-free, MvT is applicable when ground-truth annotation is infeasible, thereby highlighting its practical, data-efficient potential. Code is available at https://github.com/zhazhabiu/MvTracking.

Comment
16 pages, 20 figures, published to IEEE T-IP
DOI
10.1109/TIP.2026.3726353

Closest papers

Nearest neighbours by meaning across the whole index.

ARROW: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild

cs.CV ·2026-10-01 ·Ilya Fradlin, Christian Schmidt, Jens Piekenbrinck, Karim Knaebel, Gonzalo Martin Garcia +1 more
benchmarkcode/data similarity 0.74
Abstract · Project page at: https://www.vision.rwth-aachen.de/arrow

Dynamic scenes may be captured by a moving camera, multiple video streams, or images taken at different times. These observations reveal complementary aspects of scene geometry and motion, yet bringing them together requires establishing correspondence across viewpoints, capture times, and visibility changes. We introduce ARROW, a feed-forward model that unifies 3D reconstruction and 3D point tracking from arbitrary image sets. At its core is a novel order-invariant querying approach, which allows the association of queries with observations across arbitrary inputs. We show that exposing the model to more diverse sets of inputs during training results in improved task performance. Moreover, the resulting model is capable of generalization to a wider range of tasks including multi-view tracking. Trained with this strategy, ARROW establishes a new state of the art in 3D tracking on WorldTrack and TAPVid-3D and outperforms dedicated multi-view trackers on an adapted RGB-only MVTracker benchmark, while remaining competitive across 3D reconstruction tasks. Code and weights are publicly available.

0.44

Tracking-by-detection in Multi-object Tracking: Survey and Experiments

cs.CV ·2026-09-08 ·Yujin Yang, Kyujin Shim, Kangwook Ko, Changick Kim
survey similarity 0.73
Abstract

Multi-object tracking (MOT) is an essential computer vision task that simultaneously tracks multiple objects in video sequences, with various applications in surveillance, autonomous navigation, and human-computer interaction. The tracking-by-detection (TBD) paradigm, which combines object detection with temporal association, has emerged as a leading approach, driven by innovative algorithms. Despite recent progress, fair evaluation of TBD-based methods remains a challenge. Many studies introduce modules such as similarity metrics, data association strategies, or motion models, but they are often evaluated under inconsistent protocols, with different baseline trackers, hyperparameters, and datasets. Such inconsistencies obscure the genuine contribution of each module and hinder objective comparison. This survey systematically reviews TBD-based MOT techniques, including similarity measurements, data association, camera motion compensation, and interpolation strategies. Starting from a minimal baseline tracker, we fairly evaluate the contributions of each method across diverse datasets and accumulate well-balanced methods. Our findings establish a strong baseline tracker and provide a foundation for the principled design of robust and versatile MOT systems suitable for real-world deployment.

0.19

Selective Event Evidence and Persistent Temporal Priors for Event-Based Point Tracking

cs.CV ·2026-11-05 ·Xiaokai Song, Zhang Li, Leqi Liu, Yibin Ye, Kun Wang +1 more
code/data similarity 0.69
Abstract

Event cameras enable tracking under fast motion and challenging illumination. However, stable long-term point tracking remains challenging: event activity is sparse and varies with scene content and motion, complicating correspondence, while trajectory estimates can drift over long sequences. Together, these challenges motivate a tracker that selectively uses informative event evidence and retains trajectory constraints across temporal windows. We propose Selective Event Evidence and Persistent Temporal Priors (SEPT), an event-only framework for tracking any point (TAP) over long sequences. SEPT introduces two complementary modules: Task-Adaptive Event Evidence Modulation (TEEM) and Persistent Trajectory Prior (PTP). Using only tracking supervision, TEEM learns a bounded reliability field that weights the event tensor across space, time, and channels before feature extraction. PTP carries geometric information across overlapping windows by marginalizing a fixed local quadratic trajectory model with a Schur complement. Its visibility-aware correction refines overlap coordinates, while a clipped terminal velocity initializes coordinates at newly exposed timestamps. SEPT builds on the ETAP architecture. SEPT is trained exclusively on synthetic EventKubric data and evaluated on five datasets across TAP and feature tracking. Results show that SEPT outperforms the retrained ETAP baseline in average Jaccard for TAP and in feature age and expected feature age for feature tracking; it also surpasses several methods combining event and frame inputs on EC and EDS. The source code is publicly available at https://github.com/sssxxxkkkk/SEPT.

0.30

TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views

cs.CV ·2026-09-01 ·Skanda Koppula, Frano Rajic, Abdullah Faiz Ur Rahman, Yi Yang, Ignacio Rocco +7 more
benchmarkcode/data similarity 0.69
Abstract

Multi-camera systems are increasingly practical for robotics, AR/VR, and autonomous driving because complementary views reduce depth ambiguity and preserve visibility under occlusion. Existing point-tracking benchmarks, however, focus on a single video or static multi-camera rigs. None test long-term 3D point tracking across several synchronized views under camera motion. We introduce TAPVid-MV (Tracking Any Point in Video across Multiple Views), the first benchmark for this setting. It contains a curated set of 238 sequences, 1,004 calibrated camera streams, and 74,147 point tracks across seven subsets spanning indoor and outdoor domains, from robotics and human activity to driving and synthetic procedural scenes. We obtain these trajectories using dataset-specific auxiliary modalities: sensor depth, LiDAR, SLAM and SfM points, human meshes, posed object meshes, and simulation. Every sequence and trajectory is visually verified by human annotators. Across more than 30 baselines, no method comes close to solving the task. Surprisingly, existing multi-view point trackers do not consistently outperform monocular point trackers. By evaluating reconstruction and point tracking on the same datasets, TAPVid-MV helps distinguish errors in recovered geometry from errors in point correspondence. Through this joint analysis, we identify geometry recovery as a major bottleneck for accurate 3D point tracking. Beyond multi-view 3D point tracking, our released annotations support monocular 2D and 3D point tracking, future-trajectory prediction, and 4D reconstruction. Project page: https://tapvidmv.github.io/

0.46

Tracking Objects Through Assembly and Decomposition

cs.CV ·2026-11-19 ·Cheryl Lam, Yihong Sun, Bharath Hariharan
benchmark similarity 0.67
Abstract

Tracking objects through transformations is essential for understanding real-world dynamics. Recent work has extended traditional object tracking to detect state changes. However, existing methods require an initial object prompt and detect only its decompositions, missing assembly events, which begin with multiple objects rather than one. We therefore introduce a prompt-free formulation of this task, where the goal is to recover all object tracks and detect both decomposition and assembly events given only a video. Our key insight is that assembly is decomposition in reverse. Building on this insight, we present Atlas, a zero-shot system that constructs and jointly reasons over forward and reverse spatiotemporal partitions to recover object tracks and their transformations. We further introduce OmniST, a benchmark with dense annotations of objects and both transformation types. On OmniST, Atlas recovers more than twice as many state changes as prior methods while remaining competitive on existing benchmarks that only evaluate decomposition.

0.28

Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

cs.CV ·2026-10-27 ·Siyoon Jin, Dahyun Chung, Honggyu An, Sangbeom Lim, Wonjin Nam +3 more
code/data similarity 0.67
Abstract · Project Page : https://cvlab-kaist.github.io/XTrack/

Cross-view object correspondence identifies the same object across views. However, existing methods often rely on frame-wise matching or sequential tracking, which can propagate correspondence errors under appearance changes or occlusions. Therefore, we introduce cross-view temporal object correspondence, which requires preserving object identity across both views and time from a query mask. To address this, we propose XTrack, a unified framework which repurposes a pretrained video DiT with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. However, naive video DiTs do not guarantee accurate cross-view grounding and often attend to semantically similar yet incorrect objects across views. Thus, we introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies to ensure reliable grounding. Finally, beyond these architectural enhancements, we also tackle the lack of dense annotation by introducing a pseudo-label curation pipeline that converts sparse Ego-Exo4D annotations into dense supervision to enable effective video-level training. Our results demonstrate the effectiveness of XTrack as a unified framework, ensuring robust temporal consistency and identity preservation.

0.25

Video Stitching from Multiple Moving Cameras

cs.CV ·2026-11-06 ·Shira Ifergane, Roy Amoyal, Shahar Benishay, Oren Freifeld
top venuebenchmarkdatasetcode/data similarity 0.66
Abstract · Accepted to NeurIPS 2026

We address geometric video alignment for the same scene captured by multiple freely moving cameras. Existing methods, which are based on image stitching or video stabilization, are typically limited to two cameras or must assume the restrictive setting of a fixed linear ordering of the cameras. We propose MViS, a lightweight test-time optimization framework for joint multi-view alignment with temporal consistency. MViS uses a spatial-transformer-inspired net to jointly process frames from all views within a short temporal window and predict per-view, per-time homographies. Optimization proceeds in two stages: geometry-guided alignment followed by appearance-based refinement, with both stages enforcing multi-view and temporal consistency across the sequence. The method requires no scene-specific hyperparameter tuning, handles arbitrary and time-varying view-intersection graphs, and remains robust under rapid camera motion. On a standard two-view video benchmark, MViS matches or surpasses the state of the art. To facilitate evaluation in more complex scenarios, we introduce an annotated dataset with more than two freely moving cameras and establish the first benchmark for this setting. Project page: https://bgu-cs-vil.github.io/MViS-project/ .

0.42

TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

cs.CV cs.AIcs.RO ·2026-09-24 ·Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta, Fan Zhang, Tanner Schmidt +3 more
code/data similarity 0.66
Abstract

Existing point tracking models face a fundamental tradeoff: they can either track a sparse set of query points over long horizons, or track all points across only short clips. We introduce TrackEverything, a 3D point tracker that breaks this trade-off by representing videos as persistent 3D scene tracks in world coordinates. Grounded in the insight that videos are 2D projections of an underlying 3D world, TrackEverything decouples model complexity from video duration, allowing it to scale with unique physical scene geometry instead. Our approach introduces three key innovations. First, we employ a voxelization-based de-duplication mechanism at sliding-window boundaries to merge co-located tracks, preventing repeated observations of the same surface from redundantly accumulating. Second, we decompose tracking into an endpoint refiner that predicts each point's destination and static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories exclusively for dynamic points. Third, we propose 3D WAFT, replacing memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud. To the best of our knowledge, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.

0.50