CS Paper Radar

Event-guided Neural Video Compression

Jiyun Kong, Jungwoo Kim, Enes Eray Demirtas, Touradj Ebrahimi, Jong-Seok Lee

eess.IV cs.CVcs.MM ·submitted 2026-10-01 05:24 UTC · v1
code/data

Abstract

Neural video codecs derive motion and temporal contexts mainly from RGB frames, leaving room for cross-modal guidance from complementary temporal observations. Event streams can provide such observations by recording brightness changes between frames. In this work, we propose an Event-guided Neural Video Codec (ENVC) that uses events shared by the encoder and decoder to improve RGB compression efficiency. For motion coding, ENVC forms an event-guided motion prior and codes the remaining motion residual. For frame coding, an event-conditioned predictor supplies multi-scale features for gated temporal context refinement. To support training and evaluation on standard video datasets, we synthesize paired RGB-event data and assess its predictive utility through comparisons with real events. Across six benchmarks, ENVC achieves average BD-rate savings of 39.13% using PSNR-RGB and 67.63% using LPIPS relative to DCMVC. Further analyses show that our gains persist on large-motion sequences and that ENVC effectively learns to integrate event information. These results demonstrate the potential of events as a complementary modality for reducing the RGB coding rate. Our model and code are available at https://github.com/kjungwoo03/ENVC.

Comment
28 pages. 21 figures

Closest papers

Nearest neighbours by meaning across the whole index.

HR-NVC++: Unified Ultra-High-Resolution Neural Video Coding with Confidence-Guided Spatial-Temporal Cues

eess.IV ·2026-11-11 ·Feng Ye, Lijiong Chen, Kai Zhang, Li Zhang, Chuanmin Jia
similarity 0.76
Abstract

Ultra-high-resolution neural video coding (NVC) remains challenging because temporal correspondence becomes increasingly unreliable as spatial resolution increases. Although recent neural B-frame codecs have improved bi-directional temporal prediction, complex motion and fine-grained structures still lead to unreliable temporal references, while spatial information of the current frame remains underexploited. In this work, we propose HR-NVC++, a unified ultra-high-resolution NVC approach that supports both fidelity-oriented and perceptual-quality-oriented compression with a shared spatial-temporal context. The context integrates bi-directional temporal cues with complementary spatial information: confidence-guided fusion adaptively balances the forward and backward temporal cues according to their estimated correspondence reliability, while a spatial complement branch enriches regions that temporal references fail to explain. The resulting context serves as a common basis for two compression modes. Built upon this shared context, the fidelity mode retains entropy-based latent coding for pixel-accurate compression, whereas the perceptual mode removes additional latent coding and performs reconstruction directly from the context. This design demonstrates that a sufficiently informative spatial-temporal context already carries the dominant content required for perceptually favorable reconstruction, allowing different compression objectives to share the same contextual representation. Extensive experiments on UVG-4K and JVET Class A1/A2 show that HR-NVC++ achieves an average PSNR BD-rate saving of 13.01% over the H.266/VVC reference model VTM-13.2 under the random access configuration, while delivering substantially better perceptual quality and temporal consistency than existing neural video codecs.

0.08

BiCRVC: An Efficient Bidirectional Neural Video Compression Framework via Coupled Representation Coding

eess.IV ·2026-08-17 ·Wei Jiang, Junru Li, Kai Zhang, Li Zhang
similarity 0.73
Abstract · Fix some typos

Neural video compression (NVC) has achieved strong compression performance, but practical random-access coding still faces two technical challenges: existing bidirectional NVCs (BVCs) usually require costly motion-first decoding, and reliable motion estimation is difficult under long-range bidirectional prediction. To address these issues, we present BiCRVC, an efficient bidirectional neural video compression framework based on coupled representation coding. Instead of coding motion and frame information with two separate codecs, BiCRVC transforms the motion representation and the current-frame latent into a unified latent representation for entropy coding. This design enables motion and frame information to be decoded from the same bitstream with one unified codec, while still reconstructing motion-aligned contexts for frame decoding. To improve motion accuracy, we introduce multi-candidate motion estimation (MCME), which combines multi-scale motion estimation and parallel accumulated motion estimation to better handle diverse and long-range motions. To reduce motion coding overhead, we further propose bidirectional motion feature propagation (BMFP), which reuses previously decoded motion features at both the encoder and decoder as temporal priors for conditional motion coding. In addition, coupled distortion training and random GOP structure training are used to encourage joint motion-frame coding and improve adaptation to hierarchical random-access structures. Experiments show that BiCRVC achieves better compression performance than state-of-the-art BVCs while providing about 30 times faster 1080p decoding than recent BVCs.

0.30

Lossy Event Compression: From Event Stream Distortion to Task Performance

cs.CV eess.IV ·2026-08-28 ·Zahra Rezaee, Catarina Brites, João Ascenso
published similarity 0.73
Abstract

Event cameras generate asynchronous, sparse data streams with microsecond temporal resolution, but in moderate-to-high motion scenes they can produce as many as hundreds of millions of events per second, creating significant bandwidth and storage challenges. Lossy compression is therefore essential for practical deployment, yet existing event stream distortion metrics fail to reliably predict compression-induced degradation at the task level, forcing codec optimization to rely on expensive task-specific evaluations. To address this gap, this paper introduces two fundamentally different event compression pipelines: i) an aggregation-based pipeline that converts the event stream into polarity-based histogram frames for compression with the conventional image codec JPEG 2000, and ii) a frame-free point cloud-based pipeline that codes events natively as 3D points using the octree-based codec G-PCC. Both pipelines are then assessed within a unified task-driven evaluation framework that relates event stream distortion to downstream application performance across four representative tasks: i) video reconstruction, ii) object detection, iii) optical flow estimation, and a delay-sensitive task iv) asynchronous feature tracking under a reference-relative protocol. Building on this framework, five classification-based distortion metrics are applied to event compression for the first time, to the best of the authors' knowledge, and benchmarked against existing event stream metrics. Experimental results demonstrate that the proposed metrics reliably predict compression-induced task degradation across different coding frameworks. This demonstrates that event stream distortion assessment can be an efficient alternative to repeated task-specific evaluation, providing direct guidance for the development and optimization of future event data coding solutions.

0.35

tcnerv:dual-domain temporal context modeling for implicit neural video compression

cs.CV ·2026-09-15 ·Xuezhi Xiang, Yixin Zhao, Heqi Xiang, Jiayao Liu, Shanjun Zhang
similarity 0.73
Abstract

Video compression aims to minimize reconstruction distor tion under a constrained bit rate. Existing video implicit neural representations (INRs) often decode frames independently, leaving intermediate features unconditioned on previous reconstructions and content embeddings without explicit temporal prediction. We propose TCNeRV, which exploits reconstructed context in both feature and embedding domains. Its multi-scale temporal-context fusion (MTCF) module injects gated historical features at multiple decoder scales, while temporal embedding-residual coding (TERC) predicts each content embedding and codes only its residual. With approximately 3M parameters, TCNeRV achieves an average PSNR of 36.08 dB on the UVG dataset, outperforming HNeRV-Boost by 2.20 dB. It reduces BD-rate by 22.06%, 66.73%, and 29.85% relative to HM, DCVC, and HiNeRV, respectively, demonstrating competitive rate-distortion performance with limited model capacity.

0.18

Video Codec-Guided Training-Free Token Compression for Video Large Language Models

cs.CV ·2026-10-09 ·Xihua Sheng, Ruijie Liu, Chang Wen Chen
similarity 0.71
Abstract

Video large language models (VLLMs) rely on long visual token sequences to understand complex temporal content, which makes visual token compression important for efficient inference. Existing training-free compression methods mainly reduce the number of tokens sent to the language model, but the compression process itself can still require many visual feature comparisons, especially for intra-frame token protection and inter-frame temporal token merging. This paper proposes a video codec-guided training-free token compression method for VLLMs that uses coding information already available in compressed video as a low-cost search prior. The method aligns reference-frame dependencies, motion vectors, and residuals with the visual token grid. Residual information forms compact candidate pools for token protection, while motion information is propagated along reference-frame chains to construct localized candidates for temporal token merging between sparsely sampled frames. Local visual feature similarity comparison is then used within these candidates to make the final compression decisions, without modifying the base VLLM. Experiments on three representative VLLMs and four video-understanding benchmarks show that the proposed method achieves performance comparable to the state-of-the-art training-free baseline FlashVID, and in some settings surpasses it, while substantially reducing the feature comparisons required by token protection and temporal matching. These results demonstrate that codec information can effectively improve the accuracy-complexity tradeoff of video token compression.

0.14

UniDynamics: Event-RGB Fusion for Unified Future 4D Dynamic Scene Generation

cs.CV ·2026-10-02 ·Daikun Liu, Xin Zhan, Teng Wang, Xiaoping Wang, Changyin Sun
top venuecode/data similarity 0.71
Abstract · 19 pages, 6 figures, conference, code: https://github.com/KK-xi/Unidynamics

We propose UniDynamics, a diffusion-based framework for future 4D dynamic scenes (RGB, depth, and optical flow) generation from a single event-RGB pair, without requiring long histories or control priors as in existing methods, while explicitly modeling future motion fields. The core idea is to leverage event streams to offer an alternative motion prior for single-RGB extrapolation, and to enforce geometric and motion constraints throughout generation via multimodal modeling. Specifically, we design an Event Latent Enhancement (ELE) module to align and enhance event latents into diffusion-injectable conditioning features, providing robust initial motion priors and reliable texture/structure cues. We further introduce a Perceptual Dynamics Space (PDS) embedded in the multi-scale U-Net, which decouples and adaptively interacts depth and flow while continuously feeding back constraints to appearance features, improving geometric-motion consistency for physically plausible and spatiotemporally coherent prediction. Experiments on VKitti2 and DSEC demonstrate state-of-the-art performance, producing high-quality, temporally coherent, and 4D-consistent future predictions, especially under challenging high-speed motion blur.

0.53

TKCAM: Text and Keyframe to Camera Trajectory Generation

cs.CV ·2026-10-08 ·Haozhe Yang, Zhiyang Dou, Zekai Gu, Cheng Lin, Wenping Wang +2 more
top venuebenchmarkcode/data similarity 0.70
Abstract · 20 pages, 7 figures. Paper accepted to NeurIPS2026 (submission number 15461)

Generating high-quality and controllable camera motion is essential for AI-assisted cinematography, video synthesis, and 3D scene understanding. We introduce TKCAM, a Text- and Keyframe-conditioned CAMera-motion synthesis framework based on generative masked modeling. We represent camera dynamics using a 12-dimensional kinematic feature comprising position, velocity, and a continuous rotation representation and discretize them into hierarchical motion tokens via a Residual Vector Quantizer (RVQ). A two-stage masked transformer architecture then learns to reconstruct and refine these tokens, utilizing explicit self- and cross-attention modules for multimodal conditioning. A central feature of our framework is sparse visual keyframe conditioning: users can provide free-form text prompts together with RGB observations at selected timestamps, which provide temporally localized visual guidance for generating coherent in-between trajectories. Furthermore, to advance evaluation standards, we curate RealEstate10K-Cap, a large-scale text-camera dataset, and establish a cross-domain benchmark with a Universal CLaTr Evaluator. Extensive experiments demonstrate that TKCAM surpasses recent state-of-the-art baselines on Fréchet distance (FID), text-motion matching scores, and retrieval metrics (R@K), while additional analyses evaluate temporal smoothness and cross-domain generalization. Code and RealEstate10K-Cap dataset are available at https://github.com/linearalgebrayhz/TKCAM.

0.48

LiFR v2: Completion-Augmented Event Propagation for High-Rate Dense Prediction

cs.CV ·2026-09-22 ·Tao Wan, Xiaoshan Wu, Yifei Yu, Bo Wang, Xiaoyang Lyu +4 more
similarity 0.70
Abstract · 15 pages, 9 figures, 6 tables

High-rate dense perception in dynamic environments is limited by the low update rate of RGB cameras, as rapid scene changes can occur between frames. Event cameras offer temporally dense but spatially sparse measurements, complementary to spatially dense RGB observations. Direct fusion cannot fully exploit this complementarity, while event-guided propagation fails on newly appearing or disoccluded regions without valid RGB support. We present LiFR v2, a unified propagation-completion-memory framework for causal anytime and streaming dense prediction from an RGB keyframe and events. LiFR v2 introduces an Event-Guided Completion Module (EGCM) to recover task-relevant representations where propagation is unsupported, and a History Retrieval Module (HRM) to reuse completed representations across successive queries. The framework supports semantic segmentation, monocular depth estimation, and multi-task dense prediction, and we further introduce SHF-Emerge to evaluate rapid object emergence and disocclusion. LiFR v2 achieves 74.37% mIoU on DSEC and 56.13% on SHF-Emerge, improving LiFR-Seg by 1.85 percentage points on the latter, while reducing SHF-Emerge depth RMSE from 1.564 m to 1.118 m over the propagation baseline. It also exceeds 100 FPS for both segmentation and depth, demonstrating accurate and efficient high-rate perception beyond RGB frame rates.

0.26