CS Paper Radar

Resolution-Flexible Decoding for Hybrid Neural Video Representations

Taiga Hayami, Masaya Takabe, Hiroshi Watanabe

eess.IV cs.CV ·submitted 2026-09-20 11:10 UTC · v2
published

Abstract

Neural video representations (NVRs) represent videos using neural network parameters and, in hybrid formulations, frame-wise latent embeddings. In conventional hybrid NVRs, the spatial size of frame-wise latent embeddings and the decoder upsampling schedule depend on the target resolution, potentially requiring large and non-uniform upsampling factors. In this paper, we propose a resolution-flexible decoder framework for hybrid NVRs. The decoder is constructed from uniform \(2\times\) upsampling stages, whose target feature sizes are obtained by tracing the spatial resolution backward from the final output resolution. After each upsampling stage, the feature map is aligned with the target size by minimal padding or cropping when necessary. To support this progressive decoding process, we further use intermediate reconstruction supervision and a reconstruction-difficulty-aware frame sampling strategy based on recent frame-wise losses. The framework preserves the basic representation format of hybrid NVRs and can therefore be applied to different backbones. Experiments on the UVG dataset at resolutions of \(1080\times1920\) and \(810\times1440\) show that the proposed approach improves reconstruction quality over the corresponding NVR baselines.

Comment
Accepted to VCIP 2026

Closest papers

Nearest neighbours by meaning across the whole index.

tcnerv:dual-domain temporal context modeling for implicit neural video compression

cs.CV ·2026-09-15 ·Xuezhi Xiang, Yixin Zhao, Heqi Xiang, Jiayao Liu, Shanjun Zhang
similarity 0.70
Abstract

Video compression aims to minimize reconstruction distor tion under a constrained bit rate. Existing video implicit neural representations (INRs) often decode frames independently, leaving intermediate features unconditioned on previous reconstructions and content embeddings without explicit temporal prediction. We propose TCNeRV, which exploits reconstructed context in both feature and embedding domains. Its multi-scale temporal-context fusion (MTCF) module injects gated historical features at multiple decoder scales, while temporal embedding-residual coding (TERC) predicts each content embedding and codes only its residual. With approximately 3M parameters, TCNeRV achieves an average PSNR of 36.08 dB on the UVG dataset, outperforming HNeRV-Boost by 2.20 dB. It reduces BD-rate by 22.06%, 66.73%, and 29.85% relative to HM, DCVC, and HiNeRV, respectively, demonstrating competitive rate-distortion performance with limited model capacity.

0.18

CoANeRV: Coordinate-Aware Token-Space Neural Video Representation

cs.CV ·2026-08-14 ·Jialong Guo, Ke Liu, Mengxuan Li, Jiajun Bu, Haishuai Wang
code/data similarity 0.68
Abstract · 27 pages, 13 figures. Code is available at https://github.com/jialong2023/CoANeRV

Neural representations for videos (NeRV) have shown strong reconstruction fidelity by storing video-specific information in network weights. However, existing formulations typically require either costly per-video optimization or video-specific weight generation, making it difficult to scale to efficient amortized video representation. We propose CoANeRV, a coordinate-aware token-space framework that adapts the broader token-conditioned neural-field paradigm to amortized video representation. CoANeRV forms compact video tokens in one feed-forward pass and uses a shared coordinate-conditioned decoder to reconstruct continuous spatio-temporal queries, avoiding per-video decoder optimization or generation while retaining coordinate-level reconstruction flexibility. To make token-space reconstruction effective, CoANeRV introduces a coordinate-aware decoding architecture that aligns spatio-temporal queries with video tokens through axis-adaptive positional encoding and temperature-modulated cross-attention. Block-wise coordinate querying further reduces peak attention memory, making high-resolution reconstruction practical. Experiments on diverse video datasets show that CoANeRV consistently improves reconstruction quality over prior feed-forward NeRV and INR baselines, reduces peak memory compared with attention-based coordinate decoders, and provides efficient amortized encoding without per-video optimization. These results support the proposed video-specific combination of feed-forward token formation, spatio-temporal coordinate retrieval, and memory-bounded dense querying. The code is available at https://github.com/jialong2023/CoANeRV.

0.30

V-RAE: Rethinking Video Latent Spaces for Generation

cs.CV ·2026-08-13 ·Minghui Guo, Shengqiong Wu, Hao Fei
code/datahighly cited similarity 0.66
Abstract · 26 pages, 8 tables, 13 figures, project page: https://v-rae.github.io/

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.

0.65

Rethinking Video Storage for AI Training: Breaking Cross-Frame Dependencies for Efficient Random Access

cs.CV eess.IV ·2026-10-21 ·Zetian Song, Qize Yu, Yue Chen, Bohan Zhou, Bowen Ping +8 more
code/data similarity 0.66
Abstract · 18 pages, including supplementary material. Project page: https://skywalker-yqz.github.io/codec4AI/ Code: https://github.com/kaname-madoka18…

Large-scale AI training increasingly relies on massive video datasets with random temporal sampling patterns. However, existing video storage paradigms are optimized for sequential playback and employ sophisticated inter-frame reference structures to maximize compression efficiency. When applied to the random-sampling workloads of AI training, these paradigms face substantial decoding-efficiency bottlenecks. Workarounds such as independent JPEG frames and short groups of pictures (GOPs) reduce decoding work at the cost of larger storage and transfer volumes. In this paper, we identify two sources of decode amplification: sequential reconstruction of unrelated frames and reconstruction of the reference chains required by sampled targets. We address both sources by co-designing the decoding schedule and video reference structure. Dependency-aware sparse decoding reconstructs only requested frames and their required references, skipping unrelated reconstruction without transcoding the source videos. Optional offline Dependency-Decoupled Random Access (DDRA) encoding further reduces reference reconstruction by restricting each interior frame to two bounding intra-coded anchors. Together, sparse decoding and DDRA retain inter-frame prediction while bounding reconstruction to at most three frames per requested target, independent of the intra period. Experiments on a large-scale robotics corpus show that, at comparable reconstruction quality, our method achieves $1.45\times$ the loading throughput of LeRobot v3 with 58.7% less media payload.

0.45

Video Compression with Graph-inspired Neural Representation

cs.CV ·2026-09-02 ·Changqi Wang, Ge Gao, Fan Zhang, Yue Li, Kai Zhang +2 more
similarity 0.64
Abstract · 12 pages, 6 Figures

Implicit Neural Representations (INR) provide a compact and content-adaptive paradigm for video compression, typically representing a video through shared network parameters and frame-indexed embeddings. Compared to conventional or autoencoder-based codecs, these approaches exploit temporal redundancy within videos in an implicit manner, which potentially results in sub-optimal compression performance. In this paper, we propose G-NeRV, a graph-inspired INR that explicitly improves temporal redundancy exploitation in the implicit latent space. Motivated by the total correlation principles in information theory, we construct a temporal neighborhood over frame embeddings and perform message passing to aggregate reusable information from neighboring frames through an adaptive gate controlling the injection of neighboring information. Inspired by the reference frame buffer in conventional video coding, a memory bank mechanism has been further designed to enable efficient temporal-neighbor retrieval under random frame-index sampling in INR training. This new representation model has been integrated into an advanced representation compression framework and compared with existing conventional and neural video codecs. The results show that the G-NeRV codec outperforms the state-of-the-art INR-based codec, NVRC, and the latest standard video codec, VVC VTM, by 8.86\% and 14.68\% (in BD-rate), respectively, measured by PSNR on the UVG dataset.

0.23

Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding

cs.CV cs.LGcs.MM ·2026-10-12 ·Shuo Gao, Chenhao Zheng, Zhongzheng Ren
published similarity 0.63
Abstract · NeurIPS 2026 (https://zeld0n.github.io/framepack-U/)

Hour-long videos, streaming feeds, and movie-length content are pushing vision-language models (VLMs) past their breaking point -- the standard recipe of encoding each frame into visual tokens yields sequences that grow linearly with video length, causing prohibitive memory consumption and inference latency. Existing approaches, such as token reduction and keyframe selection, shorten the token sequence but leave the O(n) scaling intact, failing to resolve the fundamental bottleneck. We propose FramePack-U, a strikingly simple, training-free frame packing framework that compresses arbitrarily long videos into a fixed size visual representation at constant memory cost. FramePack-U requires no fine-tuning or architectural modification -- it plugs directly into off-the-shelf VLMs at inference time. Inspired by FramePack from the generation literature, we propose a content-aware compression strategy for visual understanding under a fixed token budget. Specifically, FramePack-U (1) scores each frame by its semantic similarity to the text prompt and ranks frames in a diversity-driven manner, and (2) packs the selected frames at varying resolutions into a fixed-length context window. This concentrates the memory budget on frames that are both instruction-relevant and visually diverse. While FramePack-U can in theory pack arbitrarily many frames into the fixed budget, we identify an empirical instantiation that generalizes robustly across settings. Extensive experiments on LongVideoBench, MLVU, and Video-MME show that FramePack-U consistently outperforms full-resolution baselines and prior token reduction and keyframe selection methods across diverse model families, parameter scales, and frame budgets. Notably, on Qwen3.5-35B-A3B with a 128-frame budget, FramePack-U yields gains of +12.9\%, +10.6\%, and +9.7\% over the official model on the three benchmarks, respectively.

0.42

Video Codec-Guided Training-Free Token Compression for Video Large Language Models

cs.CV ·2026-10-09 ·Xihua Sheng, Ruijie Liu, Chang Wen Chen
similarity 0.63
Abstract

Video large language models (VLLMs) rely on long visual token sequences to understand complex temporal content, which makes visual token compression important for efficient inference. Existing training-free compression methods mainly reduce the number of tokens sent to the language model, but the compression process itself can still require many visual feature comparisons, especially for intra-frame token protection and inter-frame temporal token merging. This paper proposes a video codec-guided training-free token compression method for VLLMs that uses coding information already available in compressed video as a low-cost search prior. The method aligns reference-frame dependencies, motion vectors, and residuals with the visual token grid. Residual information forms compact candidate pools for token protection, while motion information is propagated along reference-frame chains to construct localized candidates for temporal token merging between sparsely sampled frames. Local visual feature similarity comparison is then used within these candidates to make the final compression decisions, without modifying the base VLLM. Experiments on three representative VLLMs and four video-understanding benchmarks show that the proposed method achieves performance comparable to the state-of-the-art training-free baseline FlashVID, and in some settings surpasses it, while substantially reducing the feature comparisons required by token protection and temporal matching. These results demonstrate that codec information can effectively improve the accuracy-complexity tradeoff of video token compression.

0.14

CASR: Content-Adaptive Neural Super-Resolution Post-Filter for Versatile Video Coding via Low-Rank Overfitting

eess.IV ·2026-09-29 ·Khoa Pham-Dinh, Francesco Cricri, Maria Santamaria, Honglei Zhang, Hamed R. Tavakoli +3 more
similarity 0.62
Abstract · Accepted to the 2026 IEEE 28th International Workshop on Multimedia Signal Processing (MMSP 2026), Best Student Paper Award

The use of super-resolution as a post-processing step following a video codec allows videos to be encoded at reduced spatial resolution at the encoder side and to be reconstructed and upsampled to the original resolution at the decoder side. In this way, the required bitrate is reduced and the quality of the reconstructed frames is improved without modifying the core coding architecture. However, generic SR models are typically trained offline and lack sufficient adaptability to the diverse content characteristics and compression artifacts produced by video codecs, which limits their effectiveness during test time. To address the limitation, this paper proposes CASR (Content-Adaptive Super-Resolution), a content-adaptive SR post-filter framework for Versatile Video Coding (VVC), based on encoder-side overfitting on each input sequence. In order to limit the bitrate overhead required for signalling the content adaptation signal, i.e. the weight-update, Low-Rank Adaptation (LoRA) is leveraged. The method freezes the convolution kernels of a pretrained SR network and fine-tunes only lightweight rank-r matrices attached to selected convolution layers, using VVC decoded frames and quantization-parameter (QP) maps of test sequences as supervision. The resulting low-rank update is compressed with the MPEG Neural Network Compression and Representation (NNR) standard. Experiments on the JVET common test conditions (CTC) class A1 and A2 sequences indicate that LoRA-based content adaptation provides bitrate savings over a non-adapted SR post-filter at a small signalling cost. Compared with the VVC Test Model (VTM21), the proposed method achieves BD-rate savings of -10.93% (Y), -15.39% (U), -24.41% (V) under random access and -13.43% (Y), -5.75% (U), -22.94% (V) under all-intra. An ablation of the LoRA rank r further shows that r=4 provides the best trade-off between coding gain and signalling cost.

0.19