CS Paper Radar

Multi-Scale Latent Temporal Compression Attention and Video-Text Cross Attention Fusion for Video Action Detection

Yicheng Qiu, Keiji Yanai

cs.CV cs.MM ·submitted 2026-10-21 04:08 UTC · v2
published

Abstract

In video action detection, modeling the temporal dependencies in continuous video streams is crucial. Previous methods based on deep neural networks can perform frame-level video recognition tasks. However, these methods struggle to model dynamic temporal patterns, facing challenges such as redundant frame information, difficulty balancing global and local features, and insufficient fusion of visual and textual features. To address these issues, we propose a video temporal attention module based on the Multi-Scale Latent Temporal Compression Attention, which aggregates continuous temporal sequences into feature representations across diverse temporal scales while efficiently reducing redundancy. We also introduce a Video-Text Cross Attention Fusion module that leverages cross-attention at both latent and standard feature levels, achieving fine-grained fusion between video and textual features to enhance semantic alignment and feature interaction. Experimental results across multiple video recognition benchmarks demonstrate that our proposed methods achieve superior performance compared to existing methods, validating the effectiveness and generalization capability.

Comment
This paper has been accepted for publication in the proceedings of the 33rd International Conference on Multimedia Modeling (MMM 2027)

Closest papers

Nearest neighbours by meaning across the whole index.

TF-PRVR: Training-Free Partially Relevant Video Retrieval

cs.CV ·2026-10-06 ·Giyeol Kim, Chanho Eom
similarity 0.74
Abstract

Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos containing moments relevant to a given text query. Despite recent progress, existing PRVR methods suffer from two key limitations: a fixed video decomposition scheme that causes semantic dilution, and source-domain overfitting induced by task-specific training. In this paper, we propose TF-PRVR, the first training-free framework for PRVR. TF-PRVR leverages frozen vision-language features to construct video-specific hierarchical representations. It derives temporal semantic signals from frame-level features and applies frequency-based multi-scale analysis to identify adaptive temporal boundaries, producing hierarchical segments with coherent event-level semantics. Built on these segments, TF-PRVR constructs a unified multi-scale graph and propagates query relevance across temporally and semantically related nodes. A moment-aware scoring strategy then aggregates temporally aligned relevance across scales, emphasizing consistently supported moments while suppressing isolated false responses. Without task-specific training, TF-PRVR preserves the general-purpose alignment capability of pre-trained vision-language models and avoids dataset-specific overfitting. Extensive experiments demonstrate consistent performance across datasets with diverse visual and temporal characteristics, suggesting a practical direction for training-free PRVR.

0.13

Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction

cs.CV ·2026-10-26 ·Jingxue Huang, Xiaosong Li, Yi Lin
benchmark similarity 0.73
Abstract

Multi-modal video fusion aims to generate temporally coherent videos with salient targets and rich structural details by integrating complementary information from heterogeneous sensors. However, existing video fusion methods commonly rely on optical flow estimation and feature warping for inter-frame alignment and cross-frame information aggregation to maintain temporal consistency, thereby introducing considerable computational overhead and making the fusion performance vulnerable to flow estimation errors. To address these limitations, this paper proposes an efficient multi-modal video fusion network with dynamic region-guided temporal interaction. Specifically, the proposed spatio-temporal interaction module leverages the Test-Time Training paradigm to model temporal dependencies with linear complexity. Building on this design, the module restricts temporal modeling to key dynamic regions in the video sequence, as static background regions provide limited temporal cues. By excluding low-variation tokens from unnecessary temporal interaction, this selective strategy contributes to improved fusion quality and temporal consistency. In addition, leveraging the strong ability of pretrained MAE features to capture global structural dependencies, we introduce a feature-level temporal consistency constraint to suppress local flickering by aligning the inter-frame variation directions of the fused and source videos. We further propose a new MAE-based temporal metric that quantifies directional distance between their temporal variations within dynamic regions. Experimental results demonstrate that, compared with existing state-of-the-art methods, our method achieves superior fusion quality and temporal consistency on multiple benchmark datasets, while exhibiting high computational efficiency under edge-device deployment settings.

0.13

TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrieval

cs.CV ·2026-09-02 ·Uicheol Jung, Juyoung Hong, Hojung Kwon, Yukyung Choi
publishedcode/data similarity 0.72
Abstract · 17 pages, 6 figures

Text-Video Retrieval (TVR) retrieves videos that match a natural-language query, but extending image-text models such as CLIP to videos is fundamentally limited by the lack of temporal modeling. Videos exhibit frame-wise heterogeneity in appearance and motion, and compressing all frames into a single representation often obscures temporal structure and semantic transitions. To address this, we propose Temporal-Aware Mixture-of-Experts for Text-Video Retrieval (TAME), a CLIP-based framework that jointly models frame-level structure and temporal relations. First, we integrate sparse Mixture-of-Experts (MoE) layers into both CLIP encoders and apply frame-consistent routing on the vision branch so that experts specialize according to frame-level visual patterns while preserving the original vision-language alignment. Second, we introduce Frame-Temporal (FT) tokens that aggregate global cross-frame information and feed it back to each frame, enabling the visual encoder to capture long-range temporal dependencies without harming local details. Third, we design a Cross-Temporal Interaction and Aggregation (CTIA) module that refines frame-wise sentence-video similarities through staged temporal filtering and fusion. Experiments on standard TVR benchmarks show that TAME consistently improves over CLIP-based baselines. On MSR-VTT, it improves R@1 by 4.0 over CLIP4Clip, and also achieves consistent gains on DiDeMo, MSVD, LSMDC, and ActivityNet. The code is available at https://github.com/sejong-rcv/TAME.

0.62

Intrinsic Temporal Adaptation of CLIP for Partially Relevant Video Retrieval

cs.CV ·2026-09-04 ·Hyun Seok Seong, Woojin Jun, SuBeen Lee, Jae-Pil Heo
top venuecode/data similarity 0.71
Abstract · EMNLP 2026 paper

Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos that contain moments relevant to a text query. Since the target moment occupies only a portion of the video, PRVR requires retrieval based on fine-grained understanding beyond coarse video-level matching. However, existing methods often rely on frozen CLIP frame features, which lack temporal understanding. Even with recent progress in parameter-efficient CLIP adaptation, video-level predictions can still be supported by imprecise frame-level evidence. In this paper, we propose an Intrinsic Temporal Adaptation (ITA) framework for PRVR. First, our Backbone-Internal Temporal Adaptation allows the last few visual transformer layers to attend over groups of neighboring frames. This provides temporally aware frame embeddings while keeping CLIP frozen and training only adaptation parameters. Second, we introduce Affinity-Weighted Gradient Propagation to address the weakly supervised nature of PRVR, softly aggregating top-$k$ frames based on text-frame affinities and propagating learning signals to multiple query-relevant frames. Our method achieves state-of-the-art performance on PRVR benchmarks, demonstrates robust cross-dataset transfer, and retrieves substantially more accurate frame-level evidence within ground-truth query-relevant moments. Our code is available at github.com/hynnsk/ITA.

0.42

Text-Video Retrieval via Multi-Dimensional Saliency Assessment and Granularity-Aware Query Decomposition

cs.IR ·2026-09-30 ·Shuquan Wei, Xi Chen, Xu Chen, Xiangyang Jia
similarity 0.71
Abstract

Text-video retrieval, which aims to bridge visual and textual modalities by learning a joint embedding space, has become a crucial task in multimodal intelligence. Despite extensive efforts to mitigate visual redundancy, previous methods typically rely on a single-aspect criterion to assess visual importance, overlooking the multifaceted spatiotemporal nature of video. In addition, encoding text into a single global embedding to align with videos compresses temporal events and spatial entities into a unified representation space, further aggravating cross-modal misalignment. To address these issues, we propose MMTI, a method that jointly mitigates visual redundancy and enables multi-grained text-video interaction to achieve accurate multi-grained semantic alignment. Specifically, a key feature selection (KFS) mechanism adaptively identifies and aggregates informative frames and patches by jointly evaluating multi-dimensional saliency and learnable importance scores, effectively compacting dense visual features and mitigating visual redundancy. Furthermore, our proposed multi-grained text-video interaction module (TVIM) employs a dynamic gating mechanism to decompose the text query into sentence, frame, and patch queries (SFP), enabling multi-grained text-video alignment. Complementary alignment at different granularities is thereby achieved. Extensive experiments on four standard benchmarks demonstrate that our method outperforms state-of-the-art methods.

0.20

STAR-VF: Spatio-Temporal Aware Representation Condensation for Data-Efficient Video Fusion

cs.CV ·2026-11-07 ·Yuchen Xian, Yunqiu Xu, Guosheng Lin, Yang He
top venuecode/data similarity 0.71
Abstract · Accepted at ACM Multimedia 2026 (MM '26)

Video fusion aims to integrate complementary cues from heterogeneous sources, producing visually informative videos with coherent spatio-temporal structures. Jointly learning cross-modal complementarity and temporal dependencies typically relies on large-scale, high-quality video datasets, whose storage and repeated use during optimization incur substantial costs, making data-efficient learning highly desirable. To address this challenge, we rethink video fusion from the perspective of dataset condensation and propose STAR-VF, the first framework that condenses spatio-temporal representations for data-efficient video fusion. STAR-VF condenses the original training set into a compact set of learnable spatio-temporal representations, which are decoded into synthetic video clips and optimized using feature-level supervision provided by a frozen proxy model. The resulting condensed data preserves fusion-relevant spatial and temporal cues without requiring ground-truth fused videos as supervision. To adapt fusion learning to condensed representations, we equip the final fusion model with flow-free inter-frame temporal modeling and query-driven intra-frame aggregation, enabling direct temporal interaction and adaptive selection of fusion-relevant information in representation space. Extensive experiments on infrared-visible, multi-exposure, multi-focus, and medical video fusion benchmarks demonstrate that STAR-VF, trained with only approximately 1/16 of the original temporal training set, achieves fusion quality comparable to methods trained on the full data. Further analyses validate the contributions of learned condensation and flow-free temporal modeling, and demonstrate the robustness of the condensed supervision across different proxy encoders and downstream fusion models. Project page: https://zju-xyc.github.io/STAR-VF-Project-Page/.

0.38

MATA: Motion-aligned Temporal Adapter

cs.CV ·2026-11-04 ·Ashish Kumar Singh, Prakash Chandra Chhipa
publishedcode/data similarity 0.71
Abstract · Accepted at WACV 2027. Joint first authors with equal contributions. Code - https://github.com/prakashchhipa/MATA-Motion-aligned-Temporal-Ad…

Few-shot fine-grained video action recognition requires recognizing novel action classes from only a few labeled examples. The difficulty is that visually similar actions often share actors, scenes, poses, and objects, differing only in the order, timing, and short duration of motion changes. We study this problem under three practical constraints: few tunable parameters, low compute cost, and use of the video modality alone. We propose Motion-aligned Temporal Adapter (MATA), a parameter-efficient adapter on top of a frozen CLIP backbone. MATA specializes CLIP features along the temporal dimension using two components: temporal phase alignment through learnable temporal positional embeddings, and a local motion transition operator through depth-wise convolution. With only ~3.6M tunable parameters, MATA achieves state-of-the-art performance across multiple fine-grained benchmarks while remaining robust to both visual corruptions and temporal disruptions induced by frame dropping. Code is available at https://github.com/prakashchhipa/MATA-Motion-aligned-Temporal-Adapter

0.38

MELON: A Large-Scale Dataset for Multi-Event Text-to-Long-Video Retrieval

cs.IR ·2026-08-31 ·Chan Hur, SeungWoo Song, Jeong-hun Hong, Won Jun Oh, Hyeyoung Park +1 more
dataset similarity 0.71
Abstract

Existing text-video retrieval datasets primarily consist of short-form clips containing a single dominant event. While suitable for measuring basic vision-language alignment, they are limited in capturing real-world retrieval scenarios, where long-form videos naturally contain multiple semantically distinct events and a single text query may correspond to several non-contiguous temporal segments. To bridge this gap, we introduce MELON, the first large-scale dataset designed to extend text-video retrieval to long-form videos featuring complex, multi-event structures. MELON explicitly annotates multiple event intervals per video along with their corresponding textual descriptions, enabling both training and evaluation of multi-event understanding in long, untrimmed videos. In addition, we propose a multi-event aware loss that encourages models to differentiate between full-event and partial-event matches, yielding substantial improvements in retrieval accuracy. Together, the MELON dataset and our proposed loss establish a robust foundation for expanding text-to-video retrieval to complex long-form scenarios and provide a more realistic evaluation setting for future research in the field.

0.16