CS Paper Radar

MEDR: Query-Independent Frame Selection via Multi-Signal Event Modeling and Dynamic Rescoring

Xinlei Pu, Weijie Shi, Wen Yang, Yi Cao, Yao Zhao, Hao Chen, Yuanjun Liu, Wenwei Ding, Jia Zhu, Jiajie Xu

cs.CV ·submitted 2026-08-15 06:10 UTC · v2

Abstract

Frame selection is a fundamental component of multimodal large language models, enabling long videos to be processed under limited visual-token and computational budgets. Uniform sampling preserves temporal coverage but may miss informative content that appears only briefly. To alleviate this limitation, query-dependent methods can retrieve question-relevant frames. However, because the selected frames depend on the current question, the same visual input cannot be directly shared across different questions, and frame selection must be repeated in multi-turn video dialogue. This motivates us to seek a query-independent frame selection method that preserves the reusability of a fixed visual input while improving the coverage of informative events beyond uniform sampling. We propose Multi-Signal Event Modeling and Dynamic Rescoring (MEDR), a training-free and query-independent frame selection method. Multi-Signal Event Modeling organizes complementary visual, motion, and text signals into signal-specific temporal events. Dynamic Rescoring then iteratively reevaluates each candidate relative to the current selected set, updating its score according to frame-level signal strength, additional event coverage, and balanced temporal coverage. The resulting fixed frame set is constructed without observing the query and can be reused across different questions. Experiments on Video-MME and LongVideoBench demonstrate that MEDR achieves competitive performance across two video MLLMs. Under the 64-frame budget, MEDR improves Video-MME accuracy over the strongest evaluated query-independent baseline by 1.33-1.78 percentage points. We further evaluate the reuse of a fixed frame set in multi-question video dialogue.

Comment
11 pages, 3 figures, 4 tables; revised manuscript and updated author list

Closest papers

Nearest neighbours by meaning across the whole index.

Query-aligned video frame selection for long video understanding

cs.CV cs.AI ·2026-09-15 ·Md. Safayet Islam, Dilip Sarkar, Liang Liang
similarity 0.74
Abstract · 16 pages, 5 figures, 22 references

Multimodal large language models (MLLMs) process multimodal inputs by converting text, images, and videos into token sequences that are subsequently processed by a backbone language model. While MLLMs have achieved excellent performance in understanding the content of individual images, video understanding remains significantly more difficult because videos contain large number of video frames. MLLMs typically process only a subset of these frames, usually ranging from 8 to 64. MLLMs usually sample frames uniformly, regardless of their relevance to the question being answered. To address this limitation, several training-free, model-agnostic methods for selecting question-relevant frames have recently been proposed. In this work, we introduce a frame-selection method designed specifically for multiple-choice questions. We extend the query text by appending semantic cues derived from the answer choices and employ a direct query-frame alignment scoring mechanism. To the best of our knowledge, our method is the first to directly utilize answer choices as inference-time cues for selecting frames relevant to answering a question. The method first constructs a compact candidate pool by subsampling video frames at a fixed rate. The frames are then scored according to their maximum cosine similarity across all question-answer pairs to identify the most relevant frames for a given query. This approach preserves a fixed token budget while improving the relevance of the visual evidence provided to the downstream MLLM. We evaluate the effectiveness of our frame-selection method on the MLVU, Video-MME, and LongVideoBench benchmarks using three MLLMs: LLaVA-Mini, Qwen2-VL, and LLaVA-Video. Experimental results demonstrate that answer-aware frame selection generally outperforms uniform sampling and existing training-free frame-selection methods under the same frame budget.

0.36

VTG-RAG: Long-Video Reasoning via Query-Adaptive Coarse-to-Fine Sampling and Temporal Graph Retrieval

cs.CV ·2026-11-11 ·Yao Lu, Yunhao Wang, Yaxin Gao, Feifei Teng, Chenxiao Liu +5 more
similarity 0.74
Abstract

Long-video reasoning requires a system to localize sparse evidence over extended temporal sequences, preserve temporal structure, and generate answers despite substantial multimodal distractors. Existing video retrieval-augmented generation methods typically build indexes from uniformly sampled video segments. In long-video settings, this design readily introduces irrelevant context and is unreliable for questions with explicit temporal constraints. We propose VTG-RAG, a framework for long-video question answering that combines query-adaptive coarse-to-fine sampling with temporal graph retrieval. First, a two-stage sampling strategy selects core segments with high mean query relevance or high intra-segment relevance variance. The framework then integrates subtitles, speech transcripts, visual descriptions, entities, and relations from these segments into a temporally ordered multimodal graph. Finally, it performs temporally filtered, two-pass retrieval and reasoning across frame, text, and graph evidence spaces. A lot of experiments have proven the effectiveness of the VTG-RAG. Specifically, we evaluate the framework on Video-MME and LongVideoBench. On LongVideoBench, the reported Qwen2.5-VL-32B and 72B results improve from 48.16% to 50.56% and from 52.13% to 53.32%, respectively, with 24 answer-input frames. These correspond to gains of 2.40 and 1.19 percentage points. The ablation experiments of VTG-RAG further support that the contributions of each component are very important.

0.05

FORTE: Adaptive Scoring and Exact Keyframe Selection for Long-Video Question Answering

cs.CV ·2026-09-30 ·Haifeng Huang, Biyin Xu, Chunsheng Xin, Yang Li
similarity 0.73
Abstract

Query-aware keyframe selection enables multimodal large language models (MLLMs) to process long videos using only a small set of question-relevant frames. Existing score-based methods, however, typically search within a fixed, uniformly sampled candidate pool, preventing evidence outside this pool from ever being selected. Given a limited relevance-scoring budget, the key challenge is to allocate evaluations adaptively to promising frames while continuing to explore underrepresented temporal regions. We introduce FORTE, a training-free framework that addresses this challenge through two stages: adaptive relevance scoring and global keyframe optimization. Starting from sparse, uniformly distributed observations, our efficient Gaussian-process relevance predictor estimates relevance for unscored frames, exploiting temporal locality and the approximately banded kernel structure to reduce the core computation from cubic to linear time in the number of frames for fixed bandwidth. The scoring stage then selects which frames to score next by balancing predicted relevance with temporal coverage, prioritizing promising regions while also exploring less-represented parts of the video. The optimization stage selects the final keyframes by maximizing an objective that jointly captures measured relevance and temporal coverage. We derive an exact algorithm that leverages the logarithmic coverage structure to identify the optimal subset of the scored candidate pool in time linear in the pool size, for a fixed final-frame budget. Experiments on four long-video question-answering benchmarks show that FORTE achieves the highest observed mean accuracy among the compared selectors under every tested scoring budget. Further evaluations demonstrate its consistent effectiveness across different relevance scorers and downstream MLLMs.

0.15

EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding

cs.CV ·2026-10-06 ·Yuhao Qin, Junbo Wang, Yuke Li, Yining Zhu
similarity 0.72
Abstract · 12 pages, 7 figures, 7 tables, including supplementary material

Current large video-language models (LVLMs) still face challenges when dealing with long videos, mainly because frames are often processed independently, making it difficult to capture temporal dependencies across events. Although retrieval-augmented approaches have been introduced to provide additional context, most of them operate at the frame or snippet level, which limits their ability to model how events evolve over time and relate to each other. In this paper, we propose Event Chain Retrieval-Augmented Generation (EC-RAG), a training-free framework that organizes video content into an explicit event chain before question answering. Instead of retrieving isolated frames or text segments, EC-RAG first partitions the video into semantically coherent segments, represents each segment using multi-modal signals, and then links them into a structured chain that preserves temporal order and captures inter-event relationships. Given a query, the system identifies relevant events within this chain and gathers supporting evidence from the associated modalities. Our approach offers several practical advantages: (i) event-level abstraction that better reflects how video content is naturally structured, enabling more reliable localization compared to frame-level retrieval; (ii) structured multi-modal fusion that aggregates speech, text, and visual cues at the event level, allowing complementary information to be more effectively utilized during reasoning; and (iii) plug-and-play compatibility with existing LVLM backbones, requiring no additional training or reliance on proprietary models. Experiments on Video-MME, MLVU, and LongVideoBench show that this event-centric design consistently outperforms frame-level retrieval baselines, highlighting the importance of modeling temporal structure for long-video understanding.

0.16

VideoJev: System One Model for Video Understanding and Interaction

cs.CV ·2026-10-13 ·Yuhan Zhu, Xiangyu Zeng, Qingyi Si, Chenxu Yang, Yifei Li +13 more
code/datahighly citedcommunity pick similarity 0.71
Abstract · 23 pages. Project page: https://mcg-nju.github.io/VideoJev

Visual agents must turn video observations into timely decisions to track progress and choose what to do next. Building on the emerging System One paradigm, we introduce VideoJev, a family of models that cast video understanding and interaction as structured decision tasks. We adapt pretrained multimodal large language models (MLLMs) by replacing autoregressive answer decoding with a shared head that scores temporal intervals and textual options against video evidence. These scores yield event spans or answer choices, unifying temporal grounding and multiple-choice video question answering. This combines the semantic flexibility of MLLMs with efficient decisions that agents can use directly. As an interaction unfolds, new goals and intermediate results prompt further queries about the same visual history. VideoJev therefore reuses visual computation through Cached inference for queries as they arrive and Joint inference for multiple queries in a single forward pass. Across three temporal grounding benchmarks, VideoJev-4B and VideoJev-35B-A3B improve mean mIoU over their backbones by 6.7 and 10.8 points, respectively, with measured inference speedups of $2.8$-$3.1\times$. On four video QA benchmarks, both achieve performance comparable to their backbones in Thinking mode, with measured inference speedups exceeding $100\times$. These results offer a unified, reusable decision interface for composing structured visual judgments into complex, open-ended workflows.

0.61

MetaSampling: Making Frame Samplers Efficient for Long-Video Question Answering

cs.CV ·2026-09-27 ·Ashim Dahal, Bikramjit Banerjee
benchmark similarity 0.71
Abstract

Frame selection is an important component of long-video question answering (VQA) with Multimodal Large Language Models (MLLMs). Existing frame-selection methods improve over simple top-$k$ embedding retrieval and uniform sampling, but are typically applied under a fixed global selection budget. We introduce \textbf{MetaSampling}, a training-free, plug-and-play sampling strategy that can be applied on top of existing frame selectors. MetaSampling improves downstream VQA efficiency by dynamically reducing the number of frames passed to the MLLM while preserving, and in some cases improving, answer accuracy. We evaluate MetaSampling across 36 paired frame-selector--MLLM-backbone--VQA-benchmark configurations. MetaSampling reduces the number of selected frames in all 36 configurations and improves accuracy in 25 of them, yielding an average frame reduction of $8.9\%$ while slightly improving accuracy overall.

0.11

APS: Adaptive Partitioning-Based Frame Selection for Long Video Understanding

cs.CV cs.AI ·2026-11-18 ·Junxiang Wen, Yinxi Zhao, Jiawei Chen, Jingying Hu, Lijun Wang +2 more
similarity 0.71
Abstract

Recent advances in large vision-language models (LVLMs) have significantly improved video understanding capabilities, yet processing long videos remains challenging due to the substantial computational cost of extensive visual inputs. Frame selection offers an efficient alternative, but many existing region-based methods rely on fixed or predefined temporal partitions, which may result in suboptimal temporal regions, compromising subsequent budget allocation and frame selection. In this work, we propose Adaptive Partitioning-Based Frame Selection (APS), a training-free framework that adapts temporal partitions to local variations in query-frame relevance. Specifically, APS formulates temporal partitioning as a complexity-regularized fitting problem, jointly optimizing partition boundaries and local models through dynamic programming. The resulting adaptive temporal partitions guide frame budget allocation based on query relevance and temporal extent, enabling informative frame selection under limited visual budgets. Extensive experiments on Video-MME, MLVU, and LongVideoBench demonstrate that APS consistently improves performance across diverse LVLM backbones, achieving the highest average gains over uniform sampling among competing methods. Further analyses validate the effectiveness of adaptive partitioning and demonstrate its generalizability across model scales, similarity encoders, and advanced LVLMs.

0.17

Reason First, Retrieve What Matters: Hypothesis-Guided Keyframe Selection for Long Video Understanding

cs.CV cs.AI ·2026-11-07 ·Yisong Hu, Junbo Wang, Yuke Li, Jiangbin Zheng
similarity 0.71
Abstract · 14 pages, 4 figures, 3 tables

Long video understanding requires models to identify and reason over sparse, temporally distributed evidence within limited frame budgets. Existing keyframe selection methods typically retrieve query-relevant frames from a densely sampled candidate set before inference. However, global image-text matching incurs unnecessary computation when a small set of uniformly sampled frames already provides sufficient evidence. Query relevance alone may also overlook evidence that distinguishes competing answers, while one-shot selection offers no opportunity to correct earlier selection errors. We propose HypoKFS, a training-free, plug-and-play framework that couples keyframe selection with iterative inference by multimodal large language models (MLLMs). HypoKFS first reasons over sparsely sampled frames to construct an initial hypothesis space. The resulting answer distribution determines whether additional frame selection is needed. When needed, the retained hypotheses guide the selection of discriminative frames, followed by renewed inference and hypothesis refinement. This iterative process directs additional visual computation toward unresolved hypotheses while avoiding unnecessary global frame matching. Experiments on Video-MME, LongVideoBench, and MLVU demonstrate consistent improvements across multiple MLLM backbones. HypoKFS improves accuracy by up to 6.7 percentage points and achieves 72.9% accuracy on Video-MME at only 1.3% of the reference frame-hypothesis matching cost.

0.12