CS Paper Radar

FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit

Jiaxuan Sun, Biao Wang, Shuai Li

cs.CV cs.AI ·submitted 2026-10-19 08:45 UTC · v1

Abstract

Reinforcement learning with verifiable rewards, particularly Group Relative Policy Optimization (GRPO), has improved video reasoning in multimodal large language models. However, standard video GRPO varies only linguistic trajectories while conditioning every rollout in a group on the same sampled frames, which can produce homogeneous rewards and weaken group-relative learning when the shared visual evidence is insufficient or distracting. We propose \textbf{FrameGRPO}, a simple and effective reinforcement-learning framework that requires neither auxiliary networks nor changes to the model architecture. Specifically, FrameGRPO combines two lightweight components: (1) \textbf{Multi-Frame Rollouts}, which use only uniform temporal subsampling of a single maximal-frame decode to construct aligned views that diversify visual evidence and reasoning trajectories; and (2) \textbf{Gated Temporal Efficiency Credit}, which selectively reinforces a correct lower-ratio trajectory when all rollouts in a fixed full-ratio reference subset fail, without imposing an unconditional preference for fewer frames. Extensive experiments across diverse video understanding and reasoning benchmarks demonstrate robust overall improvements over standard GRPO, highlighting temporal-view diversification as an effective mechanism for exploration and credit assignment in video MLLMs.

Closest papers

Nearest neighbours by meaning across the whole index.

Frame Differential On-Policy Self-Distillation for Video Reasoning

cs.CV ·2026-09-30 ·Haiying He, Xin Zheng, Shaoli Hu, Shijun Xiao, Xuanhe Liu +2 more
hot direction similarity 0.79
Abstract

Reinforcement learning (RL) has substantially improved the reasoning ability of multimodal language models through verifiable rewards and increasingly fine-grainedvisual or temporal credit assignment. In video reasoning, however, current RL methods typically train with a fixed sparse frame budget: increasing the number of frames makes autoregressive rollouts expensive, while too few frames may miss temporally localized events and fine-grained visual details. We present \textbf{Frame Differential On-Policy Self-Distillation (FD-OPSD)}, which transfers the useful evidence of dense frame observations to a sparse frame policy during RL training. FD-OPSD compares the policy's token level preferences for the same sampled response under sparse and dense views, and distills the resulting frame differential signal without an external teacher or dense autoregressive rollout. The method preserves sparse-frame rollouts and leaves inference unchanged. Across Qwen2.5-VL-7B and Qwen3-VL-4B on six video reasoning benchmarks, FD-OPSD yields higher overall average performance than the strongest corresponding GRPO, T-GRPO, or Video-KTR baselines across the 16, 32, and 64 frame evaluation settings. These results show that dense visual evidence can be transferred selectively during training through token level self-distillation while retaining sparse frame rollouts and unchanged inference.

0.23

Less from More: Reinforcing Sparse Video Reasoning from Dense References

cs.CV ·2026-10-07 ·Wenfang Sun, Yingjun Du, Cees G. M. Snoek
similarity 0.77
Abstract

Video-language models commonly assume that more temporal observations lead to more reliable reasoning. We question this assumption and argue that the key challenge is not merely processing more video frames efficiently, but learning to reason reliably under limited temporal evidence. We propose SAVER, a dense-to-sparse post-training framework that uses dense video views as training-time references for sparse-frame inference. During reinforcement post-training, paired dense and sparse views are optimized with grounding rewards and a reliability-gated reference reward, encouraging sparse view predictions to preserve task-relevant temporal evidence. Notably, SAVER is trained only on 1,250 randomly sampled temporal grounding examples, without using any video question answering annotations. Across three temporal grounding benchmarks and six video question-answering benchmarks, SAVER consistently improves performance across frame budgets. In particular, SAVER can match or surpass dense-frame Qwen3.5 baselines while using substantially fewer frames. These results show that temporal grounding can serve as an effective evidence-localization proxy for learning sparse video reasoning that transfers to broader video understanding tasks.

0.34

OWLET: On-Policy Weighted Learning via Evidence Transfer for Video Large Language Model Post-Training

cs.CV ·2026-10-09 ·Qitan Lv, Ke Fan, Hong Wang, Wen Wu, Yilin Wang +3 more
similarity 0.76
Abstract

Post-training has become a key stage for eliciting reasoning capabilities in Video Large Language Models (Video-LLMs). Existing post-training methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, they do not fully exploit visual evidence for effective training. RL supervises the policy with a single trajectory-level reward, which is too coarse-grained for video reasoning: the same response-level advantage is applied to every generated token, disregarding that the tokens of a video response are grounded in the visual stream to very different degrees. OPD provides token-level supervision through output-distribution matching, but lacks direct supervision of the spatiotemporal representations that produce those outputs. We observe across extensive experiments that the same model reasons more accurately when its video view combines global temporal context with question-relevant details. Comparing the same student-generated response under this view and uniform sampling reveals changes in token likelihoods and internal representations, providing a training signal for evidence use. Building on this observation, we propose OWLET, an evidence-privileged RL method that derives training signals from a richer teacher view. This view couples uniform global context with question-conditioned local windows, providing visual evidence without requiring a larger model. To better exploit this privileged visual evidence, OWLET introduces (i) \textbf{evidence-aware credit redistribution}, which compares token likelihoods across the two views and redistributes positive response-level advantages according to their relative evidence support; and (ii) \textbf{spatiotemporal state alignment}, which aligns the two views' hidden states on verifier-accepted responses.

0.15

From Label Priors to Task Evidence: Efficient Long-Video Frame Selection via Bayesian GRPO

cs.CV ·2026-10-18 ·Shuochen Chang, Bingjie Gao, Qingyang Liu, Qianli Ma, Yibo Miao +5 more
code/data similarity 0.76
Abstract

Multimodal Large Language Models (MLLMs) face significant scalability challenges in long-video understanding due to quadratic attention complexity and context window constraints. Existing retrieval-based approaches typically rely on supervised learning from static keyframe annotations. However, these label priors often fail to align with the actual information required by downstream MLLMs for complex reasoning, creating a misalignment between the training target and inference needs. To bridge this gap, we propose a probabilistic framework that reformulates frame selection as latent-variable inference. We first initialize a lightweight policy using supervised learning to capture general visual saliency. Then, we treat a frozen MLLM as a task environment and use its answer likelihood as dense task evidence to refine the policy via Bayesian Group Relative Policy Optimization (GRPO). Within this formulation, the negative log-likelihood (NLL) over ground-truth answer tokens provides a principled dense reward, while a prior-matching KL term preserves semantic consistency with the elicited label prior. This process transitions the selection principle from static label priors to task evidence. Extensive experiments on Video-MME, MLVU, and LongVideoBench demonstrate that our selector achieves competitive accuracy with MLLM-based scorers while retaining the inference efficiency of lightweight methods. The selector is trained once with a frozen MLLM environment and transfers directly to multiple downstream MLLMs without retraining. Code is available at github.com/Miraclecsc/Frame_Selection_GRPO

0.29

What Do Verifiable Rewards Teach Video-Language Models About Time? A Controlled Multi-Model Study

cs.CV ·2026-09-30 ·Avyay Sadhu, Patrick Cooper
publishedbenchmark similarity 0.76
Abstract · 8 pages, 2 figures. Accepted to WACV 2027

Reinforcement learning from verifiable rewards (RLVR) has produced large reasoning gains in language models, and verifiable video benchmarks make the recipe directly applicable to causal-temporal video question answering. We present a controlled study of what RLVR actually teaches video-language models about temporal structure. We fine-tune four open models with group relative policy optimization (GRPO) under three data recipes using annotated synthetic questions, self-supervised real-video pretext tasks, and a one-to-one mixture, plus a targeted real-video augmentation arm. We evaluate the model-recipe grid on 105 CLEVRER questions with one correct option and on two real-video transfer sets. The 8B ordered gain is 4.8 points (95% video-bootstrap CI [-5.6, 15.1]), while blind accuracy increases 10.5 points ([0.9, 20.6]). The 4B model gains an accuracy advantage for ordered over shuffled frames. On the real-video sets, the 8B synthetic-only checkpoint loses 26.7 and 25.2 points, with smaller regressions under mixed and augmented recipes. In a separate scene-disjoint, single-gold Qwen3-VL-8B experiment, supervised fine-tuning (SFT) and GRPO each complete 1,200 updates at three fixed seeds. GRPO minus SFT is -3.89 points in ordered accuracy (nominal 97.5% video-bootstrap CI [-7.36, -0.45]) and -2.78 points in ordered-minus-shuffled accuracy ([-6.60, 0.87]). Transfer differences are sensitive to parseability. Intervals quantify test-video uncertainty conditional on the fitted runs. The results show why benchmark accuracy, transfer, and presented-order sensitivity need to be evaluated separately.

0.40

Grounding Multimodal Reasoning with Evidence-Ablated Negatives

cs.CL ·2026-10-29 ·Chenxi Liu, Tianyi Xiong, Yanshuo Chen, Heng Huang
publishedcode/data similarity 0.75
Abstract · NeurIPS2026

Reinforcement learning with verifiable rewards (RLVR) has become a powerful tool for improving multimodal reasoning. However, standard outcome supervision suffers from credit assignment ambiguity, as a model may receive positive rewards by exploiting language priors or dataset shortcuts rather than grounding its reasoning in the requisite visual evidence. This makes high-quality negative responses critical, since effective negatives should remain plausible under the current policy while exposing the shortcut that should be suppressed. In this paper, we propose EVAN, a multimodal reinforcement learning framework utilizing EVidence-Ablated Negatives to train LMMs against their own language-prior failures. EVAN first identifies highly visually dependent examples using a scaled image information gain metric and preserves data diversity with a mixed-ranking strategy. It then constructs evidence-ablated images by masking the most discriminative image patches, eliciting plausible but visually ungrounded responses from the current policy. During policy optimization, these negative responses are incorporated as bounded auxiliary signals under the clean image, penalizing shortcut learning while maintaining RL stability. Experiments across challenging multimodal reasoning benchmarks demonstrate that EVAN mitigates modality imbalance and improves over strong RLVR baselines. Our code is available at https://github.com/DawnLIU35/EVAN.git.

0.26

RECAP: Looking Once Is Not Enough for Vision-Language Reasoning

cs.CV ·2026-10-22 ·Zhaolu Kang, Tailong Luo, Chenxin Li, Yingjie He, Zhenyu Yu +11 more
published similarity 0.75
Abstract · NeurIPS 2026

Long chain-of-thought reasoning improves vision-language models (VLMs), but it also exposes a temporal grounding failure: as generation unfolds, models may drift from image evidence while reinforcement learning still provides only a final-answer reward. We study this gap through visual revisit, a temporally localized reactivation of image evidence during reasoning. Across three VLM backbones, revisit peaks are predictive of correctness and causally linked to performance: masking them degrades accuracy more than masking non-peaks, while injecting revisit-like peaks into failed rollouts partially restores correct answers. Control analyses show that this signal is not explained by raw attention strength, uncertainty, logit margin, or hidden-state norm. We propose RECAP, a GRPO-compatible training method that turns detached revisit traces into a credit-assignment signal. RECAP uses revisit-conditioned advantage estimation, dynamic discounting, and gated value prediction to propagate final rewards through visually important reasoning steps, without extra rewards, additional rollouts, or inference-time changes. Across Qwen2.5-VL-7B, InternVL3-8B, and LLaVA-OV-7B, RECAP consistently improves over GRPO and VPPO, yielding $+6.1$--$+6.5$ HallusionBench gains over the SFT base while preserving reasoning and text-only performance. Results on larger LoRA-tuned models, Chinese zero-shot benchmarks, visual perturbations, and human evaluation further demonstrate robust improvements in image-grounded reasoning.

0.17

VideoJev: System One Model for Video Understanding and Interaction

cs.CV ·2026-10-13 ·Yuhan Zhu, Xiangyu Zeng, Qingyi Si, Chenxu Yang, Yifei Li +13 more
code/datacommunity pick similarity 0.74
Abstract · 23 pages. Project page: https://mcg-nju.github.io/VideoJev

Visual agents must turn video observations into timely decisions to track progress and choose what to do next. Building on the emerging System One paradigm, we introduce VideoJev, a family of models that cast video understanding and interaction as structured decision tasks. We adapt pretrained multimodal large language models (MLLMs) by replacing autoregressive answer decoding with a shared head that scores temporal intervals and textual options against video evidence. These scores yield event spans or answer choices, unifying temporal grounding and multiple-choice video question answering. This combines the semantic flexibility of MLLMs with efficient decisions that agents can use directly. As an interaction unfolds, new goals and intermediate results prompt further queries about the same visual history. VideoJev therefore reuses visual computation through Cached inference for queries as they arrive and Joint inference for multiple queries in a single forward pass. Across three temporal grounding benchmarks, VideoJev-4B and VideoJev-35B-A3B improve mean mIoU over their backbones by 6.7 and 10.8 points, respectively, with measured inference speedups of $2.8$-$3.1\times$. On four video QA benchmarks, both achieve performance comparable to their backbones in Thinking mode, with measured inference speedups exceeding $100\times$. These results offer a unified, reusable decision interface for composing structured visual judgments into complex, open-ended workflows.

0.60