CS Paper Radar

Selecting or Solving? What Best-of-N Verification Delivers in Video Reasoning

Martin Q. Ma, Yuxiao Qu, Ruslan Salakhutdinov, Louis-Philippe Morency, Paden Tomasello, Juan Pino

cs.CV ·submitted 2026-10-31 00:54 UTC · v1
publishedcode/data

Abstract

Answering a complex question about a long video is a grounding problem: the answer turns on a few seconds of evidence, and vision-language models remain well below human accuracy at finding them. The standard way to spend extra test-time compute on this kind of video reasoning is best-of-N selection: sample several chain-of-thought answers from a fixed model, then let a second model re-watch the video and pick one. Such pipelines are reported by a single headline number, their margin over a majority vote. That number conflates two effects with opposite deployment consequences: the benefit the second model adds by selecting among candidates, and the benefit it would have supplied by solving the question itself. We separate them, measuring the Selection Increment (SI) -- selection accuracy minus the same verifier's accuracy with the candidates withheld -- alongside the margin over self-consistency, across five long-video benchmarks. The Selection Increment is largest for the weakest verifiers: a verifier that solves poorly on its own leaves the candidates the most room to help. Beating the majority vote follows the opposite profile: where the vote already sits near its ceiling no verifier we test improves on it; below that ceiling only the strongest do. On Video-Holmes, the two come apart on a single pool of candidates -- a weak verifier gains +19.3 points from the candidates yet lands 7.1 points below the vote, while the frontier verifier gains the least and beats the vote. The weakest verifier also shows a failure mode: on EgoSchema it picks by where an option sits in the list rather than by what the video shows. Read as a merit score, SI puts a verifier that fails to beat a majority vote at the top of its ranking; read as an audit, it says which of the two effects a headline number delivers. Code is available at https://github.com/facebookresearch/best-n-selection-video-reasoning.

Comment
Accepted at NeurIPS 2026 VLM4WRD workshop

Closest papers

Nearest neighbours by meaning across the whole index.

Where Do Reported Long-Video QA Gains Come From? A Controlled Audit of Frame Selection, Evidence Injection, and Expert Routing

cs.CV ·2026-11-05 ·Shih-Yao Lin
code/data similarity 0.71
Abstract

Long-video question answering is usually studied as frame selection: which few frames should a vision-language model see? We introduce VideoRouteAudit, a controlled audit that separates this choice from two alternatives, injecting non-visual evidence and routing to another answering model. It holds the backbone, frame budget, decoding, prompt and grader fixed while varying one decision at a time. On the full Video-MME long bucket (n=900, K=8), no tested selector significantly improves over uniform sampling. The largest selection-only estimate is +1.9 pt (95% CI [-1.2,+5.0]) and our cascade is +1.2 pt [-1.7,+4.1]; all are equivalent to uniform within +/-5 pt. The same contrasts are unstable at the n=200 subset scale commonly used: three of four are up to 3.8x larger, one reverses sign, and resampled 200-item subsets reverse the full-bucket effect's sign in 35.8% of draws. The audit also corrects our own first results. The anomaly-domain "+ASR" arms injected no text, so the pooled +4.7 pt gain was selection plus a development prefix. With transcripts, ASR lowers XD-Violence accuracy by 10.0 pt when appended after the question and not at all when placed before it, and lowers UCF-Crime by 13.0 pt either way; on Video-MME, placing it first adds about 4 pt (7 on Qwen2-VL) to uniform sampling and to the cascade alike (post hoc, unadjusted). Swapping the answering model gives the largest contrast (+15.1 pt on MUSIC-AVQA, +33.3 pt on its worst type), but the same audio-visual expert with its soundtrack silenced keeps 92% of the gain (69% with a mismatched soundtrack), so the contrast is mostly a model swap. Code, per-item predictions and scripts that recompute every statistic are available at https://github.com/mikeslin-ir/long-video-qa-audit.

0.21

Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers

cs.LG cs.CL ·2026-10-16 ·Srinivasan Iyengar
similarity 0.68
Abstract · 27 pages (18 main + appendix), 14 figures, 11 tables. Code will be released

A common way to make language models and agents more accurate is to spend more compute at inference time: generate many candidate solutions and submit the one a verifier prefers. Success then requires that some candidate is correct and that the verifier picks it. The first grows predictably with the number of samples; the second does not, because real verifiers (unit tests, majority votes, LLM judges, reward models) accept wrong answers, reject right ones, and make correlated mistakes. With a fixed budget, a practitioner must decide how many samples to draw, which models to draw them from, and how much to spend on verification, and today these choices are made by rules of thumb. We introduce SPLIT, which makes them from a small labelled pilot. It rests on an exact formula for the probability of submitting a correct answer under accept/reject and top-score selection that accounts for varying task difficulty, blind spots shared across models, and correlated repeated verifier calls. SPLIT fits the formula by empirical Bayes and uses it to plan budgets on unseen tasks. Across seven benchmarks spanning math, science QA, code and software-engineering agents, 26 models, 175 agent systems and six kinds of real verifiers, all from public releases, SPLIT lands within 0.02-0.8 percentage points of the best allocation in hindsight, while every fixed rule we test loses at least 2.5 points in some setting. Regret depends only on how well SPLIT ranks configurations, and it stays near-optimal under compute, price and latency costs and online. The formula also explains when asking more LLM judges lowers precision. As a forecaster far beyond the pilot, SPLIT beats power-law fitting when the verifier caps success, and we report where it loses.

0.56

What Does Multiple Choice Actually Measure? Disentangling Video Understanding from Option Exploitation

cs.CV ·2026-11-08 ·Elliott Piercy, Justin Chan, Sepand Dyanatkar
published similarity 0.67
Abstract · Accepted to TAE (Trust-AI-Eval) Workshop at NeurIPS 2026 https://neurips.cc/virtual/2026/loc/sydney/178197

Video understanding evaluations are intended to measure a model's proficiency in video perception and task solving. However, current VideoQA benchmarks struggle to isolate these capabilities due to their reliance on multiple-choice (MC) answer guidance, language priors, and fragile scoring pipelines. Disentangling these confounding factors reveals how much of a model's score is supplied by the evaluation context rather than earned through faithful perception. Across six VideoQA benchmarks of varying length, we find that accuracy is significantly above chance without any visual input. This is caused by shortcutting and option-prior reliance. We express these effects in the language of faithfulness: option-exploitation and answer-conditioned reasoning traces are unfaithful even when they are correct. Together, our results show that a VideoQA score is jointly determined by language priors, option timing, and how outputs and labels are converted into correctness. We produce a novel Option-Delayed Replay (ODR) mechanism that discloses options at controlled stages of an open-ended (OE) trajectory, to obtain objective MC scores from faithful option-free inference. Our work demonstrates that agentic methods with ODR and faithful inference can match option-exploiting performance. Finally, we formalize our diagnostic collection and ODR as a reproducible evaluation protocol, and we release a human-relabelled edition of LVBench (correcting $20.5\%$ of keys) to remove annotation noise as a confounding variable.

0.30

VisConf: Quality-Aware Consensus for Best-of-N in Vision-Language Model Reasoning

cs.CV ·2026-10-17 ·Muhammad Usman, Muhammad Qasim Ayub, Usman Ahad, Muhammad Tahir
published similarity 0.66
Abstract · Accepted at NeurIPS 2026 Workshop: VLM4RWD. 20 pages, 8 figures. Code: https://anonymous.4open.science/r/VisConf-VLM-BoN/

Best-of-N methods improve text-only LLM reasoning by sampling multiple trajectories before selecting an answer. For Vision-Language Models (VLMs), answer frequency and model confidence alone can be insufficient, as sampled generations also vary in their engagement with the visual input. We find that distributional confidence and visual attention are complementary indicators of rollout quality. Each is limited on its own, while trajectories that show strong signals in both are more likely to be correct. Motivated by this, we introduce VisConf, a rollout-quality score that multiplicatively combines visual engagement with candidate-calibrated Self-Certainty, penalizing rollouts that are weak in either signal. We incorporate Rank-Weighted Consensus (RWC), a tuning-free aggregation rule that converts within-question VisConf ranks into answer-level vote weights. Across three VLMs on MathVista, MMMU-Pro, and MMStar, VisConf with RWC consistently outperforms strong BoN baselines. Our analysis shows particularly large gains when correct trajectories are present but underrepresented in the candidate pool.

0.57

When to Rethink: Learning Multi-Perspective Self-Verification for Vision-Language Models

cs.AI ·2026-10-04 ·Ziquan Zhu, Hanruo Zhu, Si-Yuan Lu, Morris Yu-Chao Huang, Yicheng Lin +8 more
similarity 0.66
Abstract

Vision-language models (VLMs) have achieved strong performance in multimodal reasoning, yet they remain prone to generating plausible but incorrect answers. Self-verification offers a practical way to improve answer reliability without relying on external judges, but existing methods typically depend on a single verification criterion or fixed prompt, resulting in incomplete and unstable reliability estimates. We first systematically analyze how verifier capability and prompt design affect verification performance. Our findings show that stronger verifiers provide more reliable judgments, while verification performance is highly sensitive to prompt choice, with no single prompt consistently dominating across tasks. Guided by these findings, we propose \texttt{MOTIVE}, a \textbf{M}ulti-View Self-Verificati\textbf{O}n wi\textbf{T}h Rel\textbf{I}ability-Guided Selecti\textbf{VE} Rethinking framework for reliable multimodal reasoning. \texttt{MOTIVE} evaluates each candidate answer from complementary verification perspectives and learns a correctness-aligned reliability score through correctness-grounded multi-view verification learning. During inference, this score governs an accept-or-rethink decision, allowing reliable answers to be returned directly while uncertain ones trigger history-guided rethinking. Extensive experiments across diverse multimodal benchmarks and VLM backbones demonstrate that \texttt{MOTIVE} consistently outperforms strong self-verification and self-correction baselines. Further results show that reliable verification improves accept-or-rethink decisions and reduces unnecessary reasoning turns, enabling more reliable and efficient self-verification without an external judge.

0.16

Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization

cs.AI ·2026-08-15 ·Suyash Maniyar, Armaan Sandhu, Abhishek Mishra
benchmark similarity 0.66
Abstract · Accepted at the AI Measurement Science Workshop, COLM 2026

When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization. Endpoint accuracy on the training distribution cannot tell the two apart, because solving the task and exploiting a surface feature can satisfy the reward equally well. We treat this as a measurement problem: what does a benchmark score measure once a model has been optimized against a correct but confounded signal? We train language models with GRPO on multiple-choice math problems where the correct answer is always option A, then evaluate on an unseen test set with unbiased answer positions. Across Qwen2.5, Llama 3.x and Gemma-3 models, biased training often drives option-A rates above 0.90 in smaller models and collapses unbiased accuracy toward chance, so accuracy stops measuring math ability and instead measures an answer-position policy. We further find reasoning-answer decoupling: capable models generate reasoning that reaches the correct numeric answer while still selecting A. We track this with numeric extraction and an LLM judge (GPT-4.1-mini; Qwen2.5-3B decoupling rate is about 0.66). The broken construct generalizes beyond the training domain: biased models inflate A-rates on out-of-domain MMLU and value-laden prompts. Continued training on unbiased data reverses the in-domain shift unevenly and only partially reverses the out-of-domain one, so a model can appear restored on its training distribution while remaining biased on unseen inputs. Reasoning-answer decoupling rate, together with answer distributions and out-of-domain behavior, separates capability loss from a learned, transferable shortcut.

0.13

Learning When to Trust and Reobserve Sparse-Frame VideoQA from Temporal Sampling Perturbations

cs.CV ·2026-10-12 ·Shujia Liu
publishedposition similarity 0.66
Abstract · Accepted to NeurIPS 2026 Workshop on Physical Understanding for Decision-Making (PhysUnderstand), Poster

Sparse-frame video-language models can change their answers when only the sampling timestamps shift. Raw agreement exposes this problem, but it need not rank errors better than token confidence or identify which additional observation will repair an answer. We argue that answer risk and action value must be estimated separately. PhaseGuard+ probes a frozen VLM at controlled temporal offsets, predicts the reliability of the phase-vote answer, and predicts the example-wise correctness gain of a specific rescue action. The second score routes only examples expected to benefit from denser frames or a stronger VLM. Both predictors use lightweight models trained on public data; the VLMs remain frozen. We use 256 Perception Test videos for method development and evaluate on a disjoint 256-video test set comprising 554 Physics questions. For Qwen3-VL-8B, reliability AUROC rises from 68.0% with token softmax to 74.8% (+6.8 pp, 95% CI [2.6, 10.7] pp). Routing 30% of questions to Qwen3-VL-32B reaches 67.87% accuracy, compared with 63.66% for budget-matched random routing (+4.21 percentage points, CI [1.89, 6.40]). All four tested rescue pairs significantly outperform random routing. Temporal sensitivity becomes actionable when the system learns both whether an answer is unreliable and whether an available action is likely to fix it.

0.27

CASE: Cost-Aware Stopping for Efficient Long-Video Agents

cs.AI cs.CV ·2026-10-04 ·Yiming Du, Chenghao Liu, Zhiyuan Liu, Fangxing Zheng, Zhao Wang +2 more
benchmark similarity 0.65
Abstract · 40 pages, including references and appendices

Long-video agents can actively gather question-relevant evidence, but they typically leave a central decision implicit: when has the agent seen enough to answer? We propose CASE, a plug-in termination framework that frames this decision as policy-conditioned sequential stopping. At each causal checkpoint, CASE combines an auxiliary multiple-choice assessment of accumulated evidence with the host agent's execution state. From complete native trajectories, we construct a cost-aware target that compares answering now with stopping later along the same search path, accounting jointly for answer correctness and the full cost of continued reasoning. A lightweight Ridge regressor learns this decision gap and produces STOP/CONTINUE decisions. We evaluate three vision-language models with VideoSeek and AVP. On Video-MME, end-to-end accuracy changes by +0.67 percentage points on average while CASE reduces model-token use by 53.63%. The same frozen policies then transfer zero-shot to LongVideoBench and MLVU, with end-to-end accuracy changes of +3.38 and +4.58 points while saving 58.78% and 51.28% of model tokens, respectively. Across all agent-model-benchmark combinations, CASE attains the highest accuracy-efficiency Pareto-frontier coverage among the compared stopping methods (83.3%) at the selected operating points. Online execution preserves this favorable accuracy-efficiency trade-off and additionally reduces measured runtime by 54.1% on average. CASE provides a plug-in termination framework for long-video reasoning agents, enabling them to decide when further evidence acquisition is no longer worthwhile.

0.24