CS Paper Radar

Video-DCI: How Far Are VLM Agents from Human Viewers in Direct Video Corpus Interaction?

Garry Yang, Yuqiao Liu, Hao Lin, Luohao Pan, Xinru Chen, Wanqi Ren, Zhikang Xu, Yizhan Huang, Huaixiao Tou, James Cheng

cs.CV cs.MM ·submitted 2026-10-22 11:30 UTC · v1
benchmark

Abstract

Humans answer complex visual questions by inspecting short moments, updating what they know, and redirecting attention toward missing evidence. Existing video benchmarks, however, often provide the relevant video, evaluate only final answers or isolated localization, or permit textual shortcuts, leaving multi-round visual evidence seeking underexplored. We introduce Video-DCI, a diagnostic benchmark and functionally matched human-agent protocol for Direct Video-Corpus Interaction. It comprises DCI-Loc for temporal evidence localization and DCI-QA for end-to-end discovery, retention, and composition of cross-video evidence. In DCI-QA, videos within each pool are topically related, yet every answer requires hidden temporal evidence from multiple videos. Given only the question and video pool, humans and agents iteratively inspect bounded local windows, retain supporting evidence, and answer with cited temporal spans. Human trajectories provide reference curves for evidence acquisition and use across rounds. Experiments reveal a consistent retrieval-to-grounding gap. On DCI-Loc, frontier VLMs often retrieve and expose the reference moment, yet convert far less of this exposure into precise final citations than the human reference. On DCI-QA, the loss begins earlier and compounds across videos. Ablation studies further confirm that iterative search outperforms one-shot retrieval. Multi-round search is therefore necessary but insufficient: current agents accumulate candidates and observations more readily than precise, jointly sufficient evidence.

Closest papers

Nearest neighbours by meaning across the whole index.

Finding the Right Evidence: Factor-Guided Coarse-to-Fine Reasoning for Long Videos

cs.CV cs.LG ·2026-08-26 ·Baixuan Xu, Yinyui Xu, Tianshi Zheng, Zhaowei Wang, Weiqi Wang +7 more
top venuecode/data similarity 0.79
Abstract

While LVLMs rapidly improve, long-video question answering still remains challenging: relevant evidence is sparse, and question-relevant context often fails to provide cues that discriminate the correct answer from plausible alternatives. Diagnostic analysis on a manually annotated subset of MMR-V shows that prior agentic systems substantially improve cue retrieval over direct VLM inference yet fail to achieve a corresponding gain in answer accuracy, indicating that the bottleneck lies in option-discriminative evidence rather than topical relevance alone. We propose PACE (Progressive Acquisition of Critical Evidence), a factor-guided framework for long-video evidence acquisition. PACE proceeds in two stages: it first indexes clip-level descriptions guided by question-derived factors without observing the candidate answers; it then uses the candidate answers to derive contrastive cues and queries the index for verification. On MMR-V with the open-source Qwen3-VL backbone, PACE achieves 42.6% accuracy, outperforming direct inference and prior agentic baselines including Deep Video Discovery (DVD). On the same diagnostic subset, PACE recovers 66.9% of the annotated cues, providing empirical evidence that its gains are associated with improved evidence recovery rather than stronger answer-side priors alone. Consistent gains over DVD on LVBench, Video-MME, EgoSchema, and LongVideoBench suggest that option-aware evidence acquisition transfers beyond MMR-V. Code is available at https://github.com/HKUST-KnowComp/PACE.

0.70

GroundedVDR: Grounding Video-to-Web Evidence Dependencies in Video Deep Research

cs.CV ·2026-10-18 ·Xiyuan Liang, Yu Wang, Zuhao Yang, Ziting Wang, Duo Li +5 more
benchmarkcode/data similarity 0.79
Abstract

Video large multimodal models have improved at identifying cross-frame visual evidence and conducting open-web retrieval, as reflected in recent video deep research (VDR) benchmarks. However, existing VDR benchmarks primarily evaluate tool-use trajectories and final responses without verifying whether a model actually uses the temporal video information or the retrieved web evidence. To fill this gap, we introduce \method{}, an evidence-grounded benchmark with three features. First, it combines human review with restricted-access probing across multiple models to exclude questions answerable from parametric knowledge or a single information source. Second, each question carries structured video-to-web evidence annotations that link the relevant video segments to their supporting web evidence. Third, its evaluation protocol separately scores video grounding, web evidence grounding, and final-answer correctness. Across eight multimodal agent systems, the best reaches 76.0\% answer accuracy, yet none recovers the complete required web evidence for more than 8\% of questions. Our benchmark is publicly available at \url{https://huggingface.co/datasets/hh721/vdr_temp}.

0.27

From Intent to Evidence: Policy-Steered Multi-Strategy Retrieval for Long-Video Agents

cs.CV ·2026-08-31 ·Can Zhang, Baofeng Zhang, Xiaotian Han, Junyuan Shang, Yuchen Ding +3 more
similarity 0.77
Abstract · 15 pages, 5 figures, 6 tables (main paper with appendix)

Existing long-video agents acquire evidence through one uniform behavior, ignoring whether the required evidence is concentrated, requires broad occurrence coverage, or must discriminate competing hypotheses---which can cause failure before substantive reasoning begins. Prescribing a fine-grained solution procedure for every question is not a satisfactory remedy, as it restricts autonomous exploration. We propose VESTA, a training-free long-video agent organized as a route-conditioned acquire--verify--consolidate loop. Before exploration, an intent router infers an evidence-acquisition policy---focused, recall, or contrastive retrieval over a shared visual--speech scene index---together with an evidence-accounting policy that configures the evidence view maintained during exploration. Policy-steered retrieval yields provisional references that multimodal evidence operations convert into observations, while the Reasoner remains free to verify them, re-query using intermediate findings, or inspect regions outside the retrieved set. A temporal evidence ledger consolidates observations into an adaptive, compressed view of temporal location, provenance, coverage, conflicts, verification outcomes, and hypothesis support, exposing missing and unresolved evidence to guide subsequent acquisition; finalization prioritizes verified observations. On Video-MME-v2, VESTA improves average accuracy by 2.7 points over VideoARM and gains across all six reported metrics. On LongVideoBench, EgoSchema, and LVBench under shared query-time models, it improves by 6.9 points on the LongVideoBench long subset and 1.5 on LVBench, and matches VideoARM on EgoSchema.

0.35

Agentic Video Understanding: A Survey

cs.CV cs.AI ·2026-09-21 ·Xinyu Deng, Siwen Luo, Daochang Liu
publishedsurveycode/data similarity 0.76
Abstract · 15 pages,3 figures, Accepted to DICTA 2026; Revised Fig. 2 to correct taxonomy inconsistencies

As large language models (LLMs) become capable of processing increasingly diverse modalities and longer temporal contexts, an emerging line of work is moving beyond fixed video-language inference toward agentic systems that actively decide what information to inspect, retain, verify, and act upon. This survey reviews video understanding agents: systems that use video as the primary information source and solve understanding tasks through adaptive state construction and action selection. We first formalize an agent loop for video understanding, then address a central question: why do agents matter for video understanding? To answer this, we organize the literature through a challenge-to-design taxonomy, linking context bottlenecks to hierarchical evidence memory, evidence sparsity to active evidence acquisition, temporal causality to state and process tracking, and multimodal ambiguity to role-specialized coordination. We further review state space paradigms, learning paradigms, supervision signals, benchmarks, and evaluation protocols. Finally, we identify open directions toward agentic-native temporal modeling and video-native agents. Project page: https://github.com/DXY0711/Awesome-Agentic-Video-Understanding

0.41

VideoACE: Benchmarking Agentic Video Understanding through High-Frame-Rate Actions

cs.CV cs.AIcs.MM ·2026-11-18 ·Luoyuan Zhang, Xuanyu Zheng, Yancheng Long, Haoyang Xu, Bin Wen +1 more
benchmark similarity 0.76
Abstract

Agentic video understanding requires models to actively interact with video environments to gather evidence and form judgments. Evaluating this capability therefore requires assessing whether models can translate opportunities for interaction into better understanding. We introduce VideoACE, a benchmark for assessing video understanding under conditions that support sustained interaction. We focus on high-frame-rate action videos, where rapid, subtle changes make targeted observation especially valuable. The benchmark comprises 1,000 multiple-choice questions spanning six categories of fine-grained action understanding, accompanied by temporal evidence annotations and user hints screened for answer leakage. Its evaluation environment combines video inspection tools with a user simulator, allowing models to adjust temporal coverage, sampling density, and spatial detail, and to revisit evidence and revise their judgments under simulated user feedback. We evaluate single-pass observation, autonomous tool use, and tool use combined with user feedback to examine how effectively models exploit these opportunities. Evaluation of five models reveals distinct patterns in how they acquire evidence and respond to opportunities for further interaction. These findings establish VideoACE as a testbed for examining agentic video understanding beyond performance under a fixed observation setting.

0.19

Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents

cs.CV cs.AI ·2026-08-24 ·Wenqi Liu, Shijie Ma, Yunxiao Wang, Meng Liu, Qile Su +19 more
benchmarkcode/datahighly cited similarity 0.76
Abstract

Open-world video understanding often requires a model to locate sparse visual evidence and acquire external knowledge that is absent from the video and its parametric memory. While Thinking-with-Videos enables active temporal perception and Deep Research supports multi-step information seeking, the two capabilities are typically developed in isolation. We introduce VideoRover, a unified Video Deep Research framework that iteratively coordinates video cropping, multimodal search, and webpage browsing. Given a video-question pair, VideoRover uses each tool result to select the next action, so localized video clips guide external retrieval and retrieved evidence triggers further video inspection and verification. To develop this capability, we construct an automated data curation pipeline, producing 26K verified SFT trajectories and 3K challenging RL instances. We also introduce VideoRover-Bench, a benchmark stratified by video duration and research difficulty. Experiments on VideoDR and VideoRover-Bench show that our VideoRover-8B-RL achieves performance comparable to proprietary models in the direct-answer setting without tool use while outperforming larger open-source models equipped with the same tool suite. Ablation studies and training dynamics further validate the complementary roles of active video grounding, external retrieval, and long-horizon reinforcement learning.

0.47

NexSight: Verifiable Tasks for Fine-Grained Agentic Long-Video Understanding

cs.CV cs.AI ·2026-11-17 ·Tianci Xie, Li Chen, Pinxue Guo, Rui Zheng
similarity 0.75
Abstract · 25 pages, including references and supplementary material

Fine-grained questions about long videos ask how many times an action occurs, in what order events unfold, or exactly when a state changes. Answering them can require finding brief moments scattered across hours of footage. Agentic models that decide when and where to look promise to find these moments, yet this ability is hard both to measure and to learn. Sparse supporting evidence may establish that an action occurred without establishing its total count; verifying timeline-wide claims requires checking for missed occurrences and double counting. Answer-only supervision, in turn, does not explicitly supervise the search for evidence. We present NexSight, which builds verifiable tasks to both evaluate and train agentic long-video understanding. Its agentic pipeline, SightForge, accepts a task only if a verifier blind to the draft answer can reconstruct it from evidence densely reacquired from the raw video and counterexample checks fail to refute it. This yields SightLine, 310 evidence-traceable tasks from 108 long videos, evaluated through SightLoop, a shared multiscale harness supporting temporal exploration, local inspection, evidence integration, and revisiting while recording returned observations. On disjoint videos, solver trajectories on verified tasks become step-level supervision for observation decisions. The strongest model reaches only 54.52% on SightLine, yet 79.35% when given the annotated evidence. With under 6K examples, Sight-35B nearly doubles its base model's interactive accuracy (21.29% to 42.26%), outperforms the 397B model of its family, begins using explicit spatial inspection after post-training, which its base model never uses, and gains over 8 points on two external long-video benchmarks.

0.33

Online Video Agent Harness for Long Video Understanding

cs.CV cs.AI ·2026-09-11 ·Sen Yang, Boqiang Duan, Jing Yang, Weihao Bo, Jie Liu +5 more
code/data similarity 0.75
Abstract · 35pages, 12 tables, 10 figures

Long video understanding often behaves like a visual needle-in-a-haystack problem: query-relevant evidence is sparsely distributed across long temporal spans, while packing dense frames into a single VLM context incurs \textit{context rot} and high cost. Existing video agents often rely on query-agnostic offline preprocessing or ad hoc tool sets, which can miss query-specific details and waste computation. In this work, we present VideoXAgent, a purely online video-agent harness for long video understanding that starts from the given video file and user query, plans and decomposes the task, invokes specialized expert tools on demand, and aggregates multimodal evidence to produce a final answer while resolving conflicts among observations. To support this on-demand invocation, we design a suite of heterogeneous expert tools guided by a data-driven taxonomy of atomic capabilities, spanning scripts, VLMs, and domain models (e.g., detection, OCR, ASR, face recognition). The harness further enforces objective evidence prompting and budget-aware control to curb hallucination and non-termination. Across Video-MME-Long, LongVideoBench-Long, LVBench, and MINERVA, VideoXAgent is competitive with frontier LMMs and video agents under a smaller context footprint---about 50k tokens of agent context per sample, even on hour-long videos. In particular, on complex video-reasoning benchmarks such as MINERVA, it matches this level while using only about 15\% of the context of a 1,024-frame dense-packing baseline. Notably, the harness remains effective with a visually weak or even text-only orchestrator, suggesting that strong long-video understanding can emerge from progressive agentic evidence seeking rather than from packing the full video into a single context. Project page: https://go-agent-x.github.io/video_agent_harness/

0.47