CS Paper Radar

Beyond Semantic Similarity: Performance and Costs of Agentic Retrieval for Complex Tasks

Reza Esfandiarpoor, Radek Osmulski, Yauhen Babakhin, Gabriel de Souza P. Moreira, Oliver Holworthy, Jie He, Ronay Ak, Jiarui Cai, Ryan Chesler, Bo Liu, Even Oldridge

cs.IR cs.AI ·submitted 2026-10-05 03:55 UTC · v1
code/data

Abstract

Modern information systems, including many agentic workflows, use dense retrieval to explore large amounts of unstructured data. However, dense retrieval relies on surface-level semantic similarity, which is insufficient for increasingly complex search applications. Here, we investigate agentic retrieval that combines the reasoning capabilities of Large Language Models (LLMs) with the efficient corpus exploration of retrievers in a ReAct agentic loop to solve complex retrieval tasks. In our experiments, we show that agentic retrieval is more effective than standard retrieval, improving nDCG@10 by 8.7 points using the same embedding model. Moreover, while specialized retrieval methods struggle on out-of-domain tasks, agentic retrieval is highly generalizable: the same pipeline achieves competitive results on both the ViDoRe v3 and BRIGHT leaderboards. However, this improvement comes at a cost. On average, agentic retrieval takes 107.4 seconds, compared to 0.67 seconds for standard retrieval, and consumes 764.1K input and 5.8K output tokens per query. In short, our study demonstrates the effectiveness of agentic retrieval in modern data systems and motivates future work on more cost-efficient retrieval agents for large-scale deployment.

Comment
Code: https://github.com/NVIDIA/NeMo-Retriever/tree/main/retrieval-bench

Closest papers

Nearest neighbours by meaning across the whole index.

Overview of the TREC 2025 Million Large Language Models track

cs.IR cs.AIcs.MA ·2026-09-25 ·Evangelos Kanoulas, Panagiotis Eustratiadis, Jamie Callan, Mark Sanderson, Yongkang Li +3 more
surveybenchmark similarity 0.74
Abstract · NIST TREC 2025 Proceedings

Agentic AI envisions ecosystems of intelligent agents collaboratively solving complex tasks with minimal human intervention. In such ecosystems, each agent possesses specialized expertise, making effective expert selection central to overall system performance. While most current approaches assume a small number of well-documented models, real-world expertise is far more diverse and cannot be adequately captured through static metadata or hand-written descriptions. We anticipate a future with millions of specialized language models (LLMs), each excelling in different domains or problem types. Rather than relying on predefined capability statements, we propose a retrieval-based paradigm in which an assistant agent infers expertise dynamically by examining models' observable behavior. Upon receiving a user query, the assistant ranks candidate LLMs based on demonstrated competence, enabling efficient and adaptive expert selection. The TREC Million LLM Track operationalizes this paradigm by shifting the retrieval target from documents to expert LLMs. Participants are given a discovery set consisting of queries, answers, and log-probabilities from more than one thousand LLMs and are challenged to infer meaningful expertise representations for each model. Given an unseen test query, systems must then rank the LLMs according to their expected performance, providing the first large-scale benchmark for expertise retrieval in agentic AI.

0.36

Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems

cs.IR cs.CL ·2026-09-08 ·Maximilian Schall, Sedigheh Eslami, Markus Krimmel, Antoine Chaffin, Louis Milliken +2 more
benchmarkcode/datahighly cited similarity 0.72
Abstract

Evaluating first-stage retrievers in large-scale production RAG requires a benchmark that pairs a large-scale corpus with a large set of agent-reformulated search queries based on real user queries and their conversation threads, and that labels many relevant documents per query. No existing public benchmark evaluates this setting: large-scale collections typically provide only a small number of evaluation queries, whereas benchmarks with many queries generally contain only millions of documents. Moreover, most benchmarks assess human-written queries, while the first-stage retrievers in agentic RAG pipelines serve machine-written reformulations whose distribution differs from human search behavior. To overcome these evaluation gaps, we introduce Q2D-Web (Query2Doc-Web), a large-scale agentic retrieval benchmark consisting of a 190M-document web corpus and 70k agentic search queries in ten languages, reformulated from real-world user queries in production systems. Q2D-Web provides three sets of fixed relevance judgments: agent citations, production rankings, and a combined set that unions both signals and adds LLM-based judgments of unlabeled pooled documents to reduce false negatives. We benchmark 13 retrievers including lexical, dense, and late-interaction models and find that their relative ordering is largely insensitive to the choice of judgment set, while diverging substantially across topical domains, query languages, and query types. To enable fast evaluation, we also study subcorpus sampling as an approximation to full-corpus evaluations. Retaining a third of the corpus, selected by reciprocal rank fusion over pooled retriever runs, preserves the full-corpus model ranking under the combined judgments while raising absolute Recall@1000 only by 4 to 7 points. The public leaderboard is accessible under: https://huggingface.co/spaces/perplexity-ai/q2d-web-leaderboard

0.57

Semantics Delivery Network: Rethinking Web Retrieval Infrastructure for LLM Agents

cs.NI cs.IRcs.LG ·2026-09-18 ·Peichun Hua, Yunming Xiao
positioncode/datahighly cited similarity 0.72
Abstract · Accepted and to appear at the 25th ACM Workshop on Hot Topics in Networks (HotNets 2026). 13 pages, 3 figures

Large language models (LLMs) increasingly ground their answers in proprietary data and live web content through both single-shot retrieval-augmented generation (RAG) and multi-turn agentic search. Yet the web path they use was built for human readers: search services return URLs and snippets ranked by generic relevance, and content delivery networks (CDNs) cache URL-addressed objects without knowing which passage an agent needs. LLMs instead consume short, semantically coherent passages - hereafter, chunks - selected for downstream utility rather than similarity alone, retrieve statefully across reasoning turns, and, as uncoordinated streams, repeatedly fetch, clean, chunk, and embed the same content. We argue that semantic chunk retrieval should become a first-class network-delivery abstraction, and propose the Semantics Delivery Network (SemDN): an origin-authorized, hierarchical substrate that indexes, searches, and caches web content at chunk granularity, serving clients and origins alike. SemDN is more than finer-grained caching: it must estimate missing or stale evidence without an explicit miss signal, let clients choose retrieval representations that origins need not see, and share state among clients with the same representation. Preliminary results show that page-first retrieval processes two to three orders of magnitude more content than agents consume, that agentic queries exhibit task-local chunk reuse invisible to URL caches, and that chunk delivery improves answer quality per context token. We organize an open research agenda around shareable retrieval state, hierarchical caching and refresh under coverage risk, and deployability, including incentives, privacy, and verifiability. Our code is available at: https://github.com/sarendis56/SemDN_HotNets26

0.62

VASTRA: Efficient Agentic Top-K Interactive Video Analysis

cs.DB ·2026-11-10 ·Tushar Tyagi, Aditya Raghavan, Haoran Qiu, Francisco Romero
benchmark similarity 0.71
Abstract

Vision-language models (VLMs) enable natural language video retrieval, bypassing traditional hand-engineered pipelines. Integrating agentic reasoning frameworks (e.g., ReAct) further advances this paradigm by allowing models to dynamically inspect video content rather than relying on a single visual pass. Applying this pattern naively to video retrieval inherits problems that video systems cannot afford: results vary from run to run under identical conditions, and cost and latency scale with every reasoning step the agent takes rather than with the work the query actually requires. We present VASTRA, an agentic video retrieval system whose execution is dictated by query complexity rather than an open-ended reasoning loop: cheap, reliably answerable queries are ranked directly from pre-ingested summaries, while harder queries trigger targeted verification only where relevance is genuinely uncertain. This keeps control flow external to the model rather than latent in a growing chain of thought, establishing deterministic control flow by construction and end-to-end rank reproducibility under greedy serving. Across 40 benchmark queries, VASTRA exposes explicit presets: VASTRA-Fast runs 12.2x faster (4.2 s vs. 51.2 s) and 10.4x cheaper ($0.0044 vs. $0.0458) than best performing ReAct agent at a 5.0-point mAP reduction, while VASTRA-Max matches peak ReAct agent accuracy (0.653 vs. 0.649 mAP) at comparable latency, trading higher spend ($0.1022 vs. $0.0458) for zero run-to-run variance.

0.05

Question's Gambit: The First Move Matters in Agentic Deep Search

cs.AI cs.CLcs.IR ·2026-09-13 ·Radin Hamidi Rad, Amin Bigdeli, Negar Arabzadeh, Sajad Ebrahimi, Charles L. A. Clarke +2 more
code/data similarity 0.71
Abstract

Deep research agents answer complex questions through iterative loops of searching, reading, and reasoning. Recent work on reasoning-intensive benchmarks such as BrowseComp-Plus shows that well-configured lexical retrieval can surface high-quality evidence, yet agents may still fail to connect documents carrying evidence to the gold documents. We identify a deep research agent's first retrieval move as an important design decision for this setting. We introduce Question's Gambit, a first-move retrieval module that decomposes the question into a set of clues, reformulates them into complementary searches, consolidates the retrieved results, and reranks the candidate pool before the agent begins its iterative search-and-reasoning process. This produces an opening context designed to support both clue aggregation and final-answer verification. We further evaluate on MultiHop-RAG to test whether these benefits transfer beyond BrowseComp-Plus to a more conventional multi-hop question structure. Experiments on BrowseComp-Plus show that Question's Gambit improves retrieval recall and downstream agent accuracy over strong baselines, improving answer accuracy from 83.1% to 90.5% with gpt-5.5 over Pi-Serini, the strongest reported agentic baseline. Our results confirm that effective agentic deep research depends not only on the tools available inside the loop, but also on the quality of the first move. We published our implementation publicly at https://github.com/radinhamidi/Question-s-Gambit.

0.47

D2-ScaleAgent: Dual-Dimensional Scaling for Long Document Understanding

cs.CL ·2026-08-17 ·Hao Zhang, Longrong Yang, Lunhao Duan, Ziyang Wang, Qing-Guo Chen +1 more
similarity 0.71
Abstract

Multi-modal retrieval-augmented generation (RAG) is a key technique for visually rich long document understanding. Existing multi-modal RAG methods are progressively advancing toward multi-agent systems: they first retrieve relevant pages based on a query, and then iteratively understand information within those pages. However, these methods typically rely on fixed workflows and lack the ability to dynamically scale computation at test time, often leading to insufficient evidence. To address this, we propose D2-ScaleAgent, an agentic framework that introduces a dual-dimensional scaling paradigm for retrieval and reasoning. The core of D2-ScaleAgent is a Verifier agent-driven dynamic routing loop based on the intrinsic difficulty of the query, centered around a continuously updated evidence bank that serves as the agent's dynamic working memory: when retrieval needs to be expanded, the agent routes outward (retrieval scaling), decomposing the query into attributes and performing parallel page retrieval, followed by adaptive pruning to ensure comprehensive evidence coverage. When fine-grained reasoning is required, the agent routes inward (reasoning scaling), dynamically selecting sub-agents with varying granularity and count to extract evidence from pages. Finally, D2-ScaleAgent achieves logical closure over the evidence chain. Extensive experiments demonstrate that D2-ScaleAgent is effective on long and visually rich document benchmarks like MMLongBench-Doc, LongDocURL, etc.

0.41

SkillQR: Agentic Skill Discovery for Reason-Intensive Query Rewriting

cs.IR ·2026-10-24 ·Kenan Jiang, Meixiu Long, Zihao Zhang, Ran Xu, Eugene Agichtein +1 more
similarity 0.70
Abstract

Dense retrievers provide grounded evidence that mitigates hallucinations in outputs from Large Language Models (LLMs). Real-world queries are often verbose, ambiguously framed, or require multi-step reasoning, introducing irrelevant details that weaken semantic matching for dense retrievers. Query rewriting is a widely adopted pre-retrieval technique that leverages LLMs to reformulate queries and bridge the semantic gap between such queries and the retrieval target. Existing rewriting approaches either rely on expensive proprietary models, substantial training data, or iterative retrieval and rewriting, while largely applying a uniform rewriting strategy regardless of the retrieval corpus. We observe that effective query rewriting instructions are not universal: the optimal strategy depends on the retrieval target, rather than the model alone. We introduce SkillQR, a query rewriting framework that automatically discovers reasoning strategy sets from the retrieval target and dynamically routes a query to the appropriate rewriting strategy at inference time. Without relying on training data, SkillQR achieves competitive performance compared to fine-tuned rewriting models on the BRIGHT and R2MED benchmarks, while also providing lower inference latency than iterative rewriting approaches.

0.13

LexiHorizon: Stabilizing Reinforcement Learning for Long-Horizon Deep Search

cs.AI ·2026-10-04 ·Zhiqing Nong, Liang Wen, Chao-Hsuan Liu
similarity 0.70
Abstract

Deep search agents tackle complex knowledge tasks through iterative retrieval, multi-hop reasoning, and evidence synthesis across multiple sources. Existing approaches typically assume relatively stable retrieval systems and operate over short-horizon tool interaction. However, when retrieval is sensitive to query formulation, even a semantically appropriate query may fail to surface critical evidence because of mismatched entity names, aliases, or keyword combinations. Recovering from such failures requires repeated query reformulation and longer interaction trajectories. This setting poses a distinct training challenge, as the policy must sustain long-horizon query exploration while managing an expanding volume of retrieved content. We propose LexiHorizon, a framework for training search agents over long horizons that expands the trajectory context budget, manages accumulated retrieval content using a window over recent tool observations while preserving the reasoning history, and introduces an outcome-gated search-effort reward that provides a bounded bonus for tool invocations to trajectories with nonzero answer reward. Experiments on XBench, WebWalkerQA, and BrowseComp-ZH show that the resulting 9B model consistently outperforms both its base model and MiroThinker-1.7-mini, with maximum absolute gains of 8.7 and 23.8 percentage points, respectively. These results suggest that combining an extended context budget with reasoning-preserving context management benefits long-horizon deep search agents.

0.16