CS Paper Radar

Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research

Eran Hirsch, David Wan, Han Wang, Elias Stengel-Eskin, Mohit Bansal, Ido Dagan

cs.CL ·submitted 2026-08-25 09:34 UTC · v1
top venuecode/data

Abstract

Deep research (DR) systems produce long-form cited reports by orchestrating multiple agents that search and synthesize information from the web. Citations are the primary mechanism for evaluating the faithfulness of these reports, yet current DR systems exhibit poor citation recall. Moreover, improving citation recall is challenging because DR systems are complex multi-agent architectures where information passes through agents like a telephone game, and both content and citations can get corrupted along the way. We propose an evaluation method that pinpoints which agent introduced each error by locally testing agent invocations for faithfulness and verifiability relative to their own inputs. Furthermore, we propose a four-type taxonomy to categorize the discovered errors: hallucination, uncited input reliance, uncited output, or insufficient citations. Applying our method to three top-ranked open-source DR systems, we obtain actionable diagnostics. Almost every agent makes a lot of mistakes with the exception being those that summarize a single document. We find that the dominant error type varies systematically across agents, where the orchestrator mistakes are mostly citation-related. We find that 84.7% of final-report errors in AI-Q originate at the orchestrator, roughly 31% of them hallucinations and the rest citation mistakes. Guided by these insights, we demonstrate that two simple interventions raise citation recall by 5% without degrading output quality.

Comment
Accepted to EMNLP 2026 (Main Conference). Code: https://github.com/eranhirs/who-is-the-agent-to-blame

Closest papers

Nearest neighbours by meaning across the whole index.

ReCite: Agentic Reasoning for Faithful Citation

cs.CL ·2026-09-08 ·Yuyang Huang, Bobo Li, Jiajia Song, Yuzhe Ding, Chong Teng +2 more
top venuepositioncode/data similarity 0.72
Abstract · Findings of EMNLP 2026. Project page: https://hyy279.github.io/ReCite

Accurate citations are the foundation of academic writing, tracing intellectual origins and substantiating core claims. However, manually navigating the growing volume of scientific literature is increasingly difficult, prompting reliance on automatic citation recommendation. While modern retrieval-augmented architectures have largely mitigated the fabrication of non-existent papers, current systems relying on semantic similarity struggle with misattribution, often citing authentic papers that fail to logically support the author's claim. To address this challenge, we argue that accurate citation requires a shift from similarity-based search to active, claim-level reasoning. We propose ReCite, a decoupled agentic framework that orchestrates location perception, intent-aware query planning, and reflective verification. Trained on synthesized reasoning trajectories, our agent verifies claim-evidence consistency and triggers self-correction loops when retrieved candidates lack logical support. Experiments demonstrate that our lightweight framework outperforms state-of-the-art massive generative models in strict citation accuracy. By grounding literature matching in verifiable logic rather than semantic overlap, ReCite establishes a reliable foundation for automated academic writing.

0.41

Self-Consistency of Entity Sets and Citations in a Deep Research System under Repeated Prompting

cs.CL ·2026-10-19 ·Lauri Pykälä, Kimmo K. Kaski
similarity 0.71
Abstract · 6 pages, 3 figures

Deep research agents (DRA) are commonly used for open-ended synthesis tasks where closed-world QA benchmarks are poorly suited to characterize relevant aspects of performance. We study the self-consistency of one commercial deep research system under repeated identical prompting, representing each report with the set of named entities it mentions, categorised by a custom schema, and the set of citations it retrieves. Across repeated runs of a single open-ended cybersecurity related prompt, we quantify pairwise overlap to characterize how stable the retrieved and synthesized output structure is under repeated identical prompts. In ten runs, the pairwise entity Jaccard results in a median of 0.12 and the pairwise citation Jaccard is below 0.02, with discovery curves that do not visibly saturate. The magnitude and structure of (dis)agreement between entity sets varies by entity class, which suggests that self-consistency in a DRA is better understood as a function of the underlying entity space than as a stable property of the system.

0.14

Agentic Data Cleaning Without a Clean Reference: An Experimental Study of Capabilities and Trade-offs

cs.AI cs.DB ·2026-08-14 ·Hadi Fadlallah
similarity 0.71
Abstract · 21 pages, 3 figures, Submitted to New Generation Computing

Data cleaning without a trusted clean reference is challenging because unusual values may represent either genuine errors or valid observations. This paper studies how different agent capabilities affect reference-free data cleaning and proposes an evidence-grounded framework that combines structured context, profiling, LLM reasoning, executable checks, controlled evidence retrieval, source ranking, citation alignment, conservative repair, reversible scripts, and provenance logging. Seven configurations are evaluated across financial, clinical, and environmental-monitoring datasets using controlled synthetic corruption and original-data descriptive analysis, resulting in 126 completed runs. The evaluation includes two comparison baselines and a progressive LLM-based sequence that adds executable tools, evidence retrieval, evidence controls, and conservative repair. In the synthetic evaluation, the deterministic profiling baseline achieved the highest detection F1-score of 0.561. Among the LLM-based configurations, the full conservative configuration achieved the highest F1-score of 0.421, but no configuration performed best across all evaluation criteria. The source-ranked configurations achieved the lowest unsupported-rule rates, while decision-level citation alignment remained weak. The full conservative configuration produced no unsafe or unnecessary modifications, although these rates were already zero before the conservative policy was added, and it performed no direct repairs. Overall, the results show that additional capabilities introduce trade-offs among detection, repair, evidence grounding, conservative behaviour, reproducibility, and operational cost rather than producing consistent improvements. The study provides a structured framework and empirical methodology for evaluating these trade-offs in reference-free agentic data cleaning.

0.12

Beyond the Text: Verifying That Agent-Written Papers Are Backed by Their Artifacts

cs.CL cs.MA ·2026-08-18 ·Qiuhong Shen, Benlong Wu, Hanjin Liu, Yuang Qi, Kejiang Chen
benchmark similarity 0.71
Abstract

Large language model agents are increasingly capable of conducting research autonomously, producing research documents alongside the code and experiments that ostensibly support them. Yet whether the reported findings are consistently supported by corresponding implementations and execution evidence remains largely unexplored: existing review practices primarily assess textual quality and cannot reliably identify inconsistencies such as hard-coded metrics, unimplemented methods, or unsupported experimental results. We present ReAgent, an automated auditing framework for assessing the consistency between agent-generated research documents and their associated repositories. ReAgent constructs structured representations of scientific claims from research documents and uses them to guide repository analysis and evidence collection. Static auditing examines whether claimed methodologies, implementations, and experimental configurations are consistently reflected in the repository, while dynamic auditing executes relevant experiments and collects execution evidence to assess empirical findings. By combining static analysis with dynamic evidence, ReAgent identifies inconsistencies that may remain hidden under either perspective alone, such as experiments that reproduce reported numbers while deviating from the claimed methodology. The collected evidence and audit decisions are organized into a structured repository-level audit report, enabling transparent evidence traceability. We evaluate ReAgent on a manually curated benchmark of agent-generated research document--repository pairs and compare it against representative static and reproduction-based baselines. Experimental results demonstrate that ReAgent effectively identifies inconsistencies between reported research findings and their supporting repository evidence.

0.20

Agents' Overreliance on Unreliable Tools

cs.AI ·2026-09-04 ·Hoyeol Yang, Woojung Song, Taewon Kim, Jonghyun Song, Seoyeon Park +1 more
similarity 0.71
Abstract · 49 pages, 4 figures

LLM agents use tools to access information and perform computations beyond their parametric knowledge. Existing tool-use benchmarks evaluate whether agents select and call the right tools, assuming that tool returns are reliable. However, tools can return plausible but incorrect outputs. We evaluate 14 models with three tools (web search, an LLM sub-agent, and a code executor), corrupting their returns to examine whether agents overrely on unreliable tools. Agents adopt corrupted returns at high rates, with mean adoption exceeding one third for every tool and reaching 68.0% for web search. Corrupted returns also frequently override correct answers that agents give without tools, and this reliance persists in more complex tasks and in tasks that combine multiple tools. Agents often make more tool calls under corrupted returns, and reasoning traces show them questioning a return and even stating the correct answer. Nevertheless, they still pass the corrupted content to users, rarely warning them of the conflict. We further test interventions spanning prompts, tool metadata, post-training, and activation steering. Verification prompts reduce adoption across all three tools while largely preserving accuracy with correct returns, and reliability labels provide partial mitigation in web search. The tested post-training methods offer limited gains, and the effect of activation steering depends on the model and intervention strength. Overall, our findings provide a basis for diagnosing tool overreliance and for developing agents that assess tool returns rather than assuming their reliability.

0.45

Benchmarking and Error Analysis of Deep Search and Deep Research LLM Agents

cs.LG ·2026-10-21 ·Zixuan Yu, Zhenheng Tang, Xiaowen Chu, Bo Han
published similarity 0.70
Abstract · Proceedings of the 1st ICML Workshop on Failure Modes in Agentic AI (FAGEN), Seoul, South Korea, 2026

As Large Language Models (LLMs) evolve into autonomous Deep Search and Deep Research Agents capable of executing long-horizon, multi-step tasks, evaluating their reliability becomes increasingly critical. However, existing evaluation paradigms predominantly focus on final task outcomes, leaving a significant gap in understanding the intermediate execution traces, where agents are highly susceptible to various errors. To address this critical gap, we introduce a comprehensive methodology and analytical framework designed specifically to diagnose errors in intermediate workflows. Building upon this framework, we establish a structured taxonomy that categorizes these vulnerabilities into two primary dimensions: System-Level Errors and Task-Level Errors. Through a systematic examination of step-by-step execution traces and agent-environment interactions, we reveal how these underlying errors affect overall performance. Finally, through empirical experiments and systematic analysis of the identified error types, we provide actionable design guidelines, aiming to enhance the robustness, reasoning capabilities, and reliability of Deep Search and Deep Research agents.

0.29

AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Question Answering over Scientific Citation Graphs

cs.CL cs.AI ·2026-09-28 ·Chanhee Park, Jeongho Yoon, Sungbin Han, Hyeonseok Moon, Heuiseok Lim
top venuebenchmark similarity 0.70
Abstract · Accepted to NeurIPS 2026 Evaluations and Datasets Track

Agentic tasks require a large language model to interact with the world, navigating information and gathering evidence across multiple steps with restricted resources. Due to this complexity, agentic task failures arise from various sources, and pinpointing these failure causes is essential to diagnose and improve agentic systems. Existing benchmarks, however, tend to focus on a single leaderboard score, leaving the underlying failure modes opaque. To fill this gap, we introduce AgentHop, a diagnostic benchmark of 1,011 multiple-choice questions paired with a controlled seven-tool sandbox under fixed token, turn, and tool-call constraints. AgentHop reveals model vulnerabilities by dissecting a single accuracy score along four axes of agent operation: retrieval, synthesis, tool-call, and resource management. Across 19 models, we find that behavior clusters by model family, with tool-call signatures revealing distinct family fingerprints: GPT models commit early, Anthropic and GLM checkpoints verify before committing, DeepSeek and Kimi over-search, and Gemini-3 Pro stays balanced. Decomposed axes further expose within-family structure: Claude Opus 4.6 and Sonnet 4.6 land within one accuracy point yet diverge on retrieval-versus-synthesis emphasis, with Opus retrieving more and Sonnet synthesizing better. We release the full benchmark set and the harness to support diagnostic agent benchmarking.

0.36

Zero-Cost Detection of Fabricated Citations: A Staged Heuristic Approach for Computing Dissertation Assessment

cs.DL cs.AI ·2026-10-05 ·Hazrat Ali, Mohamed Elawady
position similarity 0.70
Abstract · 2 pages abstract, submitted to ACM SIGCSE-virtual conference

Generative AI's growing role in academic writing has driven a rise in fabricated references across dissertations and student reports. We present the Fabrication Severity Stack (FSS), a three-tiered framework classifying citation fabrications by detection complexity - from non-existent references to subtle content misattribution. Building on the FSS, we propose a staged heuristic verification workflow, executable through mainstream LLM interfaces. We argue that this framework offers computer science educators/examiners a practical, zero-cost basis for embedding reference verification into the assessment of dissertation and students' reports. In a pilot study on 100 citations (20% seeded with Level 1 and Level 2 fabrications), the proposed heuristic workflow enabled mainstream LLM interfaces to correctly flag 9-10 of 10 non-existent references and partially detect metadata corruption. These preliminary results suggest that freely accessible, general-purpose LLMs can achieve high recall on citation verification without reliance on proprietary detection tools.

0.06