CS Paper Radar

NumTrace: Auditing the Number Provenance of Financial Analyst Agents

Tianlei Zhu, Matt White, Xue Liu, Xiao-Yang Liu

cs.CL cs.AI ·submitted 2026-10-26 05:58 UTC · v1
benchmark

Abstract

A numerical claim generated by a financial analyst agent can match a number in a company filing but refer to the wrong entity, concept, period, unit or scope. A check based on reference answers can mark such an answer as wrong but does not explain why it fails. In this paper, we present NumTrace, a new benchmark that audits the provenance of each numerical claim made by financial analyst agents: its source and financial semantics. NumTrace currently has 540 questions over 2,163 SEC EDGAR documents. To the best of our knowledge, it is the first numerical benchmark about who holds a company's shares (Form 13F) and what its executives are owed (Item 402(i)). On a frozen snapshot of these documents, we build the reference answers according to accounting rules. NumTrace then gives each claim three separate verdicts: value, binding and support. By retaining separate value and binding verdicts, NumTrace detects right number, wrong account (RNWA) errors: a claim has an RNWA error when it passes the value check but fails the binding check. The RNWA rate quantifies the gap between value-only and joint value-and-binding pass rates. Finally, we audit three agents, Claude Opus 5.5, GPT-6 Astra and DeepSeek-V4.1-Flash. RNWA errors make up 67.5-81.8% of each agent's failing claims. In particular, on 13F co-filing group questions, about 91% of claims pass the value check, but fewer than half pass the binding check.

Comment
22 pages, 8 figures, 12 tables

Closest papers

Nearest neighbours by meaning across the whole index.

FinVerify: A Verification Framework for Financial Agents

cs.AI ·2026-11-16 ·Mao Guan, Qian Chen, Ge Meng, Xiao-Yang Liu
published similarity 0.65
Abstract · 15 pages, 2 figures. Accepted to the NeurIPS 2026 Workshop on Who Verifies the Agents? Toward Reliable Agent Development

Financial agents produce numerical claims that can be wrong even when their calculations execute correctly, due to mismatched reporting periods, unsupported sources, incompatible units, or incorrect financial definitions. We introduce FinVerify, an evidence-grounded framework for verifying financial agents' numerical claims. Based on an analysis of 190 incorrect trajectories from the Agent Finance Reasoning dataset, we define five error categories--temporal integrity, source fidelity, comparability, accounting integrity, and financial-model fidelity--and operationalize them as verification protocols. An LLM structures claims and supporting evidence, while deterministic validators check financial requirements; when authoritative formulas are available, Z3 checks consistency with grounded inputs. FinVerify accepts only if all applicable checks pass, rejects proven violations, and abstains when evidence is insufficient. We evaluate the framework against a Direct baseline and an LLM judge across five fixed-candidate benchmarks. On Agent Finance Reasoning, accepted-claim precision increases from 56.4% for the LLM judge to 76.7% at 81.2% coverage. On XBRLFiling, FinVerify achieves 99.8% accepted-claim precision and 97.0% decision accuracy; on FinanceBench-67, it reaches 87.9% accepted-claim precision versus 69.7% for the judge. Improvements are smaller on FinQA and TAT-QA, where authoritative evidence is limited. These results suggest that explicit, evidence-backed verification can improve financial-agent reliability while exposing trade-offs between correctness, false rejection, and abstention.

0.37

VeriFin: A Neurosymbolic Framework for Verifying LLM-Generated Financial Claims

cs.CE ·2026-08-10 ·Bethel Hall, Sachi Shome, William Eiers
benchmarkhighly cited similarity 0.65
Abstract · 9 pages

Large Language Models (LLMs) are being adopted in financial workflows, where accuracy and reliability are critical. One such application is financial question answering over public company filings. However, LLMs struggle with quantitative financial reasoning, including identifying financial facts, interpreting reporting periods and units, and determining appropriate metrics, leading to incorrect answers. Recent RAG-based methods have improved financial question answering but do not ensure factual and numerical correctness. LLM-based verifiers may also accept incorrect numerical claims. To address this, we introduce VeriFin, a neurosymbolic framework for verifying LLM generated numerical financial claims. Given a question and a generated answer, VeriFin retrieves filing evidence, grounds values in XBRL facts, and establishes the required formula independently of the answer. It then uses the Z3 SMT solver to check the resulting arithmetic constraints. VeriFin accepts consistent claims, rejects inconsistent claims, and abstains when required facts or formulas cannot be established. To evaluate VeriFin, we constructed XBRLFiling, a dataset of 600 numerical questions from corporate filings, and selected 67 and 403 questions from FinanceBench and TAT-DQA, respectively. Across six answer models and three datasets, VeriFin achieves 100% observed precision among accepted answers with known correctness. For Qwen3-30B on XBRLFiling, VeriFin accepts 304 correct answers compared to 166 for an LLM judge, while reducing false accepts from 87 to 0. We release our benchmark to support research on reliable financial question answering.

0.49

ClaimScope: Label-Blind Witness Verification for Mathematical Claim Auditing

cs.AI cs.LG ·2026-10-15 ·Mushir Akhtar
publishedbenchmark similarity 0.63
Abstract · Accepted at NeurIPS 2026 Workshop (Mathematical Reasoning and AI)

Mathematical agents are commonly evaluated by answer accuracy or proof acceptance. We introduce ClaimScope, a 72-task benchmark for auditing proposed claims through a verdict, scope, counterexample, and repair. Tasks cover optimizer-state transport, finite-horizon affine recurrences, and exact $2\times2$ linear algebra. We evaluate Qwen3-8B with direct generation and four matched two-pass conditions, each starting from the same draft. A witness checker evaluates only the model's submitted state, without consulting reference verdicts or supplying a replacement witness. Its feedback still conveys information about validity: a verified refutation establishes that the claim is false, and rejection reasons can suggest structural fixes. In this evaluation, verifier feedback yields $23/39$ accepted counterexamples on invalid claims and $25/72$ safe completions, compared with $15/39$ and $20/72$ for checker-free generic feedback. Strict completion remains $17/72$; verdict accuracy changes from $47/72$ to $46/72$. The claim-grouped safe-completion difference is $.094$, with a bootstrap 95\% interval of $[.000,.198]$. All seven task-level gains follow horizon/input-length diagnostics. A post-hoc schema-only correction of the direct outputs reaches the same 25 safe tasks without another model call. These results separate claim classification from executable evidence, while locating the observed gains in witness-structure correction rather than demonstrating broader reasoning improvement.

0.34

Beyond fabrication: Misattributed numbers in papers written by AI research agents

cs.AI cs.CLcs.DL ·2026-10-28 ·Mosab Alfaqeeh, Omar Mohammed Horani, Mou'th Hourani, Hashem Alfaqeeh
benchmarkcode/data similarity 0.59
Abstract · v2: corrected author names; clarified the second-judge comparison and two table captions. 43 pages, 5 figures, 13 tables. Code and data: htt…

Verifying numerical claims in research written by AI agents is an important but challenging problem. Recent benchmarks show that many agent-written papers fail integrity checks, but such verdicts, often produced by a single LLM judge, identify failure without explaining its nature, and existing checks either test a paper for internal consistency or test whether a reported value appears somewhere in an agent's output. We re-analyze all 540 released judgments from a recent public benchmark of AI research agents (nine agents, fifteen open research problems) and manually label 256 numerical claims from 18 submissions. Failures concentrate on alignment between the paper and the underlying work rather than overt cheating, and about four in five unsupported claims correspond to values in the agent's own files that come from the wrong experiment, metric, or run. We introduce a deterministic claim-to-evidence checker that scores whether the surrounding context identifies the claimed result and calibrates each claim against numerical decoys. With its rules fixed before labeling, it certifies claims with over 90% precision against blinded annotators (78 of 85; Cohen's kappa = 0.89), while verifying only a subset of claims; most claims it cannot verify are nevertheless traceable, so failure to verify is not evidence of falsehood. An LLM verifier given the same evidence recovers more traceable claims but accepts about three times as many misattributed ones, so the two are best used together. Applied to 99 submissions, the checker reveals large differences in traceability across agents, and on 30 submissions a second judge configuration, with a different model and harness, orders the agents' vendor groups as the official judge does while passing twice as many papers. These findings motivate provenance-first auditing of AI-produced research.

0.37

Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge

cs.CL ·2026-10-06 ·Chuhong Xu, Bo Su, Ziyao Chen, Ruiyang Xu, Shimeng Dai +1 more
published similarity 0.59
Abstract · counterfactual citation perturbation, evidence attribution verification, financial document question answering, Jev, LLM-as-a-judge, probabi…

Financial reports repeat values across periods, metrics and accounting lines, allowing an LLM-generated calculation to be numerically correct while citing the wrong financial role. We evaluate what probabilistic evidence verification adds beyond number matching using Jev as a source-support verifier for GPT-4.1-mini calculation traces. A signed-number-at-pointer baseline explains most recovery over exact quotation checks. To isolate the remaining role-recognition problem, we hold operands and arithmetic fixed, move citations between same-number cells, and retain controls that express equivalent facts. These contrasts reveal both wrong-role citations that pass and valid alternative citations that are withheld. Explicit column labels improve selected wrong-role decisions while also lowering support for some equivalent evidence. A constructed follow-up on 36 new source pages, labeled by a non-author reviewer, extends this evaluation and exposes the same tradeoff between detecting role errors and retaining valid citations. The contribution is a controlled evaluation that identifies what a probabilistic financial verifier distinguishes when numerical matching is held fixed. For LLM-based financial assistants, it makes numerical correctness, cited-role support and acceptance outcomes separately assessable.

0.30

LLM-Generated Investment Research Memos Need Validation

cs.CE ·2026-11-03 ·Xukang Gong, Chenri Luo, Binyan Yu, Xiao-Yang Liu
publishedhot direction similarity 0.59
Abstract · 24 pages, 4 figures. Accepted at the TAE (Trust-AI-Eval) Workshop, NeurIPS 2026

LLMs are increasingly adopted to draft long-form investment research memos, but their reliability is judged claim by claim, while readers read each memo as a whole. In this paper, we show that a low claim-level error rate does not necessarily mean that a memo is reliable: even with a refutation rate of just 1-2%, roughly four in ten memos contain at least one claim contradicted by official filings. First, we generate 1,440 memos with four recent LLM-powered agents for 30 companies over 12 quarters, decompose them into 77,272 financial claims, and verify each against the same official source bundle by retrieving evidence, matching it to the claim, and verifying the outcome. Second, we find that only 1.17-1.77% of claims are refuted, yet 36.94-41.11% of memos contain at least one refuted claim, and agents differ in severity: Codex has the lowest memo-level rate; a qualitative review further groups errors into four patterns (time, unit, context, concept).

0.70

Right Numbers, Wrong Questions: Evaluating LLMs on Underspecified Financial Queries

cs.AI cs.CLq-fin.GN ·2026-10-28 ·Radha Yadav, Prasanna Devadiga, Peeyush Mishra, Apoorv Singh
similarity 0.58
Abstract · Accepted for oral presentation at the 3rd Workshop on LLMs and Generative AI for Finance (AI4F) at ACM ICAIF '26, Milan, Italy, November 14,…

Financial assistants answer questions that sound factual but depend on what the user leaves unsaid: what a data figure measures, which product variant is meant, and the investor's situation and the rules that apply. An assistant that ignores these can still answer with a real number from its data, one that answers a different question. We call this frame collapse. We study it in Indian mutual funds, where product variants and tax rules are formally defined, so the correct answer can be computed for every reading. One study tests each challenge. Given an ambiguously labelled figure, models used it, often under the user's label. Given every alternative labelled, they listed them all, but the layout decided which came first. Missing a fact about the investor, weaker agents assumed a default or an outdated tax rule; requiring the fact did not guarantee its use. Because such answers match their data, checks against the data pass them. An AI judge caught mislabelled figures only when told what each field means, and still passed answers that silently chose another reading. We show how these failures arise, not how often, and propose controls for financial assistants.

0.37

FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems

cs.AI cs.IR ·2026-08-19 ·Pratik Ghawate
benchmarkcode/data similarity 0.57
Abstract · 19 pages, 4 figures, 7 tables. Code and data: https://github.com/PratikGhawate/FinRCA-AI-Bench

Large language models are increasingly used to support financial operations, but their apparent reasoning performance can depend on whether they receive the right evidence. In financial reconciliation, the evidence needed for diagnosis is distributed across invoices, purchase orders, approvals, allocations, payments, ledger entries, and bank activity, linked by transactional relationships rather than textual similarity. End-to-end accuracy can therefore conflate evidence access with reasoning quality. We introduce FinRCA-Bench, a deterministic synthetic benchmark of 2,250 accounts-payable-to-bank reconciliation cases spanning 14 operational tables, including 1,500 injected failures across 15 causal categories and 750 legitimate or hard-negative cases. Root-cause labels and record-level evidence contracts are hidden from the model, allowing retrieval to be evaluated independently of answer correctness. We compare Rules/SQL, classical machine learning, dense semantic retrieval, deterministic relational expansion, and Typed Provenance Graph Retrieval (TPGR), a typed traversal restricted to persisted transaction relationships. Rules/SQL reaches 84.97% held-out exact accuracy and classical ML reaches 95.44%. Holding the reasoning model, prompt, and generation settings fixed while changing only retrieval increases macro required-record recall from 0.83% to 77.70% and exact 16-class accuracy from 2.05% to 72.44%. Structural retrieval failures outnumber reasoning failures with sufficient retrieval by 95 to 15; 254 correct predictions occur despite incomplete retrieval, and strict returned-evidence contract accuracy is only 5.72%. On FinRCA-Bench, retrieval architecture strongly shapes observed AI-system performance, and a correct root-cause label is a weak proxy for an auditable diagnosis.

0.59