CS Paper Radar

Reading Is Not Using: Retrieval, Judgment, and the Design of AI Financial Research Workflows

Miao Liu, Zhizhe Liu

cs.CL cs.AI ·submitted 2026-08-25 17:31 UTC · v1

Abstract

Large language models (LLMs) are increasingly deployed as AI analysts to process financial disclosures and support AI-assisted investment decisions. Yet such systems are usually evaluated by what they can retrieve, not whether retrieved information affects their judgments. We identify a retrieval-integration gap in long-context financial analysis. Holding focal-firm information fixed and varying only unrelated context from 2,000 to 128,000 tokens, we find that a risk disclosure's influence on investment judgments falls to the experimental noise floor even as direct retrieval remains accurate. The pattern replicates across model families and judgment tasks and in experiments removing real disclosures from actual 10-K filings. More capable models postpone but do not eliminate the gap. Causal memory interventions show that compressed summaries and source-text lookup jointly transmit disclosures into judgments. Workflow architecture determines whether this transmission succeeds: chunk-and-summarize pipelines evict relevant information, whereas a targeted, structured restatement adjacent to the decision restores its influence. AI analyst performance is therefore jointly determined by model capability and workflow architecture. Retrieval-based evaluations can certify systems whose investment judgments ignore information they demonstrably retrieved.

Closest papers

Nearest neighbours by meaning across the whole index.

The Analyst in the Prompt: Role, Retrieval, and Memory Biases in LLM Financial Analysis

cs.CL cs.CEq-fin.PM ·2026-09-02 ·Ahmed Asaad, Amr Mohamed, Yang Zhang, Omneya Abdelsalam
top venue similarity 0.72
Abstract

Large Language Models (LLMs) increasingly use user context such as memory, profiles, and role prompts to personalize their responses. This personalization can affect evidence-based judgment: the same evidence may lead to different conclusions under different user contexts. Finance provides a high-stakes setting to study this problem because decisions often depend on interpreting long and complex documents. We test this using 3,575 SEC filings across twelve LLMs. We compare persona-conditioned retrieval, neutral retrieval, and memory-framed context to separate the effect of evidence selection from the effect of interpretation. We find that most user-context spillover comes from how models interpret the same evidence under different roles, rather than from retrieving different evidence. We then test two simple mitigation strategies: expressing the same investor mindset as a user profile instead of an assistant role, and separating evidence-based and personalized outputs. Both reduce spillover, but neither removes it completely, and their effectiveness varies substantially across models.

0.57

BuySideFlow: A Point-in-Time Benchmark for LLM Agents in Investment Research

cs.AI ·2026-11-07 ·Dong Bo, Fanjin Zhang, Ke Cheng, Xinxin Zhao, Xiaowei Su +5 more
benchmark similarity 0.69
Abstract

Large language models (LLMs) can generate SQL and SQL--Python workflows for investment research, but ensuring point-in-time (PIT) validity remains challenging. Agents must enforce disclosure cutoffs and select the appropriate historical states and record versions throughout data retrieval and financial analysis. We introduce \textsc{BuySideFlow}, a benchmark of 408 practitioner-derived Chinese-language tasks spanning equities, funds, fixed income, and macroeconomics over a fixed snapshot from a licensed production financial database. We provide executable references and a taxonomy of temporal errors, pairing result checks with expert audits of executed workflows. We evaluate nine agent configurations on all tasks, with result accuracy ranging from 16.2\% to 36.8\%. An audit of 900 runs on 100 shared tasks confirms PIT violations in 193 of the 380 runs that pass result checks (50.8\%), showing that output agreement alone does not establish temporal validity.

0.23

Beyond LLM Wrappers: Context Engineering and the Three-Engine Architecture for Reliable Financial AI

cs.AI ·2026-10-29 ·Pradeep Kumar GS, Ravi Tanniru
position similarity 0.67
Abstract

Financial AI systems must support decision-bearing questions involving payments, collections, cash flow, and compliance while ensuring numerical accuracy, traceability, and auditability. Large language models (LLMs) enable natural language interaction but cannot independently guarantee reproducible calculations or evidence-grounded recommendations. We present a production-grounded architecture integrating three components: a financial knowledge graph for enterprise relationships, temporal context, and provenance; a deterministic optimization engine for financial calculations and decisions; and a multi-agent LLM layer for intent interpretation, task orchestration, explanation, and claim validation. A coordination layer, termed context engineering, selects, ranks, composes, and validates query-specific context across ERP systems, tools, and tenants. Based on observations from 15 deployments over approximately 11 months, single-agent pipelines exhibited hallucination rates of about 7-9% (5-10% across query types), including invented invoices and unsupported financial claims. Graph-backed multi-agent validation reduced observed rates to below 1%, although these findings are observational rather than causal. ERP data quality and tenant-specific schemas remained major implementation challenges. Reusable mappings, graph templates, and validation policies coincided with a reduction in deployment time from roughly six to two weeks. We argue that financial context engineering extends beyond text-centric retrieval-augmented generation by requiring structured relationships, temporal validity, exact numerical evidence, access controls, and validation against source records and solver outputs. We outline research challenges in evaluation, reproducibility, privacy, context quality, and cross-tenant generalization.

0.12

FinancialAuditBench: Benchmark Construction under Differential Privacy Using Real-World Priors

cs.AI ·2026-09-26 ·Jerry Huang, Sarvesh Babu, Matt Van Buren, Alexander Wang, Pranav Pillai +3 more
benchmark similarity 0.67
Abstract

As AI agents are becoming widely adopted in the financial services industry, careful measurement is essential to understand where they can be reliably deployed and where oversight and professional review remain necessary. Such measurement, however, is constrained by limited access to proprietary or privacy-sensitive data. Existing benchmarks therefore often rely on publicly available data, human- and/or LLM-authored tasks, or simplified settings. We introduce FinancialAuditBench, a benchmark for evaluating agents on financial statement audit tasks, along with a framework for systematically generating synthetic engagements. Our task generation framework leverages differentially private aggregate statistics from historical audits along with audit expertise contributed through over 1,100 hours of benchmark development and review. FinancialAuditBench consists of 90 tasks spanning workpaper completion and review across six synthetic audit engagements, each containing an average of 179 files. Evaluation on eleven frontier models shows that while agents complete substantial portions of staff-level audit tasks well, they sometimes perform inappropriate procedures or produce incorrect documentation. Beyond financial auditing, our framework offers an approach for systematically generating synthetic tasks for model evaluation and training in privacy-sensitive domains.

0.13

FinLLM-KG: Knowledge Graph-Enhanced LLMs for Financial Sentiment Analysis in Algorithmic Trading

cs.CL cs.LGq-fin.STq-fin.TR ·2026-10-26 ·Giorgos Iacovides, Oliver Hannibal, Wuyang Zhou, Danilo Mandic
similarity 0.67
Abstract

Recent advances in Generative AI have substantially improved financial sentiment analysis through post-trained financial large language models (LLMs). However, existing approaches remain confined to predicting the sentiment of each article in isolation and operate as black boxes, which limits contextual understanding and hinders deployment in high-stakes financial decision-making. To this end, we introduce FinLLM-KG, a model-agnostic framework that combines causal LLMs with a financial knowledge graph, in order to inject temporally grounded, company-level knowledge from SEC 10-K filings at inference time, without updating model weights. Within FinLLM-KG, context is supplied either as a natural-language company profile or as a small set of relevance-ranked factual triples. This enables sentiment predictions to account for broader firm-level information rather than the article alone, at almost zero computational cost and for any causal LLM. In addition, attributing each article-level sentiment score to individual sentences and to the retrieved knowledge-graph facts makes the evidence behind the prediction inspectable, thereby enhancing explainability and supporting human-in-the-loop review. Simulations demonstrate that FinLLM-KG consistently improves portfolio performance over the state-of-the-art baseline.

0.27

FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review

cs.AI cs.CL ·2026-08-26 ·Suyang Zhong, Jingzhe Zhu, Qi Xu, Liyao Sun, Yin Wang +3 more
benchmark similarity 0.66
Abstract

Deploying large language models for professional financial review requires more than measuring general financial competence: models must perform the specific review operation required by a workflow and determine whether available evidence is sufficient for a defensible decision. Existing financial benchmarks cover knowledge, reasoning, compliance, and professional tasks, but their evaluation units are often organized around datasets or task formulations rather than the decisions that deployed systems support. We introduce FinRiskAtlas, a Chinese-language benchmark that evaluates financial LLMs along two complementary dimensions: operation execution under fixed evidence states and evidence-state control under evolving review conditions. The static benchmark contains 9,742 instances across 53 task families, including 42 Domain Knowledge families and eleven downstream review operations defined by explicit evaluation contracts. FinRisk-Ask extends this framework through offline replay of 680 pre-action states from 104 de-identified professional trajectories, withholding future evidence during inference and using it only to construct expert-verified evidence targets. Across 33 model configurations, operation-level evaluation yields non-redundant rankings (mean pairwise Spearman correlation 0.42 across downstream operations), and knowledge-based shortlisting can incur up to 18.01 points of regret on individual operations. FinRisk-Ask further shows that entering the Ask branch more frequently does not necessarily improve request targeting or end-to-end evidence acquisition. These results show that broad financial capability scores do not fully capture where models are reliable in professional workflows, motivating evaluation units aligned with the decisions and evidence states that deployed systems must support.

0.44

Frontiers in FinTech: Multimodal Foundation Models for Financial Reporting and Decision Science

cs.CE ·2026-08-24 ·Yulu Huang, Niannian Yu, Yaxin Yang, Yong Huang
similarity 0.65
Abstract

Financial information no longer arrives in a single format. Research reports come as PDFs, financial statements live in spreadsheets, market trends are captured in images, and policy documents reach analysts as scans, each carrying part of the picture the others cannot supply. Accounting information systems built around single-modality extraction pipelines and rule-based tools therefore struggle to assemble the full picture, slowing financial statement analysis, complicating audit evidence corroboration, and limiting investment decision support. This study presents FinVision, a multimodal large language model that unites vision-language models with domain-specific financial reasoning. Instead of processing documents in isolation, FinVision reads text, tables, and images together, converts them into consistent structured data, and verifies cross-modal agreement, in the same spirit as auditors corroborating evidence from independent sources. The model is trained in two stages, pre-trained on large-scale public financial corpora and fine-tuned on institution-specific investment data, so it can apply established valuation methodologies and audit risk assessment frameworks while outperforming zero-shot and single-stage baselines. A natural-language decision pipeline lets users describe what they need and turns those descriptions into executable workflows, supporting portfolio optimization, real-time risk monitoring, and refinement through multi-turn dialogue. Across 200 listed companies, FinVision reduced valuation error by 19 percent relative to the strongest baseline; a user study with 48 accounting and investment professionals reported a 51 percent reduction in task completion time. These results carry implications for audit automation, financial reporting quality, and more inclusive access to expert-level financial analysis.

0.08

DealsFlow: An Evidence-Gated Agentic Harness for Venture Capital Deal Triage

cs.AI ·2026-10-23 ·Ming Ze Tang, David Cutting, Boyd Carson, Vasiliki Carson
similarity 0.65
Abstract · 13 pages, 15 tables, 2 figures. Submitted to the AI4F 2026 workshop (LLMs and Generative AI for Finance) at ICAIF 2026

Large language models (LLMs) can read startup pitch decks, but letting them emit investment scores directly invites score inflation, run-to-run variance and rationales that are hard to audit. We describe DealsFlow, a human-in-the-loop agentic harness deployed at a UK early-stage fund. The LLM reads the deck, cites evidence and rates each category; code combines the ratings, caps them and only ever lowers them. Three mechanisms implement this: a stage-aware corroboration ceiling (78, 70 or 60 out of 100) on factual categories without external evidence; a sceptic call that may lower but never raise a score; and a pre-fetch pattern that retrieves and indexes all evidence before a single generation pass. We evaluate on $N=243$ SEIS/EIS investments with recorded outcomes. Because recent investments have unresolved outcomes, we use the 142 that are at least three years old. There, the full pipeline's score has Spearman $ρ=0.45$ (95% CI 0.32 to 0.57) with the total-value multiple and AUC 0.71 for identifying write-offs, against $ρ=0.38$ and AUC 0.68 for an unconstrained LLM. The gain is small and comes mostly from the sceptic call; we did not test whether its lower-only rule matters. Investments that later exceeded $2\times$ had 64% of factual categories corroborated, against 38% for write-offs (omnibus Kruskal-Wallis test over five outcome groups, $p=0.031$), and our retrieval was not date-restricted, so these figures may overstate screening skill. In a study of 20 inbound applications, analyst time per application fell from 44.7 to 18.9 minutes including verification and correction, and 84% of 240 cited claims were fully supported by their sources, but 70% of reports needed at least one correction.

0.23