CS Paper Radar

CARLAR Influence: A Structured Hallucination Risk in Multimodal Cheque Extraction and an Exploratory Compressed-Image Countermeasure

Joong Ho, Choi, Aditya Paul, Nidhi Singh

cs.CV cs.AI ·submitted 2026-10-19 19:29 UTC · v1

Abstract

Multimodal large language models (LLMs) are increasingly used for structured extraction from financial documents, but their behavior under visual ambiguity is poorly understood. We study CARLAR (Courtesy Amount Recognition and Legal Amount Recognition) influence: a model returns the same amount for both cheque fields even though their annotations differ, including cases with a blank annotation encoded as zero. Across ten visual prompting conditions, we evaluate GPT-4.1 and Claude 4.5 Sonnet on 243 primary cheques (124 with two observed unequal amounts) and 598 strict mismatches from two additional datasets. Under courtesy redaction, GPT-4.1 CARLAR reaches 73.7 percent (95 percent bootstrap interval: 68.3 to 79.0; p below 0.001 versus baseline). Model-reported confidence can remain high when predictions agree; expected calibration error reaches 0.93 among CARLAR cases. We also test a compressed-image few-shot intervention that presents examples as one visual table without model fine-tuning. On the 243-case cohort, Claude's Control condition increases paired joint extraction accuracy by 11.5 percentage points (95 percent interval: 6.2 to 16.9; adjusted p = 0.00054), but its strict-124 gain is smaller and not significant after correction. The intervention's effect on predicted-field agreement is model-dependent: it decreases for Claude and increases for GPT-4.1. These results motivate validation methods that account for visual context and dependencies between extracted fields.

Comment
12 pages

Closest papers

Nearest neighbours by meaning across the whole index.

FiReIQ: Field-Level Confidence Estimation with Reasoning and Image Quality in Document Extraction

cs.CV ·2026-11-13 ·Xiao Liu, Yage Wang, Qi Yu, Giri Iyengar
benchmark similarity 0.62
Abstract · Work in Progress

Vision-Language Models (VLMs) have demonstrated strong performance in structured document extraction, yet their deployment in high-stakes applications requires reliable confidence estimates for individual fields. Existing uncertainty estimators often aggregate signals across an entire generation, obscuring field-specific reasoning uncertainty while overlooking visual degradations that may lead to confident extraction errors. To address these limitations, we introduce FiReIQ, a field-level confidence estimation framework that combines field-aligned reasoning signals from a single VLM generation with local and global image-quality features through quality-conditioned visual fusion. To facilitate systematic evaluation, we further introduce FiReIQ-Bench, a benchmark spanning financial receipts and legal contracts. Experiments with two reasoning VLMs show that FiReIQ consistently improves confidence discrimination, selective prediction, and probabilistic accuracy over existing estimators. These gains extend to private financial service data, where FiReIQ improves AUROC by 6.8% and coverage at 95% precision by 10.6% over an existing baseline. Together, these results highlight the potential of FiReIQ to reduce manual review while enabling more reliable document automation.

0.16

VERDICT: Visual Extraction-Result Decoder with Integrated Confidence and Traceable Evidence

cs.CV cs.LG ·2026-10-07 ·Harikrishnan P M, Goutham Vignesh, Siva Prasad, Zishan Ahmad, Mahesh Basveshwar Deshmukhe +3 more
published similarity 0.62
Abstract · Accepted at NeurIPS 2026, Workshop on VLM4RWD

Enterprises run schema-guided structured field extraction on documents with frontier multimodal models, and accuracy keeps improving as new models and systems arrive. Two things have not kept pace: the evidence for each extracted value, meaning the location of the document it was read from, and how far to trust the value and that evidence. Neither is a component of most extraction systems today, so we build them as a framework that attaches to an existing extractor and leaves it untouched. We present VERDICT, a plug-in framework that uses a small Vision--Language Model (VLM) to ground/localize the evidence once per document over a fixed external extractor's output, then calibrates value and evidence confidence. Fine-tuning the VLM for evidence localization substantially improves visual evidence quality and, more importantly, expands VERDICT's selective-prediction operating range. At a strict confidence threshold $0.80$, VERDICT with the fine-tuned VLM gives usable precision--coverage trade-offs for both value and evidence confidence across the in-distribution and out of distribution datasets, whereas the frozen setting often collapses to little or no coverage. Running an open-weight VLM for localization rather than a closed API makes its internal states available to the calibrator, and they are worth more than output log-probabilities: in-distribution AUROC rises from $0.730$ to $0.914$ for value confidence and from $0.681$ to $0.906$ for evidence confidence. This advantage does not survive a distribution shift. We show that the signal is displaced rather than destroyed, and that labeling roughly $120$ fields from the target dataset takes evidence confidence from $0.641$ to $0.905$ on one dataset and from $0.675$ to $0.829$ on the other.

0.29

Perception, Layout, and Validation: Calibrated Confidence for Reliable Straight-Through Processing of Financial Documents

cs.AI ·2026-09-17 ·Yichao Jin, Yushuo Wang, Yuxuan Han, Kwan Ching Yee Sonia, Weiyang Song +5 more
similarity 0.62
Abstract

Straight-through processing (STP) on extracted key-value fields from financial documents without human review requires a calibrated probability together with a bounded guarantee on the residual error of the auto-approved tier. The emergence of modern Vision Language Models (VLMs) provides an out-of-the-box capability for extracting the key-values, but their verbalized confidence signals are unreliable and weakly track field correctness. This paper introduces a decomposed confidence layer along three interpretable channels, including perception, layout, and validation. Together with a final conformal risk control, the score can be used for reliable STP of financial documents. The method is validated on three public datasets covering real invoices, synthetic invoices, and ad-buy forms, using two different VLM families (Qwen3.6-27B and Gemini-3.1-Flash-Lite). Our decomposed score consistently improves the separation of correct from incorrect extractions, substantially raising the AUROC from 0.54-0.74 for VLM verbalized signals to 0.90-0.99 with contributions from all three designed channels. Crucially for industrial deployment, this enables usable STP. The native VLM confidence signals could clear only 0.1%-7.0% of fields under risk control at a target error of <10%. In contrast, the proposed method auto-approves 49-72% of fields while holding the empirical error of the accepted tier at or below the target.

0.37

Frontiers in FinTech: Multimodal Foundation Models for Financial Reporting and Decision Science

cs.CE ·2026-08-24 ·Yulu Huang, Niannian Yu, Yaxin Yang, Yong Huang
similarity 0.60
Abstract

Financial information no longer arrives in a single format. Research reports come as PDFs, financial statements live in spreadsheets, market trends are captured in images, and policy documents reach analysts as scans, each carrying part of the picture the others cannot supply. Accounting information systems built around single-modality extraction pipelines and rule-based tools therefore struggle to assemble the full picture, slowing financial statement analysis, complicating audit evidence corroboration, and limiting investment decision support. This study presents FinVision, a multimodal large language model that unites vision-language models with domain-specific financial reasoning. Instead of processing documents in isolation, FinVision reads text, tables, and images together, converts them into consistent structured data, and verifies cross-modal agreement, in the same spirit as auditors corroborating evidence from independent sources. The model is trained in two stages, pre-trained on large-scale public financial corpora and fine-tuned on institution-specific investment data, so it can apply established valuation methodologies and audit risk assessment frameworks while outperforming zero-shot and single-stage baselines. A natural-language decision pipeline lets users describe what they need and turns those descriptions into executable workflows, supporting portfolio optimization, real-time risk monitoring, and refinement through multi-turn dialogue. Across 200 listed companies, FinVision reduced valuation error by 19 percent relative to the strongest baseline; a user study with 48 accounting and investment professionals reported a 51 percent reduction in task completion time. These results carry implications for audit automation, financial reporting quality, and more inclusive access to expert-level financial analysis.

0.08

CheXPoT: Post-Hoc Visual Verification and Correction of VLM-Generated Radiology Reports

cs.CL cs.CV ·2026-10-31 ·Joey Guo, Rayan Ansari, Alaa Youssef
similarity 0.60
Abstract · 15 pages, 12 figures

Vision-language models (VLMs) have shown promise for chest radiograph report generation, but adoption remains limited because of hallucinated findings that are not grounded in the image. Correcting these hallucinations and improving overall report quality remains a significant challenge. Prior work has addressed this problem by refining models with reinforcement learning, preference optimization, or chain-of-thought supervision, often at substantial compute cost. To address this gap, we introduce CheXPoT, a program-of-thought (PoT) framework that verifies each generated finding against image features and removes those it cannot ground. CheXPoT decomposes generated reports into predicates with RadGraph and an LLM, grounds these predicates against a DenseNet-121 and Grad-CAM visual evidence table, executes PoT verification code, and rewrites reports after removing unsupported predicates. On a 4,145-image held-out split, CheXPoT yields statistically significant precision gains on 8 of 14 CheXpert classes and F1 gains on 5, with no significant decreases in either metric. CheXPoT lifts micro-average precision from 19.2% (MedGemma-4B) to 46.8% (MedGemma-4B + CheXPoT) and micro-average F1 from 23.0% to 49.4%, absolute gains of 27.6% and 26.4% respectively. We also tested CheXPoT on other VLM report generators. On LLaVA-Med, CheXPoT lifts micro-average precision from 4.5% to 17.7% and micro-average F1 from 6.3% to 19.4%. On MAIRA-2, CheXPoT lifts micro-average precision from 15.6% to 42.1% and micro-average F1 from 21.0% to 45.3%. Our results suggest that inexpensive post-hoc visual verification may improve VLM trustworthiness by correcting unsupported findings from generated reports. CheXPoT lays a foundation for tools that help validate VLM-generated radiology reports and reduce unsupported findings. These innovations may subsequently help increase trust and adoption of VLM tools in Medicine.

0.18

When Figures Disagree with Text: Evaluating Autonomous Visual Evidence Verification in Scientific Papers

cs.AI ·2026-11-18 ·Guijia Zhang, Hongbo Qiu, Jiajun Zha, Guohao Cai, Hao Zheng +8 more
similarity 0.58
Abstract · 28 pages, 5 figures, 14 tables

Scientific figures carry evidence for the claims that cite them, so reviewing a paper means checking that a figure agrees with the text, and not only describing what the figure shows. Benchmarks of figure-text verification typically hand the model the figure, the claim, or the knowledge that an inconsistency exists, which leaves untested whether a model can find the figure worth checking in a complete paper, relate a visual element to the claim it conflicts with, and recognize the conflict. We study this autonomous setting on 31 full manuscripts, each carrying a reviewer-documented figure-text inconsistency from PRISMM-Bench that is hidden from the model. Six multimodal models review each PDF under a neutral prompt, and we then add one input at a time to the same manuscripts. Of 1,445 open-ended reviews, 60.6% mention the flawed figure, yet a grading rubric accepts only 3.3% as diagnosing its contradiction; a blind audit finds the rubric lenient, so we read these as rubric-accepted rates and compare conditions rather than absolute values. In a separate controlled run, naming the figure, without stating that it is flawed, raises rubric-accepted diagnosis from 4.3% to 20.4%, while a consistency instruction and coarse visual cues do not. When the contradiction is visible only in the rendered figure, text-only review yields no accepted diagnosis, while review of the rendered pages with the figure named reaches 15.8%. A baseline that lets the model choose which figures to check, ranked by centrality, stays at 4.8%. Deciding what to verify is thus a large part of the difficulty, and the selection rule we tested does not resolve it. Multimodal document understanding should be evaluated on whether a model can propose what to verify, not only answer questions about a figure it is handed.

0.27

Read, Then Shade? How Visual Evidence Changes Incentive-Driven Misreporting in Vision-Language Advisors

cs.LG cs.AIcs.CV ·2026-11-14 ·Hamidreza Hasani Balyani
similarity 0.57
Abstract · 45 pages, 5 figures. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after w…

Vision-language models (VLMs) increasingly read a scene and report it to someone who acts on the report, sometimes under objectives that differ from that person's. We ask whether a stated incentive moves what a VLM reports, and whether this changes when the value must be read from pixels rather than given as text. In seeded synthetic scenes, a model reads the position of a target on a labelled axis and reports one integer to a fixed receiver; it is paid, in simulation, for accuracy or for a report a few units to one side, and every scene is also presented with the true value as text. Across a pilot and three follow-up studies with hash-frozen protocols, four small open VLMs from three families, and 15,170 held-out trials, a stated payoff moved reports in both modalities, and exact compliance at three units was significantly lower with the image than with text for three of four models; Qwen2.5-VL 7B showed no reliable difference. Adding the true value as text to the image prompt removed the gap in every model. For InternVL3 the gap vanished on scenes it read correctly when it wrote its reading first; for Gemma 3 it persisted. For Qwen2.5-VL 3B, which read every scene correctly, the gap held in four of five payoff paraphrases, nearly vanished when the model wrote its reading before its report or when the prompt stated that the client wants the truth, vanished with an unpaid goal, and persisted when the model reported before stating its reading; how often it shaded at all, even with text, depended strongly on the wording. When told that the client wants the true value, three of four models still shaded every text report. Visual input did not make these advisors reliably more truthful; it made compliance with a stated incentive depend on whether the read value is made explicit and on output order, a dependence that text-only evaluations and mean-shift metrics do not reveal.

0.23

Mandela-Bench: Multimodal Models Remember Canonical Images Instead of Seeing Them

cs.AI ·2026-09-26 ·Yicheng Bao, Zhenkun Gao, Xiahui Guo, Mingqian Yang, Xueheng Li +5 more
similarity 0.56
Abstract

Historical photographs and other canonical images can now be edited seamlessly with a single instruction, often leaving no reliable pixel-level trace. In such cases, the only evidence of manipulation may be a fact about what the image depicts. Existing benchmarks instead rely on generator artefacts, image-caption inconsistencies, visual implausibilities, or external references, and therefore do not test whether a model can use its own world knowledge to verify a recognized image. We introduce Mandela-Bench, containing 1,507 edits of canonical images: 1,359 knowledge-only forgeries, each contradicting one verifiable fact, and 148 anchor-free controls that preserve the editing process without introducing a factual contradiction, together with 474 untouched originals. We score not only whether a model detects a forgery, but whether its explanation identifies the inserted entity or the fact being violated. Across 36 multimodal models, from 0.8B parameters to frontier scale, we find a consistent failure mode. When a public figure is removed from a familiar photograph, models still name that person in up to 72.7% of responses. Some models can distinguish the replacement face from the original when shown in isolation, yet still judge the full edited photograph as authentic. Providing the true event and date does not improve knowledge-grounded detection, whereas providing the same information after cropping away the recognizable composition does. Even under explicit verification prompts, only one of the 36 models meets the KGR criterion on at least half of the forged images. These results suggest that the failures cannot be explained by missing knowledge or inadequate perception alone. Instead, they are consistent with recognition biasing verification toward the remembered canonical image rather than the observed edit.

0.29