CS Paper Radar

Question-Guided Evidence Acquisition for Multimodal Visual Question Answering

Alin-Ionut Popa

cs.CV cs.AIcs.LG ·submitted 2026-08-20 07:37 UTC · v2

Abstract

Multimodal LLMs can see a document, but they often can't read it reliably. Small text, tables, visual cues, and topological elements still trip them up under direct visual inference, even when the page is already sitting in the model's context. Most document-VQA systems treat perception as fixed: they encode the page once, ask the question, and answer from whatever the model happened to extract in that single fast pass. We think document VQA needs slower, more deliberate perception: rather than answering from one fixed encoding, the model should spend a bit of extra compute at inference time working out what to look at next, and only then answer. We build this into \textbf{Q-Guide}, a small agent that reads a question, works out what evidence it is still missing, and calls targeted tool(s) to recover it---reading text where text is needed, zooming in where detail is needed, or grounding a region where position matters. On DocVQA2026 and Manga109, Q-Guide outperforms both direct prompting and recent multi-agent document systems ($65.0\%$ vs.\ $40.0\%$ on DocVQA2026, $32.4\%$ vs.\ $24.4\%$ on Manga109), and the improvement holds across three Claude backbones (Opus 4.6, Sonnet 4.6, and Opus 4.5). We find that accuracy scales with the perception budget---most of the gain appears within two to three deliberate rounds---and that the gain comes from directing perception to the right place, not from complex control logic: adding planners, routers, or multiple collaborating agents does not help.

Closest papers

Nearest neighbours by meaning across the whole index.

Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?

cs.CV ·2026-10-12 ·Aneerban Saha, Vijay Kumar, Sameer Madhav D., Priyam Srivastava, Dhruv Bansal
publishedbenchmark similarity 0.73
Abstract · 13 pages, 6 figures. Accepted at ADCIS 2026

Some questions about an image need commonsense or factual knowledge that its pixels do not contain. Large Language Models (LLMs) can supply this knowledge to Visual Question Answering (VQA) systems, but the depth at which their guidance should enter a vision-language model remains unclear. We prompt the 70-billion-parameter Llama~3.3 model to describe, for each question, the visual concepts to focus on, the distractors to ignore and the external facts needed. We inject this guidance into a frozen Vision Transformer (ViT) paired with an instruction-tuned Text-to-Text Transfer Transformer (T5) in two ways: decision-level late fusion and attention-matrix injection. On the Outside Knowledge VQA (OK-VQA) benchmark, and contrary to the common view that deeper is better, the shallow route wins: late fusion reaches 39.4% exact match, +10.0 points above an unguided baseline, while attention injection reaches only 31.8% under data-scarce fine-tuning. An LLM paraphrasing pipeline also enlarges the training set more than tenfold. The results suggest that integration depth should be matched to the data regime.

0.43

DoubleTake: Focused Verification of Visual Evidence for Long-Document Understanding

cs.CV ·2026-11-13 ·Kejun Xue, Ang Li, Jinghua Tang, Xincheng Wang, Yongbin Gao +2 more
code/data similarity 0.73
Abstract

In long-document question answering, a vision-language model can answer incorrectly even when the retrieved pages contain sufficient evidence. Rereading those pages with the original question gives the model little guidance about which fact to check. We introduce DoubleTake, a training-free framework for focused verification of visual evidence. An initial reading produces a candidate answer and a separate evidence record that guides retrieval for a second reading. A planner compares the two candidates and their supporting evidence to formulate a factual question. A verifier answers it from the source images without seeing either candidate or evidence record, and a reasoner uses this observation to resolve the original question. With a frozen Qwen3-VL-8B-Instruct backbone, DoubleTake achieves the highest overall accuracy among the evaluated methods on each of four benchmarks. With the same pages and number of model calls, asking a focused factual question yields higher accuracy on all four benchmarks than asking the original question again. Our code is available at https://github.com/small-code-cat/double_take.

0.21

Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA

cs.AI ·2026-09-18 ·Nikhil Reddy Pottanigari, Sepideh Kharaghani, Saverio Vadacchino, Alejandro Posada, Kurt MacDonald +1 more
similarity 0.73
Abstract · Accepted at the Context Beyond the Window (CBW) Workshop at COLM 2026 and the DocInsights Workshop at EMNLP 2026

Every document QA system begins with a choice that is rarely studied on its own: whether to feed the model page images, extracted text, or both. We isolate this choice, holding the prompt, judge, and scoring pipeline fixed, across four commercial model endpoints, two corpora, and two context regimes (gold evidence pages and the full document). On documents that fit the image budget, page images lead on accuracy at every document length on both corpora, but this advantage carries a growing latency and cost premium: text latency stays roughly flat as documents lengthen while image latency rises steadily. Text and images also fail on different questions, with exactly one representation correct on 19--25% of items across the reported cells, so neither subsumes the other. Exploiting this complementarity, a lightweight TF-IDF router that reads only the question text gains 2.6 points over always-text while cutting median latency 30% relative to always-vision, on a document-disjoint held-out split.

0.17

The Answer Knows Where to Look: Zooming before MLLMs Answer under Visual-Token Budgets

cs.CV ·2026-10-23 ·Zhensheng Hu, Guanglong Xu, Guanqiang Hu
similarity 0.73
Abstract

Multimodal large language models (MLLMs) read images through visual tokens, and under the small per-image budgets that efficient deployment requires, the fine text of documents, charts and infographics becomes illegible. Zooming in recovers the evidence, but the region must be chosen before the model answers. We find that MLLMs know where the evidence is, but not in a form that can be read before they answer: the answer tokens localize it from the middle layers on, whereas the attention available before answering does so only in the top layers, at a depth that varies across models, and diffusely. Supervising the pre-answer attention to point degrades reading, and the benefit of zooming is governed by each image's information gap. Building on these findings, we introduce ours, which lets an MLLM look where its answer would. A dedicated look token, processed in a single prefill and kept out of the answering pass, learns in hindsight to attend as the model's own correct answers do, extracting this knowledge into a concentrated pointer before any answer exists. Its hidden state then selects how to look, from no zoom to crops, square windows, windows that adapt to the pointer and foveated warps, supervised by the counterfactual outcome of every view. ours needs no box annotations, per-model layer search, external localizers or generation before the answer. On Qwen3-VL at three scales and five text-rich benchmarks, it surpasses the strongest baseline by 4.8-9.1 points at 98 visual tokens per view and attention-guided zooming by 11.1-17.0 points, while running faster than attention-guided and grounding-based zooming.

0.32

Found but Not Read: When Extracted Text Closes the Retrieval-Reading Gap in Document Vision-Language Models

cs.CV ·2026-10-02 ·Qingtao Xia, Siyao Cheng, Jiahua Bao, Jiaxing Du, Jie Liu
benchmarkcode/data similarity 0.71
Abstract · 5 pages, 5 figures, 3 tables. Submitted to ICASSP 2027

Retrieval-augmented document question answering assumes that once the right page is found, a vision-language model (VLM) can read it. We show that this assumption often fails, leaving a retrieval-reading gap: evidence found but not used. A paired protocol isolates this gap by comparing answers from the retrieved page images alone with answers from the same images plus their extracted text. On FoveDoc-Bench, our benchmark with traceable evidence, retrieval finds nearly every evidence page, yet adding CPU-OCR text raises strict accuracy by 13 to 16 points. An exact text layer roughly doubles the gain, which appears across six VLMs from three families and, within one family, narrows with scale without closing. The reader can read this evidence but cannot find it: crops of it recover most of the text gain, boxes around it on the page none. The same protocol identifies two boundaries. Extracted text helps on textual evidence but is neutral or harmful on charts and figures. Its advantage shrinks as retrieval degrades, and unrelated text of the same form adds nothing detectable. Extracted text is an amplifier of retrieval that works, not a substitute for retrieval that does not. Our code is available at https://github.com/atoz03/fovedoc-sup.

0.49

VLMs Read Aloud: The Gap Between Recognizing and Using Visual Text

cs.CV cs.CL ·2026-11-01 ·Zhiyun Shi
similarity 0.71
Abstract · 23 pages, 7 figures, 21 tables

Visual-text compression renders long contexts as images and is engineered around how much of the text a vision-language model (VLM) can recover. Yet recovered text is not usable text. VLMs can transcribe the text in an image word for word and still fail to answer from it; given back their own transcript, they answer nearly as well as from the original text. We call this difference the read-use gap. Recognizing and using visual text have different resolution requirements: Qwen3-VL-8B's recognition saturates at 15-pixel characters while use catches up only at 24 pixels, and scaling from 2B to 32B raises recognition without narrowing the gap. Putting the evidence into words closes it: thinking VLMs recite the evidence in their thinking, and their own transcript, written without seeing the question, recovers most of what thinking adds. In a preregistered held-out test, the gap ranks how much eight VLMs gain from their own words over pixels (Spearman $ρ$ = 0.86), which recognition rates do not. We propose ALOUD, which spends visual resolution on finding and recognizing evidence and language tokens on using it. Without keeping the source text, it doubles the accuracy of a frozen Qwen3-VL-8B on long documents compressed 5x and 10x: at 5x, from 27.7% to 54.8%, near the original text's 58.5%, with a context smaller than the compressed pages themselves. For less than the cost of reading the original text, it leads at equal budget every method that does without it, at every level, and beats AGAR and VERA, which paste the original text back; at 5x it also beats LensVLM-9B, which is trained and keeps the original text. Visual-text compression is limited not by how much text a model can recover, but by how much it can use.

0.22

Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMs

cs.CV ·2026-09-22 ·Dingyang Lin, Yingfeng Luo, Chenglong Wang, Chenwei Zhu, Anxiang Ma +2 more
similarity 0.69
Abstract · 6 pages, 3 figures, and 6 tables. Preprint

Vision-language models (VLMs) have achieved strong performance on tasks such as visual question answering, yet small image resizes can turn correct answers into errors. We investigate whether changes in visual configuration, such as image tiling and token arrangement, contribute to this instability. Across seven checkpoints and four benchmarks, equally small resizes cause more correctness flips when they switch configurations. Surprisingly, in over half of these cases, models answer the question incorrectly but can still read the correct answer when told what to read. Furthermore, attention interventions in LLaVA-NeXT suggest that configuration changes can weaken the use of readable information during answering. We therefore guide models using field cues and their own transcriptions. With annotation assistance, these forms of guidance together correct 97.2% of errors with readable information. These findings show that configuration changes can affect how models use information they can still read.

0.42

InSight-doc: Agentic Visual Perception for Long-Document Understanding

cs.CV cs.CLcs.LG ·2026-08-11 ·Kaican Li, Weiyan Xie, Lewei Yao, Jiannan Wu, Lanqing Hong +2 more
top venuecode/data similarity 0.69
Abstract

Long-document understanding often requires reasoning over many visually rich pages, making inference costly and prone to context rot. In this work, we propose InSight-doc, an agentic visual perception framework that treats visual resolution as an adaptive reasoning-time resource. InSight-doc starts from low resolution and selectively zooms into high-resolution regions for finer evidence, without relying on any external retriever. To train such an agent, we construct an active-perception corpus of 17.9K high-quality SFT examples with region-level zoom-in trajectories, accompanied by 19.2K hard RL examples. Through SFT+RL, InSight-doc-8B improves the baseline by 4.3--16.4 accuracy points over document VQA benchmarks. On long documents, it reduces hallucination by more than 40% and inference latency by 41%--68% while maintaining an accuracy lead. Our code, datasets, and model are released at https://github.com/m-Just/InSight-doc .

0.89