CS Paper Radar

Usable String Matching by Humans through Overlaid Presentation

Enka Blanchard, Ted Selker, Alan T. Sherman, Levi Gabasova, Mehdi Bouhamidi, Ryan Robucci

cs.CR cs.HC ·submitted 2026-10-29 00:24 UTC · v1
published

Abstract

We explore the ability of humans to compare strings, an error-prone activity which occurs in critical contexts (e.g., bank wire transfers), where it is not always possible to trust a machine to help. We hypothesize that understanding human cognitive and perceptual limitations can improve how people verify whether two character strings are identical. We vary the strings' relative positions and presentation formats to measure comparison speed and accuracy. We present results from two online experiments with 178 human subjects who compared pairs of strings to determine whether the two strings were identical. The strings varied by length (5 characters to small paragraphs), difference (matching versus making a random change or a hard-to-detect one), alignment (vertical versus horizontal), and presentation (Overlay versus Adjacent). Our experiments test whether, by encoding and memory tasks into perceptual tasks, overlay formats can offer advantages in speed and accuracy. Results show that overlay offers a substantially faster method, with improved accuracy for long strings. Nevertheless, mean error rates were at least 4 percent, indicating that even the improved approach may be insufficiently accurate for many verification tasks.

Comment
A shorter version of this paper (without the appendices) has been conditionally accepted to the International Journal of Human-Computer Studies (IJHCS)

Closest papers

Nearest neighbours by meaning across the whole index.

REEL: Same Frames, Different Answers and Different Costs

cs.CV ·2026-10-28 ·Xin Liang, Jiangpeng He
published similarity 0.58
Abstract · Accepted at the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2027, Datasets and Evaluation Track

Video understanding keeps moving toward longer recordings, and the longer a recording is, the more questions can be asked about it. But processing it again for every question does not scale, so many systems watch the video once, build a memory, and answer later questions from it. Yet what to remember is decided before anyone knows what will be asked, and whatever is left out cannot be recovered later. Moreover, whether the savings pay off is judged by token counts, which fall when the input gets shorter but not when the model reuses what it has already computed. We present REEL, an evaluation framework that compares three saved representations built from exactly the same observed frames, so that a change in answers can come only from what the system remembered. Text writes a short record of what happened, Visual keeps the encoder outputs, and Prefix State also keeps the language-model prefix. Each is compared with full reprocessing (Frames) by paired correctness changes, cumulative GPU time, and storage size. Across six vision-language models and two benchmarks, Text and Frames differ in correctness on 20\% to 29\% of questions, even when their mean accuracy is almost the same. By contrast, Visual and Prefix State stay much closer to Frames. With unlimited storage, Prefix State is the cheapest in total GPU time among memories that keep almost all of the answers Frames gets right, for every model on both benchmarks. The trade-off is storage: Prefix State keeps up to about 21 GiB per video, whereas Text needs only kilobytes. Reusing computation and discarding information both look like efficiency, but only the second removes evidence that later questions may need, and neither mean accuracy nor token count tells them apart. REEL makes that difference measurable for any system that answers many questions from one saved memory.

0.33

Practice in Text, Act in Pixels: Correcting Where Perception Fails

cs.CV ·2026-11-17 ·Zhongling Xu, Hairong Wang
code/data similarity 0.57
Abstract · 25 pages, 8 figures, 19 tables. Code: https://github.com/Carlxuzhl/practice-in-text ; aligned reader: https://huggingface.co/xuzhl/practice-…

GUI agents can learn cheaply from a text view of an interface and act on screenshots through a vision-language screen reader. Yet on a simulated mobile platform with a low-cost reader, policies that complete most tasks in text fail on many of the same tasks on screenshots. We ask where a small budget of real screenshots is best spent to close this gap. We find that the reader's misreads are systematic and recur on the same interface elements. Noise matched to the reader's error rates, a natural sim-to-real remedy, misses this structure, and in a pre-specified comparison it did not beat training the policy directly on screenshots. We therefore correct the misreads at their source with reader alignment, which fine-tunes the reader on screenshots paired with their text views. The reader learns to output each screen's text view, as in text practice, and the policy stays unchanged. Spent on the reader, the screenshot budget gives 80.9% full-task success on new task instances, against 55.6% when spent on training the policy. Given the same labeled screenshots, adapting the reader also beats adapting the policy, and a single aligned reader improves every text-trained policy we tried without retraining any of them.

0.45

From Retrieval to Recognition:How Vision--Language Models Become OCR Specialists

cs.CV ·2026-09-18 ·Yuanxiang Huangfu, Hanmeng Zhong, Linqing Chen, Jeffrey Tiong Jee Hui
similarity 0.54
Abstract

Does a general vision--language model acquire specialized OCR ability by developing a new reading circuit or by reusing an existing mechanism? We address this question in the setting of full-sequence OCR, rather than local-answer retrieval. Using an evidence-grounded protocol with held-out causal interventions, we identify sparse and stable OCR-head sets in GLM-OCR, MinerU2.5, and PaddleOCR-VL-1.6. We then investigate the mechanistic origin of these OCR heads by comparing them with independently identified textual retrieval/copy heads in general VLMs. Across two general VLMs, visual OCR heads strongly overlap independently identified textual retrieval/copy heads, yielding untuned top-20 intersections of 73.3% and all-head Spearman correlations of 0.677-0.886. The overlap and causal interventions suggest that full-sequence OCR operates as dense sequential multimodal copy-and-paste, repeatedly retrieving visual evidence and routing it to the current output position. Finally, we examine how this shared circuit changes as a general VLM becomes an OCR specialist. Matched base-to-specialized comparisons show that OCR specialization largely preserves head identity, retaining 17-20 of the top 20 heads per task with all-head rank correlations of 0.874-0.942, while redistributing their functional and causal strengths.

0.18

Visual Embellishments are Potential Distractions in Double-Column Reading

cs.HC ·2026-09-18 ·Songwen Hu, Chase Stokes, Marti Hearst, Cindy Xiong Bearfield
similarity 0.54
Abstract · 4 pages

Eye-catching graphics, such as circular figure labels and word-scale visualizations, are increasingly being placed directly within long-form text paragraphs. Some research has claimed that inline visualizations can help readers understand data-rich passages more clearly. However, research in the science of reading calls into question the introduction of images within the flow of text. In this work, we conduct an exploratory study of eye movement in both the presence and absence of visual embellishments. Using a high-resolution eye-tracker (EyeLink Portable Duo), we observed small mean increases in vertical and cross-column saccade rates among six participants, with substantial variation among readers and no detected difference in comprehension accuracy. Mean subjective ratings for the data-light circular-glyph passage indicated it is more distracting than its unembellished comparison passage. These exploratory observations differ from prior findings and motivate a larger, fully crossed study of inline graphics.

0.35

VLMs Read Aloud: The Gap Between Recognizing and Using Visual Text

cs.CV cs.CL ·2026-11-01 ·Zhiyun Shi
similarity 0.53
Abstract · 23 pages, 7 figures, 21 tables

Visual-text compression renders long contexts as images and is engineered around how much of the text a vision-language model (VLM) can recover. Yet recovered text is not usable text. VLMs can transcribe the text in an image word for word and still fail to answer from it; given back their own transcript, they answer nearly as well as from the original text. We call this difference the read-use gap. Recognizing and using visual text have different resolution requirements: Qwen3-VL-8B's recognition saturates at 15-pixel characters while use catches up only at 24 pixels, and scaling from 2B to 32B raises recognition without narrowing the gap. Putting the evidence into words closes it: thinking VLMs recite the evidence in their thinking, and their own transcript, written without seeing the question, recovers most of what thinking adds. In a preregistered held-out test, the gap ranks how much eight VLMs gain from their own words over pixels (Spearman $ρ$ = 0.86), which recognition rates do not. We propose ALOUD, which spends visual resolution on finding and recognizing evidence and language tokens on using it. Without keeping the source text, it doubles the accuracy of a frozen Qwen3-VL-8B on long documents compressed 5x and 10x: at 5x, from 27.7% to 54.8%, near the original text's 58.5%, with a context smaller than the compressed pages themselves. For less than the cost of reading the original text, it leads at equal budget every method that does without it, at every level, and beats AGAR and VERA, which paste the original text back; at 5x it also beats LensVLM-9B, which is trained and keeps the original text. Visual-text compression is limited not by how much text a model can recover, but by how much it can use.

0.22

Robustness of Internal Signals for Hallucination Detection in Vision-Language Models

cs.CV cs.AI ·2026-10-19 ·Allison Chen, William Yang, Salma Abdel Magid, Jonathan Williams, Olga Russakovsky +1 more
published similarity 0.53
Abstract · NeurIPS 2026 Evaluations and Datasets track

When given an image and a text prompt, vision-language models (VLMs) can produce seemingly sophisticated text responses. However, they are also prone to hallucinate objects in their output, which can have detrimental consequences in safety-critical applications. To detect these hallucinations, certain methods interestingly leverage internal signals from the VLM as it generates the text response; potentially, these methods use the same signals that led to a hallucination to detect it. We conduct an in-depth analysis to understand the robustness of these detection methods. First, we show that a naive baseline of a generated word's token position in the VLM text response achieves competitive performance. Further, we reveal how some detection methods rely on token position and use this heuristic as a shortcut. Second, we explore detection methods' sensitivity to language and vision modalities. Our experiments suggest that strong detection performance is often driven by detecting language-based hallucinations, and when methods rely on token position, they tend to be less visually grounded and exhibit reduced robustness when hallucinations occur earlier in a text response. Lastly, we demonstrate that different detection methods can carry complementary signals; when ensembling methods together, adding weaker methods to an ensemble of higher performing ones still offers performance improvements. Altogether, while there may be useful internal VLM signals to detect hallucinations, our investigation reveals important limitations of the current methods that try to leverage them.

0.15

Can Scene Text Recognition Read Rare Compositions?

cs.CV ·2026-09-01 ·Genpei Zhang
benchmark similarity 0.52
Abstract · Preprint. Under review. Main text plus appendix: 10 figures, 13 tables

Scene text recognition is reported as 89--97% accurate on the six standard benchmarks, and the problem is widely treated as saturated. We present an alternative reading. When the same test images are stratified jointly by ground-truth word rarity and character n-gram novelty against a reference corpus, accuracy at the rare-word x rare-trigram corner of the resulting 5x5 grid drops 10--18 pt below the q3/q3 centre across nine English specialised recognisers, and the same direction (corner below centre) holds on all 13 of 13 (language, model) pairs we test across four writing systems (Latin, Han, Han+kana, Arabic). The drop is not a capacity bottleneck. A 6x vision-backbone scale-up (CLIP4STR-Base 158M -> CLIP4STR-Huge 1.0B, OpenCLIP ViT-H/14 LAION-2B) leads every benchmark in aggregate accuracy yet leaves the stress corner unchanged (86.9 -> 86.5, within paired-bootstrap noise). Four converging probes--layer-wise probing, confidence-when-wrong, attention re-balancing, and a cross-script commit-vs-abstain error split--localise the failure to the autoregressive decoder's lexical prior. We then ask how much of the gap existing techniques recover. Of 16 non-architectural mitigations, the largest mean q5/q5 gain is +1.3 pt and none clears the paired-bootstrap noise floor; the only intervention that does is the architectural shift from autoregressive to CTC decoding (SVTRv2, +2.5 pt, p=0.02, n=474). A confidence-routed AR-CTC ensemble adds a directionally consistent +0.6 pt that stays within noise, and its dominant learned coefficient is each model's own minimum-softmax confidence--independently echoing the mechanism above. No configuration we test improves both the compositional corner and aggregate accuracy. The rare-input long tail thus points to architectural change rather than added capacity.

0.27

MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects

cs.CL cs.AI ·2026-08-30 ·Jason Luo, Saibilila Abudukelimu, Judy Song, Andrew Feng, Shivank Garg +2 more
benchmarkcode/data similarity 0.52
Abstract · 12 pages. Accepted to the Context Beyond the Window (CBW) workshop at COLM 2026 (non-archival). Code and data: https://github.com/luoojason/…

Document question-answering systems increasingly answer questions over collections of retrieved documents rather than one clean source, so robustness to distracting context matters as much as reading ability. When such systems fail, it is often unclear whether the context was too long or the distractors were too close to the topic, because prior work tends to conflate these two effects. We present MUDDLE, a controlled benchmark that separates them. MUDDLE uses 270 human-annotated questions, each tied to a single source document, and instantiates every question in five conditions: the source alone, the source with two or four topically similar hard negatives, and the source with two or four random distractors. The random distractors are matched to the hard negatives in length and provenance, so an accuracy gap between the two arms reflects topical similarity rather than length. All five conditions are rendered in markdown, page images, and raw PDF, but the distractor sweep reported here is run in markdown, since a source plus its distractors exceeds current image and PDF input limits. We score answers with an LLM judge across three model families. In the complete markdown sweep, hard negatives lower accuracy more than length-matched random documents at both context sizes for gpt-5-mini, while random documents stay near the no-distractor baseline. The effect is small but directionally consistent, and for gpt-5-mini hard negatives significantly underperform length-matched random distractors when pooled across context sizes. We release the data and evaluation code for a reproducible study of context degradation.

0.69