CS Paper Radar

Handwritten Text Recognition Lives in the High-Pixel Variance Subspace

Carlos Garrido-Munoz, Jorge Calvo-Zaragoza

cs.CV cs.LG ·submitted 2026-09-28 15:54 UTC · v2
top venuebenchmarkposition

Abstract

In self-supervised pretraining for Handwritten Text Recognition (HTR), pixel reconstruction methods outperform contrastive methods, unlike in natural-image classification. We argue that this difference follows from where discriminative signal lies in pixel space: for HTR, it is concentrated in high-variance directions and largely absent from low-variance ones. This predicts that objectives preserving high-variance pixel content will transfer best. We test six SSL methods from three families (pixel-grounded MIM, JEPA, and contrastive) under matched encoder, data, and evaluation protocols on six handwriting benchmarks across five languages. With full labels, pixel-groundrounded SSL achieves the lowest CER on every benchmark and both frozen probes, exposes per-position character information that other families recover only through the readout, and is the only family to benefit from pretraining on real handwriting. Pixel-grounded representations are also more label efficient. Across datasets, encoder alignment with the high-variance pixel subspace predicts CER within every method. With a pretrained LLM decoder, a frozen pixel-grounded encoder is competitive with fully fine-tuned supervised baselines; full fine-tuning achieves the lowest mean CER and ranks first or second on every benchmark. These results show that the value of pixel reconstruction depends on where discriminative signal lies in the input.

Comment
Accepted at 40th Conference on Neural Information Processing Systems (NeurIPS 2026)

Closest papers

Nearest neighbours by meaning across the whole index.

Dataset Distillation for Handwritten Text Recognition: An Empirical Study

cs.CV ·2026-11-16 ·Félix Fuentes-Hurtado, Aric Ayllón, Juan C. Alfaro
similarity 0.64
Abstract

Dataset distillation (DD) replaces large training datasets with smaller synthetic ones to reduce computational costs. While traditionally used for classification, we present, to our knowledge, the first DD study for line-level handwritten text recognition (HTR). Unlike classification tasks, which use discrete labels, HTR employs unaligned sequences from a virtually infinite output space as labels. To adapt classification-based DD methods to this scenario, we define text pseudo-classes by clustering transcriptions based on character n-grams. We adapt gradient matching (GM), distribution matching (DM) and trajectory matching (MTT) to a Convolutional Recurrent Neural Network (CRNN) trained with Connectionist Temporal Classification (CTC). In this process, synthetic line images are generated by optimising only the pixels of real starting lines (either randomly chosen or selected as text medoids), while keeping their original transcriptions fixed. Across four datasets (IAM, George Washington, READ\,2016, LAM) and three budgets (1--10\,\% of the training lines), no distilled set outperforms its real initialisation lines. Random lines prove better than medoids, and further optimisation only degrades recognition. Methodologically, DM lowers its objective without benefit, while GM and MTT barely move theirs. We turn these findings into guidelines for evaluating data compression in HTR.

0.18

inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild

cs.CV cs.AI ·2026-10-28 ·Sidra Hanif
benchmarkcode/data similarity 0.63
Abstract

Human handwriting is produced as an ordered sequence of pen movements, but an offline image keeps only the final ink. Handwriting trajectory recovery aims to restore the ordered stroke points and their pen-up/pen-down transitions. Existing image-trajectory pair datasets are recorded with digitizing tablets under controlled conditions, which limits writer coverage and stylistic diversity. Moreover, there is no benchmark that measures the visual grounding capabilities of general-purpose Vision-Language Models (VLMs) for the stroke trajectory recovery task. In this work, we introduce inkGround, a benchmark for stroke trajectory recovery from full-page, camera-captured handwriting. inkGround pairs 537 page images (~18.7k words, ~163k strokes) with reference trajectories produced by a pretrained offline-to-online model, filtered by OCR agreement, and verified by human raters. Because these references are pseudo-labels that represent one plausible pen trajectory, we also evaluate VLMs on the IAM-OnDB dataset, which provides recorded pen trajectories. We cast the task as text-conditioned point grounding: given a page image and a target word, the model must locate the word and output its ordered stroke points. Outputs are scored by trajectory fidelity (DTW, Chamfer distance), readability of the rendered strokes (Accurate Rate, Correct Rate), and style similarity. We benchmark four open-source and three frontier closed-source VLMs with a shared in-context prompt. Closed-source models, notably Gemini-3.1-Pro, produce legible trajectories close to the reference, and this ranking also holds on IAM-OnDB.

0.26

Exploring In-Context Learning for Handwritten Text Recognition

cs.CV ·2026-09-29 ·Eric Ayllon, Abel Gandia, Jorge Calvo-Zaragoza
similarity 0.63
Abstract · 19 pages, 3 figures

Handwritten Text Recognition (HTR) systems have become an indispensable tool for the digitization of historical documents. Not only do they cut down time and cost, but they also allow democratizing access and processing of their contents by generating their transcripts. However, literature in HTR currently focuses mostly on specialized models that require large amounts of annotated samples to achieve satisfactory performance. We explore the use of In-Context Learning with pre-trained Vision-Language Models (VLMs) to create a transcription pipeline without updating the model's parameters. We then evaluate this pipeline across multiple collections and models, and demonstrate that general-purpose VLMs can be effectively taught how to transcribe handwritten text from images. To assess how our observations may translate to practical applications, we evaluate the performance in a Cross-Domain (CD) scenario, where context examples are drawn from a different collection than the query image. Results in both the controlled In-Domain (ID) scenario and the realistic CD scenario follow the same patterns. First, as context size grows, the error range is expected to narrow towards the average performance. Thus, larger context sizes sacrifice the performance of the oracle-best sampling for lower expected error rates. The results obtained show that, without any parameter updates, this methodology has strong potential to compete with traditional HTR in the presence of domain shift. Moreover, we show and argue that some context samplings work better than others and suggest more effort should be put into finding an ideal sampling method in future work.

0.30

Measuring Annotation Efficiency for Handwritten Devanagari Recognition: Sample-Complexity Curves for Four Pretraining Regimes

cs.CV cs.LG ·2026-09-15 ·Manglesh Kumar Pandey, Sumit Kumar Banshal
similarity 0.61
Abstract

To train handwritten text recognition systems we need word images and their corresponding transcriptions, and these transcriptions are produced manually. For a script that can be read by only a small number of specialists, this manual transcription is a limitation, because the trained models are supposed to save the time of those same specialists. A relevant question therefore arises: how many transcriptions are needed before a recogniser becomes useful, and how much of that cost can pretraining remove? In this study the answer is measured directly for handwritten Devanagari. We keep the recogniser, optimiser and evaluation protocol the same and change only the number of real transcribed words used for fine-tuning across nine budgets from 10 to 4,000 and four initialisation regimes, with six seeds at every point. The resulting curves are then converted into annotation-equivalent terms. A CER of 0.50 is reached by supervised synthetic pretraining using only 81 transcribed words, whereas random initialisation requires 355, which gives a label multiplier of 4.40 [3.56, 4.99]. There is a zero-shot reference point as well: with no real transcribed words at all, this pretraining is worth about 136 of them. This advantage gets smaller as the target accuracy improves, and at the most demanding target we measure, it cannot be distinguished from no saving at all. A fourth arm in which only the encoder is transferred separates the effect of the pretraining method from that of transfer scope, and masked image modelling is observed to transfer negatively over a bounded range of budgets. We emphasise that the scarcity in this study is constructed by subsampling a large corpus.

0.18

DRA-HTR: Disentangling Representation and Adaptation for Handwritten Text Recognition

cs.CV ·2026-11-13 ·Gayan H. Pathirage, Florent Imbert, Denis Coquenet, Elisa H. Barney Smith
similarity 0.61
Abstract

Handwritten text recognition (HTR), particularly when applied to historical documents, remains challenging due to the scarcity of annotated data and the substantial domain shifts caused by variations in handwriting styles, document degradation, acquisition conditions, and linguistic characteristics. While large amounts of unlabeled target-domain data are often available, effectively exploiting them requires adaptation strategies tailored to the specific requirements of HTR. We propose DRA-HTR, an unsupervised generative domain adaptation pipeline specifically designed for HTR and built upon the DRANet framework. The proposed approach combines source-domain pretraining, unsupervised adaptation to the visual characteristics of the target domain, relying on the generation of target-style source samples for task-specific training. We further investigate the integration of post-HTR correction to complement visual adaptation with language adaptation, relying on large language models, which we introduce as DRAL-HTR. The proposed framework is evaluated across seven HTR datasets spanning four languages, using a single set of hyperparameters for all source-target combinations. The recognition model achieves state-of-the-art performance on four datasets and competitive results on the remaining three. In addition, the proposed DRA-HTR establishes a new state-of-the-art for domain adaptation across all seven datasets and substantially improves overall robustness and training stability compared to the original DRANet framework: it reduces the Character Error Rate (CER) in average by 13.52 percentage points, and the average standard deviation decreases from 7.23 to 1.11 percentage points. Finally, post-HTR correction further reduces the CER by 2.16 percentage points.

0.22

Scaling Multilingual Pretraining of Pixel Language Models

cs.CL cs.AI ·2026-10-21 ·Ilker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott
similarity 0.61
Abstract · 33 pages, 25 figures, 14 tables

Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages. Transfer to unseen scripts improves with multilingual pretraining, but does not further improve with scaling. On typologically distant low-resource languages, our models outperform subword and character-level baselines across many writing systems. We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added. Overall, our findings characterize how multilingual scaling improves pixel language models.

0.13

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

cs.CV cs.AI ·2026-08-19 ·Zinuo Guo, Min Zhang, Bo Jiang
top venuebenchmarkhot direction similarity 0.61
Abstract · CIKM 2026

Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to faithfully read real handwriting remains underexplored. Existing OCR benchmarks focus largely on printed text or clean single-line inputs, leaving limited coverage of realistic handwritten OCR scenarios such as multilingual handwriting, writer errors, and structurally complex mathematical expressions. We introduce OmniHandwritingOCR, a diagnostic benchmark for evaluating MLLMs and OCR systems on handwritten OCR. It covers handwritten text recognition and handwritten mathematical expression recognition across six subtasks and twelve subsets, totaling 77.57K labeled images from public datasets and newly collected student writings. A key component is a difficulty-stratified multi-line formula corpus designed to test robustness under increasing structural complexity. We evaluate thirteen open- and closed-source systems with five complementary metrics under a unified protocol. Results show that current systems remain far from faithful transcription: performance drops sharply on complex multi-line formulas, model rankings vary across language and formula settings, and several generative models hallucinate plausible but visually unsupported corrections. OmniHandwritingOCR provides a challenging testbed for diagnosing language, content, structural, and visual-grounding failure modes of multimodal models in handwritten OCR scenarios.

0.51

How Far Can Synthetic Data Take Thai OCR?

cs.CL cs.AIcs.CV ·2026-09-03 ·Kunat Pipatanakul
similarity 0.61
Abstract · 20 pages, technical report

We investigate what makes synthetic OCR supervision transfer to real Thai documents and use the resulting insights to build Wayu-Paxa-OCR-Zero, a Thai OCR model adapted without OCR labels from real Thai document pages. Synthetic data provide exact labels at scale, but "realism" conflates source domain, page context, typography, spatial structure, and glyph variation. We disentangle these factors with a controlled document-reconstruction pipeline and evaluate each variant under page- and crop-level training on printed and handwritten Thai documents. Non-text context has little consistent effect, whereas typeface diversity, two-dimensional structure, and real handwriting glyphs improve transfer; moreover, source-domain matching depends on training granularity, with in-domain reconstruction approaching real printed supervision under page-level training (1.82% versus 1.31% median character error rate) but underperforming out-of-domain reconstruction under crop-level training (15.59% versus 5.52%). Guided by these findings, we adapt the 0.9B-parameter PaddleOCR-VL-1.6 into Wayu-Paxa-OCR-Zero using 45,723 synthetic pages: relative to its base checkpoint, it reduces median character error rate from 6.64% to 1.24% on printed pages and from 74.87% to 20.55% on handwriting and outperforms Typhoon OCR v1 7B on all five evaluation sets, showing that synthetic-only training can be competitive.

0.19