CS Paper Radar

All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts

Xingsong Ye, Yongkun Du, Jiaxin Zhang, Zhixian Li, Chong Sun, Chen Li, Jing Lyu, Lianwen Jin, Zhineng Chen

cs.CV ·submitted 2026-09-21 03:34 UTC · v1
code/datahighly cited

Abstract

Multilingual scene text recognition (STR) remains challenging due to the scarcity of training data for most languages and the difficulty of serving diverse scripts within a single model. Existing solutions either deploy one recognizer per language, inflating cost and introducing error accumulation, or rely on massive vision-language models (VLMs) that are expensive and still inaccurate on many scripts. In this work, we pursue an all-in-one multilingual recognizer that is simpler than per-language experts, lighter than VLMs, and more accurate than both. First, we construct TextMuSS-10M, a large-scale synthetic scene text dataset spanning 10 scripts and 229 languages. It provides balanced and sufficient supervision where real data is unavailable. Second, we propose ScriptMoE, a script-aware Mixture-of-Experts (MoE) architecture. It shares a single visual encoder and replaces the dense decoder with a sparse MoE block, which consists of an image-level router dispatches each image to the top-2 script-aligned experts and a shared expert absorbs cross-script knowledge. Extensive experiments on our assembled TextMuSS-Bench (10 scripts, 10,899 images) show that ScriptMoE achieves the highest accuracy of 82.06%, outperforming the strongest STR baseline by 1.31%. On the CC-OCR end-to-end multilingual task, replacing only the recognizer in PP-OCRv5 with ScriptMoE lifts F1 score from 65.71% to 80.89%, slightly surpassing the best VLM (80.73%) at a fraction of the parameter count.

Comment
Code: https://github.com/YesianRohn/ScriptMoE & https://github.com/Topdu/OpenOCR

Closest papers

Nearest neighbours by meaning across the whole index.

What Can Low Resource Languages Learn From Each Other?

cs.CV ·2026-08-27 ·Achyuth P, Kahaan Shah, Chetan Arora
top venue similarity 0.70
Abstract · 16 pages, 4 Tables, 6 Figures. To appear at ICDAR 2026. This version predates reviewer revisions. A revised version will be posted

Despite the rapid advancement of Vision-Language Models (VLMs), their linguistic reach remains largely confined to high-resource languages, leaving the majority of the world's 7,000+ living languages on the wrong side of a growing digital divide. This disparity is especially pronounced in Optical Character Recognition (OCR), where low-resource scripts lack the massive datasets required for traditional scaling laws. We investigate OCR adaptation in extreme data-scarce regimes (<10K real and <250K synthetic images), demonstrating that conventional fine-tuning strategies often reach a performance ceiling. Our key finding reveals a structural inefficiency in language-specific adaptation: while higher layers of specialized models diverge to capture unique script nuances, the lower layers learn redundant, highly similar features. Motivated by this observation, we propose PSMC (Pre-train, Specialize, Merge, and Co-train), a data-efficient framework that capitalizes on a cross-script "transfer effect". Our approach first derives language-specific experts from a high-resource base model, then employs task arithmetic to fuse these experts into a unified, high-performance multilingual back- bone. Extensive evaluation across 10 Indian scripts (supporting 20+ languages) shows that PSMC achieves a ~2% average improvement in Word Recognition Rate (WRR) over individual specialist models without increasing parameter count. Our results indicate that joint training in the merged latent space facilitates a constructive knowledge transfer that benefits all constituent scripts, providing a scalable pathway for inclusive VLM development. Source code and datasets will be released post publication.

0.55

FlexReader: Towards Adaptive Decoding Paths for Efficient Arbitrary-Length Scene Text Recognition

cs.CV ·2026-10-30 ·Zhibin Ma, Pengwen Dai, Zhigang Yao, Xugong Qin, Wenqi Ren +1 more
similarity 0.69
Abstract · 11 pages, 5 figures

Existing Scene Text Recognition (STR) models are inherently coupled with a specific decoding strategy by their training paradigms, limiting their flexibility in deployment across diverse recognition scenarios. In this paper, we propose FlexReader, an STR model that supports multiple decoding strides, enabling a single model to achieve an optimal accuracy-efficiency trade-off across different datasets. Specifically, we first introduce a Sub-string Perception module to perceive the target sub-string from visual features, followed by multiple parallel decoding blocks that independently decode each character in the sub-string. During inference, FlexReader supports decoding with arbitrary strides within the perceived sub-string range. Furthermore, we propose Adaptive Prefix Decoding, which adaptively selects the decoding stride at each step based on the model's confidence in the decoded characters, thereby identifying the optimal decoding path for each input. Experimental results demonstrate that FlexReader exhibits robust recognition capability across a diverse range of benchmarks, including those featuring long text, curved text, and occlusions. It consistently outperforms existing decoding strategies in recognition accuracy while achieving up to 6-flod decoding speedup over autoregressive decoding.

0.22

ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts

cs.CV cs.LG ·2026-09-11 ·Dang Hoai Nam, Nguyen Duy Hieu, Quang Huu Hieu, Vo Nguyen Le Duy
benchmark similarity 0.68
Abstract

Handwritten text recognition resources are often small and distributed across collections that differ in language, script, document structure, and annotation format, making joint page-level training difficult. We propose ExpertHTR, a unified vision-language framework that addresses this problem through complementary supervision and conditional model capacity. Structural annotations from heterogeneous datasets are first organized through a common Page-Region-Line representation and used to construct four related training tasks for complete transcription, physical-line coverage, text localization, and localized recognition, without requiring additional manual labels. Building on a jointly trained dense model, ExpertHTR introduces a sparse Mixture-of-Experts architecture with an always-active shared branch and conditionally routed full-MLP experts. Sparsegen allows the number of active routed experts to vary with the hidden representation, while routing regularization reduces persistent concentration on a small subset of experts. Experiments on seven heterogeneous handwriting benchmarks show that complementary supervision consistently improves training with page transcription alone, while joint multi-source training provides further gains on most datasets. The proposed sparse expert model further improves the dense baseline on six of the seven sources. The final unified model also substantially outperforms the evaluated general-purpose OCR and vision-language systems on most benchmarks and achieves state-of-the-art performance on the IAM paragraph-level benchmark, while specialized HTR systems remain stronger on several challenging collections.

0.19

From Zero to OCR: A Data-Centric Framework for Unsupported Low-Resource Scripts

cs.CV cs.CL ·2026-10-18 ·Faizan Iqbal, Haq Nawaz Malik, Hafsah Pervaiz
code/data similarity 0.67
Abstract · 9 pages, 9 figures, 4 tables. Model: https://huggingface.co/Faizaniqbal/KoshurOCR-CRNN-Baseline

Optical character recognition (OCR) remains inaccessible to many low-resource scripts because annotated document images are unavailable and synthetic training data can fail to preserve script-specific visual structure. We present a data-centric framework that builds end-to-end OCR for such scripts from only an unannotated text corpus and digital fonts, without manually annotating a single training image. A synthetic data engine renders word, line and page images with correct complex-script shaping and automatically derived labels; a YOLOv8 word detector and a convolutional recurrent recognizer trained with connectionist temporal classification learn from these images; and a stage-gate evaluation protocol isolates recognition error before detection is introduced. We demonstrate the framework on Kashmiri, a Perso-Arabic Nastaliq script with approximately 6.8 million speakers that none of six widely used OCR engines supports. Three line-level recognizers, including one transferred from Arabic, stalled between 41% and 79% character error rate (CER), motivating a word-level formulation; subsequent improvements came primarily from rendering fidelity, diacritic-preserving crop margins and tuned non-maximum suppression. The resulting system achieves 0.39% CER on held-out words, 10.0% on synthetic pages and 13.5% on browser-rendered digital pages, and establishes reference results of 18.45% on typeset book pages and 49.8% on degraded historical documents. We publicly release the trained models, the inference code and a synthetic dataset of 285,864 Kashmiri text-line images, and we describe a step-by-step procedure for building OCR for other unsupported scripts.

0.35

Benchmarking Multilingual Document Understanding in Vision-Language Models

cs.CL cs.AI ·2026-11-10 ·Matteo Gabburo, Daniela Baiamonte, Stefano Simonazzi, Silvia Gardini, Sean Cesare Lazzeri +1 more
benchmark similarity 0.67
Abstract · 18 pages, 4 figures

Recent Vision-Language Models (VLMs) perform strongly on multimodal tasks, Document Understanding (DU) among them, but the benchmarks that measure them mostly cover English documents and narrow problems. How well current VLMs generalize across languages, tasks, styles and layouts therefore remains unclear. We introduce MDU, a multilingual DU benchmark of 23146 examples spanning 13 tasks and five languages: English, French, German, Italian, and Spanish. The tasks isolate complementary capabilities, from surface-level text and Markdown extraction through summarization, classification and named-entity recognition to layout-sensitive passage and layout extraction. We evaluate 45 state-of-the-art open-weight VLMs from 1.7B to 235B parameters, plus one commercial system under the same protocol. The benchmark is far from saturated: the best average is 69.0, only eight of the 45 evaluated systems reach 65, and the two best results come from 27B models rather than from the largest systems we evaluate. Performance is uneven in ways that scale does not fix. The gap between the strongest and weakest language does not close as models grow, and it survives controlling for the domain composition of each language. Capabilities also do not improve together: once we partial out overall model quality, per-task scores are largely uncorrelated, and transcription ability in particular trades off against semantic extraction.

0.19

Reading Between the Layers: Cross-Modal Alignment and Representational Hierarchy in Vision Transformers

cs.CL ·2026-10-23 ·Sugam Karki, Raúl Vázquez, Yova Kementchedjhieva
published similarity 0.65
Abstract · Accepted to AACL 2026 Findings

While large vision-language models excel at optical character recognition, it remains unclear how their underlying vision encoders process text and its meaning, and whether they support anything resembling reading comprehension. We systematically study transformer-based vision encoders, using cross-modal alignment between rendered text and visual concepts together with probing and retrieval-based analyses as a controlled window into these broader capabilities. Our layer-wise analyses span orthographic, grammatical, and semantic organization, and reveal that strong cross-modal mapping emerges only in the deepest layers, after unimodal object and text recognition have already developed, coinciding with a progression from surface-level to semantic representations. Although this hierarchy is strongest in contrastively trained models, larger self-supervised models and text-rich training data partially support similar behaviors without explicit language supervision. Together, our results suggest that increasing model and training-data scale, as well as text-rich training data, can partially compensate for the absence of language supervision.

0.25

Mobilizing the Archive: Bitext Extraction at Scale from Archival Documents for Extremely Low-resource Languages

cs.CL ·2026-10-29 ·Chutong Meng, Alec J. Bateman, Antonios Anastasopoulos
similarity 0.64
Abstract

While mining parallel sentences from web pages has proven effective for low-resource machine translation, the majority of the world's languages have little to no web presence. What they do have, though, are resources that are the result of centuries of efforts by linguists, such as grammars, dictionaries, and other documentation materials. We propose extracting translation examples from such linguistic resources, which offer parallel data for languages under-represented online and could potentially scale to thousands more languages. We develop a pipeline to achieve this, leveraging an Optical Character Recognition (OCR) model as well as a vision language model (VLM), to ensure identified content to be aligned with visual layout information. We demonstrate the utility of the extracted bitexts for bootstrapping translation systems across 20 languages, both by training machine translation models and by applying in-context learning for large language models. For in-context learning, we observe improvements of +15.78 and +6.94 chrF2++ over zero-shot translation for into low- and high-resource languages, respectively.

0.24

A Free Lunch? Adapting PP-OCRv6 for Historical Text Recognition

cs.CV ·2026-09-17 ·Benjamin Kiessling
similarity 0.64
Abstract

Despite impressive reported scores, large vision-language models have seen limited practical uptake in historical automatic text recognition because of their computational cost, dependence on large-scale pretraining, and hallucination. Historical ATR therefore continues to rely largely on compact CRNN line recognizers, which are visually grounded and trainable on modest data. Lightweight recurrence-free recognizers promise the accuracy of larger models with the practical advantages of CRNNs, yet have not been comprehensively evaluated on historical writing. We adapt PP-OCRv6, a recent compact text recognizer without strong language modeling, for historical line recognition and compare it with a conventional CRNN across generalized pretraining, domain-specific training, corpus-level fine-tuning, and manuscript-specific few-shot adaptation on multilingual Latin- and Arabic-script material. While PP-OCRv6 does not consistently outperform the baseline when trained from scratch, heterogeneous pretraining produces markedly better generalization. Comparisons with the Qwen3.5-based Medusa recognizer further show that fine-tuned PP-OCRv6 can outperform a large VLM tailored towards historical Latin-script HTR.

0.14