CS Paper Radar

What Not to Erase: Saliency-Constrained Augmentation for Low-Resource Handwritten Character Recognition

Paraskevi Platanou, John Pavlopoulos

cs.CV cs.LG ·submitted 2026-10-19 21:30 UTC · v1

Abstract

Optical Character Recognition (OCR) and handwritten character recognition (HCR) in low-resource historical material collections often rely on data augmentation, yet augmentation strategies are typically evaluated almost exclusively through predictive accuracy. This leaves open whether a model's improved performance is accompanied by saliency stability. This question is important in low-resource historical HCR, where recognition is performed on variable and partially degraded letterforms and two models with similar predictive performance may differ in how consistently their saliency patterns respond to such variation. We therefore study saliency-guided augmentation as an explanation-aware strategy for low-resource HCR. Rather than treating saliency maps only as post-hoc interpretations, we use them to define augmentation variants that perturb, preserve, or constrain visually informative regions of character images. We compare classical document-style transformations, random lacunae, and several saliency-constrained lacuna variants. To evaluate the effect of augmentation beyond accuracy, we introduce the Explainability Consistency Index (ECI), a metric that quantifies the stability of saliency patterns under controlled perturbations. We find that saliency-constrained augmentation can improve recognition over classical augmentation, but no single saliency strategy is consistently best. The preferred form of the constraint varies with both architecture and corpus, indicating that saliency is better treated as a mechanism for controlling augmentation than as a fixed augmentation strategy. In some settings, saliency-constrained training improves explanation consistency even when predictive performance is similar. This shows that predictive metrics alone can obscure differences in model sensitivity under perturbation, motivating explanation consistency as a complementary evaluation dimension.

Closest papers

Nearest neighbours by meaning across the whole index.

Assessing Representation Semantics Under Language Pretraining in Fine-Grained Classification

cs.CV cs.AI ·2026-10-20 ·Md Raisul Kibria, Sébastien Lafond
similarity 0.65
Abstract

Understanding how supervision signals shape semantic visual representations remains challenging, particularly in fine-grained, low-data settings. Standard evaluations of vision models rely on downstream accuracy, which often fails to capture structural differences in learned representations. We propose a compact explainability-driven framework for comparing encoders trained with classification-only and caption-generation supervision on CUB-200. Using a controlled two-stage protocol in which both encoders are ultimately optimized for the same classification task, we enable a direct comparison of the resulting representations. Saliency maps are generated using attention rollout and gradient-weighted rollout and evaluated with five metrics covering faithfulness (positive and negative perturbation) and localization (keypoint emphasis, background attribution, and Earth Mover's Distance). Across random seeds and explanation methods, caption supervision yields representations that are more sensitive to the removal of salient regions, better aligned with annotated object parts, and less reliant on background regions, whereas classification supervision produces representations that are slightly more robust to the removal of low-saliency regions. An ablation study on the Oxford-IIIT Pet dataset shows consistent trends across key metrics. These results demonstrate that the proposed framework makes supervision-induced representational differences observable and quantifiable beyond downstream accuracy alone. More broadly, it provides a systematic basis for assessing learned representations and comparing alternative training and transfer-learning strategies.

0.18

UniLipi: A Unified Multi-Script OCR for Historical Indic Manuscripts

cs.CV ·2026-08-28 ·Tathagata Ghosh, Sai Madhusudan Gunda, Simran Singh Sandral, Ravi Kiran Sarvadevabhatla
top venue similarity 0.63
Abstract · To be published in ICDAR 2026

Optical character recognition (OCR) for handwritten Indic manuscripts is essential for large-scale digitization and computational access to manuscript heritage. However, existing approaches are typically developed for one script at a time and require substantial script-specific customization. This limits scalability and practical deployment across diverse collections. We present UniLipi, a unified multi-script OCR model for handwritten Indic manuscripts trained jointly across 13 Indic scripts within a single framework. UniLipi directly handles realistic manuscript conditions, including extreme variation in line geometry, large variation in line length, and partial interruptions caused by non-textual manuscript entities such as holes, stains, or pictorial illustrations. To operate effectively under ultra low-resource conditions, the model leverages script-aware synthetic manuscript data generation, substantially reducing reliance on large volumes of real annotated data. Beyond historical manuscripts, we show that UniLipi serves as an effective foundational pretrained model. Specifically, its learned representations enable good OCR performance for contemporary Indic handwriting and extend to several non-Indic scripts, including Tibetan, Italian, Latin, and Chinese scripts. In addition to transcription, UniLipi predicts script identity and per-line native character counts, supporting practical manuscript cataloging workflows.

0.35

DDVR-Net: Degradation-Aware Decoupling and Visibility-Guided Restoration for Historical Document Recognition

cs.CV ·2026-10-12 ·Jiu Sha, Yu Weng, Mengxiao Zhu
top venuebenchmarkcode/data similarity 0.60
Abstract · 18 pages. Accepted to the 34th ACM International Conference on Multimedia (ACM MM 2026)

Historical document analysis is crucial for cultural heritage preservation and low-resource language digitization. However, real historical documents often suffer from severe degradation, such as ink erosion and structural damage, causing mixed-visibility character sequences. Conventional OCR systems struggle in such evidence-limited scenarios, while contextual language models may over-correct visually clear characters. To address these challenges, we propose DDVR-Net, a degradation-aware decoupling and visibility-guided restoration framework for historical document recognition. DDVR-Net decouples intrinsic glyph structures from physical corruption at the feature level and predicts local visibility to estimate character reliability. A contextual decoder is selectively activated through a visibility-guided gate, restoring degraded characters while preserving clear visual recognition. Furthermore, we construct a low-resource multilingual benchmark covering Tibetan, Mongolian, and Uyghur scripts with 187000 document pairs, combining large-scale synthetic data and progressive training with real-world annotations to reduce the synthetic-to-real gap. Extensive experiments demonstrate that DDVR-Net consistently outperforms existing methods across three languages and diverse reading orders, providing an effective framework and benchmark for low-resource historical document recognition. Code and Benchmark: https://github.com/Shajiu/DDVR-Net

0.47

Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment

cs.AI ·2026-10-22 ·Qiuyu Kong, Zanxi Ruan, Marco Cristani, Yiming Wang
code/data similarity 0.59
Abstract

Long-text image-text alignment is essential for fine-grained scene understanding, yet existing methods mainly focus on extending context length or improving local alignment. We identify an overlooked failure mode, saliency collapse, where models over-rely on text that features visual prominence while underutilizing non-salient context. Consequently, retrieval performance degrades sharply when salient content is reduced, despite that the non-salient context remains informative. To address this, we propose RoSA, a saliency-aware finetuning framework that decomposes image-text pairs into complementary salient and contextual views, enabling joint learning of prominent and contextual semantics. We propose a modality-independent decomposition strategy to combine object-level visual saliency estimation with LLM-based text decomposition to bridge vision and language. We employ auxiliary objectives to explicitly align these views, ensuring robust representation with both salient and contextual features. RoSA greatly improves long-text retrieval and demonstrates strong robustness to saliency collapse across benchmarks. Moreover, our auxiliary objectives are plug-and-play, bringing consistent gains to existing tuning methods. Code and checkpoints are available at: https://github.com/intelligolabs/RoSA.

0.26

Amplify What You Gaze At: Target Saliency Boosting in Text-to-Image Generation

cs.CV ·2026-09-25 ·Shengqi Dang, Zhengxi Yu, Feilin Han, Xingyu Lan, Nan Cao
dataset similarity 0.59
Abstract

Text-to-image generation has advanced in controlling what, where, and how objects appear, yet how visual attention is distributed among objects remains largely unexplored. In this paper, we introduce Target Saliency Boosting, a new task aimed at boosting the visual saliency of a specific object during text-to-image generation without requiring any visual priors. Our key insight is that visual saliency is inherently relative: boosting the saliency of a target object also depends on the global saliency distribution across all objects in the scene. Based on this insight, we propose GazeME, a lightweight framework that uses saliency-marked prompts, inserting learnable marker tokens around object descriptions to indicate which objects to visually emphasize or suppress. To learn these markers, we construct a saliency-semantics dataset that associates objects in image--prompt pairs with object-level saliency scores, and propose Saliency Prior Marker Activation (SPMA), a saliency-aware stochastic marker activation strategy that exploits relative saliency relationships for robust training. During inference, GazeME automatically inserts appropriate markers into the prompt, thereby directly enhancing the visual saliency of the target object. Extensive experiments demonstrate that GazeME effectively boosts target saliency while preserving both semantic alignment and image quality.

0.11

AVAtt: A Free-Viewing Eye-Tracking Dataset and Benchmark for Visual Attention on Paintings

cs.CV cs.HCeess.IV ·2026-10-21 ·Mohamed Amine Kerkouri, Marouane Tliba, Aladine Chetouani, Alessandro Bruno
publishedbenchmarkdatasetcode/data similarity 0.59
Abstract · Extenstion of 10.1145/3649902.3655656 published at ETRA 2024

Saliency and scanpath models are developed almost exclusively on natural photographs, yet paintings are composed by artists to guide the viewer's eye and differ from photographs in texture, colour and abstraction. We introduce \avatt, a free-viewing eye-tracking dataset on \nImages{} paintings spanning sixteen art movements and five centuries, with \nScanpaths{} scanpaths and \nFixations{} fixations from \nParticipants{} participants (\spPerImageMin--\spPerImageMax{} observers per painting). For every painting we release individual scanpaths with fixation timing and eye--screen distance, fixation maps, saliency maps smoothed at one degree of visual angle, temporal saliency videos and rich art metadata. We benchmark classic and deep saliency models against human agreement, revealing substantial headroom for art-aware models, and rank 44 scanpath-similarity measures, including a new language-free family of image-encoder metrics built on ten CNN, ViT and self-supervised encoders. With controls that remove viewing biases, we show that faces are fixated \faceEnrich$\times$ more than expected and within half a second, that local structure draws the eye, that composition rules fitted from image content predict where people look, and that art style shapes how much viewers agree. Dataset, code and a Python package are available at: https://huggingface.co/datasets/Amine-CV/AVAtt.

0.42

OCR in the Foundation Model Era: From Specialist Models to Unified Models

cs.CV ·2026-11-03 ·Zhiheng Li, Zongyang Ma, Jiaxian Chen, Cheng Cui, Yi Liu +5 more
surveycode/data similarity 0.59
Abstract · 21 pages, 7 figures, 7 tables. Resources: https://github.com/zhihengli-casia/Awesome-OCR-in-the-Foundation-Model-Era

Optical Character Recognition (OCR) is among the oldest and most widely deployed applications of artificial intelligence. This survey reviews its development, focusing on the Foundation Model Era. We define OCR through faithful transcription: within a specified visual scope, OCR should completely reconstruct the text, notation, and document structure observable in the source image, and every textual, notational, or structural element in the output must be verifiable from visual evidence in that image. We trace the development from specialist OCR models toward unified models, examining joint text detection and recognition, structured sequence generation, and the use of pretrained vision-language models (VLMs). We organize contemporary OCR systems into five coexisting families: modular toolkits and task-specific models, specialized document VLMs, hybrid workflows, general-purpose VLMs, and commercial products and managed services. We compare their architectures, data, and training, examining hallucination and omission, insufficient spatial grounding, and inference cost in generative OCR. We review evaluation from character and scene text recognition to parsing individual structural elements and full-page transcription, highlighting gaps in consistent output definitions, attribution of performance to individual components, and coverage of real capture conditions and visible content and structure. Finally, we summarize the challenges facing OCR and discuss directions for future research. We continuously track related work at https://github.com/zhihengli-casia/Awesome-OCR-in-the-Foundation-Model-Era.

0.18

MSTypography: Multi-character Semantic Typography via Balancing Word Legibility and Object Recognizability

cs.CV cs.GR ·2026-09-29 ·Xinye Yang, Xinding Zhu, Kai Fang, Xinyi Ren, Mengjian Li +2 more
similarity 0.59
Abstract

Semantic typography is a design technique where the visual representation of a word conveys its semantic meaning, while maintaining its legibility. Existing digital typography methods mainly focus on single-character scenarios. They suffer from a lack of legibility constraints and insufficient local deformation when extended to multi-character words, as the intricate structures among multiple characters are hardly preserved during the typography process. In this paper, we propose a global-to-local typography framework for multi-character scenarios. It performs mask-driven silhouette approximation at the global level, while semantic-guided refinement at the local level, with a culling step in between to improve efficiency. To preserve word legibility, we designed structural losses (including explicit collision constraints and implicit Jacobian singular value constraints) and an OCR constraint for character-level readability. To enhance the object recognizability, we leverage semantic guidance with diffusion priors, which drives the character glyph toward the target concept while preserving its structural integrity. To the best of our knowledge, this is the first multi-character semantic typography method that effectively balances word legibility and object recognizability. Evaluations on five representative languages (English, Chinese, Japanese, Korean, Arabic) demonstrate superiority over SOTA methods. Codes will be open-sourced.

0.30