CS Paper Radar

Age-Adaptive Handwriting Reconstruction from an IMU-Based Digital Pen through Shared Representations and Domain-Specific Heads

Florent Imbert, Yann Soullard, Eric Anquetil, Hui Han

cs.CL cs.AIcs.CVcs.LGcs.NE ·submitted 2026-09-15 14:18 UTC · v1
published

Abstract

Digital pens are widely used to capture handwriting on digital devices, enabling precise trace recording and enhancing human-computer interaction. However, most are bundled with tablets and lack cross-brand compatibility. Recent digital pens equipped with kinematic sensors have emerged, designed for use on any surface. This especially opens significant potential for supporting handwriting acquisition in classrooms. Handwriting reconstruction from such an IMU-equipped pen poses a challenge due to the significant variability in sensor signals between adults and children. Even when producing visually similar traces, variations in writing dynamics, motor control, pen holding, and user confidence introduce substantial discrepancies in the captured signals. Additionally, the high variability in children's handwriting requires collecting large amounts of data, which is not feasible to implement in a school environment at scale. Furthermore, models trained exclusively on adult data fail to generalize to children's handwriting, and conversely, models trained on children's data perform poorly on adult writers. This crosspopulation degradation highlights the need for a unified model that can be deployed directly on the pen, without any user-specific adaptation. To address this issue, we propose a cross-domain learning, using an original neural network architecture based on a Temporal Convolutional Network and multiple prediction heads. The model is designed to be robust across age groups by leveraging shared features while effectively handling variability induced by differences in graphomotor development. This approach aims to improve handwriting trace reconstruction from sensor data, where each domain benefits from additional data provided by the other domain.

Journal ref
Automatically Domain-Adapted and Personalized Document Analysis workshop (ADAPTA), ICDAR 2026, Sep 2026, Vienna (AUSTRIA), Austria

Closest papers

Nearest neighbours by meaning across the whole index.

Write on Paper and Get the Online Digital Trace: A New Era for Handwriting

cs.LG ·2026-09-11 ·Florent Imbert, Yann Soullard, Eric Anquetil, Tanja Harbaum, Alexey Serdyuk +3 more
published similarity 0.71
Abstract

Capturing the digital trace of handwriting usually requires a specific stylus and a compatible substrate, be it a capacitive touchscreen, an ElectroMagnetic Resonance (EMR) tablet as used in Wacom systems or special paper. While writing on regular paper offers rich haptics, no latency and is well known for improving information retention, no low-cost and widely accepted, effective solution exists to digitize such a pen trace. The challenge is to accurately track the pen's trajectory without an external reference system while allowing unrestricted freedom of pen movement across a surface. We propose an innovative solution that combines a digital pen, advanced artificial intelligence algorithms, and adaptive AI techniques to reconstruct the digital trace of handwriting. Our approach integrates hardware development, focusing on a sensor-equipped pen, with software innovations to optimize trajectory reconstruction and processing in real time using an embedded AI. This work aims to advance the state-of-the-art in automated trace reconstruction of handwriting, enabling a seamless connection between traditional handwriting on paper and capturing the trace digitally.

0.65

Magic Pen: Automatic Pen Mode Switching for Document Annotation

cs.HC ·2026-10-08 ·Kevin Desousa, Adam Bradley, Nathalie Henry Riche, Ken Hinckley, Christopher Collins
similarity 0.65
Abstract · Technical report. 18 pages, 18 figures, 3 tables

Traditional digital pen interfaces use menu buttons to change the pen mode, which results in time and cognitive load spent on round-trip interactions and mode errors from tapping small mode selection buttons. This work presents the Magic Pen, a technique which uses machine learning to automatically switch between digital pen modes without requiring explicit mode changes. Magic Pen is driven by an LSTM model trained on pen data collected from 27 participants across two studies and uses transfer learning to iteratively tune the model towards how a specific user annotates. Error mitigation techniques using a flick gesture or on-screen tap are incorporated to correct mode errors or remove a stroke quickly. We evaluated Magic Pen in a comparative study with 18 participants, followed by iterative improvements and a deployment study with 8 participants. Magic Pen was preferred compared to a conventional menu-based approach, and transfer learning allowed for greater model predictability and stability.

0.22

inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild

cs.CV cs.AI ·2026-10-28 ·Sidra Hanif
benchmarkcode/data similarity 0.62
Abstract

Human handwriting is produced as an ordered sequence of pen movements, but an offline image keeps only the final ink. Handwriting trajectory recovery aims to restore the ordered stroke points and their pen-up/pen-down transitions. Existing image-trajectory pair datasets are recorded with digitizing tablets under controlled conditions, which limits writer coverage and stylistic diversity. Moreover, there is no benchmark that measures the visual grounding capabilities of general-purpose Vision-Language Models (VLMs) for the stroke trajectory recovery task. In this work, we introduce inkGround, a benchmark for stroke trajectory recovery from full-page, camera-captured handwriting. inkGround pairs 537 page images (~18.7k words, ~163k strokes) with reference trajectories produced by a pretrained offline-to-online model, filtered by OCR agreement, and verified by human raters. Because these references are pseudo-labels that represent one plausible pen trajectory, we also evaluate VLMs on the IAM-OnDB dataset, which provides recorded pen trajectories. We cast the task as text-conditioned point grounding: given a page image and a target word, the model must locate the word and output its ordered stroke points. Outputs are scored by trajectory fidelity (DTW, Chamfer distance), readability of the rendered strokes (Accurate Rate, Correct Rate), and style similarity. We benchmark four open-source and three frontier closed-source VLMs with a shared in-context prompt. Closed-source models, notably Gemini-3.1-Pro, produce legible trajectories close to the reference, and this ranking also holds on IAM-OnDB.

0.26

Handwriting Trajectory Recovery via Autoregressive Ordered Stroke Instance Prediction

cs.CV ·2026-09-02 ·En-Guang Wang, Yan-Ming Zhang, Fei Yin, Cheng-Lin Liu
position similarity 0.62
Abstract · 24 pages, 6 figures, 7 tables

Handwriting trajectory recovery aims to infer the dynamic writing process hidden behind a static handwritten image. Since offline handwriting preserves only the final spatial ink pattern, temporal information such as stroke order, writing direction, and pen-tip motion is lost, making recovery inherently ambiguous. Existing learning-based methods often directly predict the complete character trajectory without explicitly exploiting the stroke-level organization of handwriting. We argue that recovering the writing process should follow the writing process itself. Accordingly, we propose a two-stage framework that first recovers ordered stroke instances and then reconstructs continuous within-stroke motion. The first stage integrates stroke extraction and stroke-order recovery through autoregressive ordered stroke prediction, while direction-related structural cues further support within-stroke trajectory generation. Experiments on Chinese handwriting show that the proposed ordered prediction is more effective than post-hoc stroke ordering. Even without trajectory simplification, our full-point model achieves numerically better results than those reported by all compared baselines, while a controlled analysis shows that trajectory sampling density substantially affects measured recovery performance. Additional experiments demonstrate generalization to unseen Chinese character categories and cross-language extensibility to English and Tamil handwriting.

0.41

Grip, Trace, Write: Force-Aware Imitation and Reinforcement Learning for High-DoF Robotic Handwriting

cs.RO ·2026-11-07 ·Rongqian Chen, Yiding Chen, Jijie Zhou, Tom Z. Jiahao
similarity 0.61
Abstract · This paper is accepted by CoRL 2026 Workshop PhysWM

Most robotic handwriting rigidly mounts the pen and reduces writing to trajectory tracking. A freely held pen instead couples grasp stability, tip motion, and contact force. We present Grip, Trace, Write, a force-aware simulation environment and learning framework for high-DoF handwriting. A two-branch actor separates pen retention from arm-wrist writing while preserving their closed-loop coupling. Training learns a grasp controller, initializes writing from inverse-kinematics demonstrations by behavior cloning, and improves writing through reinforcement learning with rewards on ink, force, time, and retention. Across diverse tasks, the policy improves completion, precision, force regulation, and efficiency over behavior cloning and inverse kinematics, and generalizes without fine-tuning to unseen English and Chinese content. These results establish force-aware handwriting as a testbed for dexterous tool use.

0.15

Cross-Domain Few-Shot Writer Adaptation for Real-World Handwritten Mathematical Expression Recognition

cs.CV ·2026-09-26 ·Paulo Grane Gabriel Silva, Lorenz Bernard Marqueses, Joel Ilao
published similarity 0.61
Abstract · Accepted at the 32nd IEEE International Conference on Mechatronics and Machine Vision in Practice (M2VIP 2026)

Handwritten mathematical expression recognition (HMER) refers to the task of recognizing and converting handwritten mathematics into a parsable markup language, usually LaTeX. No current state-of-the-art-competitive system adjusts to the way a specific person writes, and the domain gap between training images (usually digital or perfectly binarized) and images physically taken with a camera used in inference has received fairly little attention for this specific problem. We characterize this domain gap through fragmentation and stroke-width analyses of the images as well as introduce a writer-adaptive fine-tuning pipeline to MFH-CoMER in an attempt to address it. We further introduce sample author-specific datasets, consisting of five handwriting category subsets from two authors, and evaluate using a McNemar's test and permutation tests adapted to limited data. Results suggest an increase in expression recognition rate and a decrease in CER for digital handwriting but more varied results for physical handwriting, with the model struggling for handwriting articles that the base model can already evaluate well. Nonetheless, adapted models were found to have improved results for four out of five subsets. Statistical testing results suggest a consistency in improvement for two of the tested author-specific subsets. Our results point towards the potential feasibility of writer adaptation for the HMER task.

0.21

Segment-Level Risk Discovery in Online Handwriting for Alzheimer's Disease Detection

cs.CV cs.AI ·2026-09-24 ·Changqing Gong, Huafeng Qin, Mounîm A. El-Yacoubi
benchmark similarity 0.59
Abstract

Online handwriting provides a non-invasive and low-cost behavioral biomarker for Alzheimer's disease (AD) detection, as it reflects both cognitive planning and fine motor control. Existing handwriting-based AD detection methods usually rely on global trajectory features or whole-sample representations, which can be strongly affected by individual writing style, task-specific variation, and acquisition noise. In this paper, we propose NormPaST-Risk, a healthy-normative Paper-Air selective trajectory state-space risk network for interpretable AD detection from online handwriting. Instead of treating the entire trajectory as a single holistic representation, our method reformulates AD handwriting detection as local disease-relevant segment discovery. Specifically, a multi-scale temporal encoder captures stroke dynamics at different temporal resolutions, while a selective Paper-Air state-space encoder models long-range handwriting progression and distinguishes on-paper motor execution from in-air planning and transition behaviors. To explicitly characterize abnormal deviations, a healthy normative branch learns normal handwriting dynamics from healthy controls, and a task-aware multi-expert segment-risk module estimates segment-level AD risk calibrated by hidden-state changes and normative deviations. A weakly supervised segment-level objective further enables high-risk segment discovery without manual segment annotations. Experiments on the DARWIN benchmark demonstrate that the proposed framework achieves superior AD/HC classification performance compared with existing methods. Moreover, the discovered high-risk segments can be projected back to the original handwriting trajectory, providing interpretable evidence associated with AD-related handwriting variations.

0.24

Japanese Handwriting Generation Based on Human Preference Feedback Using Direct Preference Optimization

cs.CV ·2026-10-10 ·Wakana Kuwata, Ryota Mibayashi, Masanori Tani, Hiroaki Ohshima
published similarity 0.57
Abstract · Accepted for publication at BESC 2026

We propose a Japanese handwriting generation framework that integrates human preference feedback using Direct Preference Optimization (DPO) to align generative models with human cognitive and stylistic preferences. While font generation technologies leveraging Generative Adversarial Networks and diffusion models have advanced significantly, applying these approaches to handwriting generation presents unique challenges. Font generation is a deterministic process yielding consistent styles. Handwriting generation, on the other hand, is inherently non-deterministic. In handwriting, glyph shapes never match exactly even when the same writer writes the same character. We define this shape difference as handwriting variability. Although this variability is essential for conveying the natural quality of human writing, it complicates model training and substantially increases data collection costs to capture diverse distribution patterns. To address these limitations, we incorporate human perceptual feedback into the training loop of handwriting generation. While collecting extensive handwritten data is costly, evaluating pairwise preferences between two generated samples remains very easy for humans. Applying DPO, an efficient RL-free preference optimization method, to our generation model enables high-quality handwriting generation without relying on large-scale dataset collection. Experimental results across four distinct Japanese handwriting styles demonstrate that DPO-based training enhances overall structural accuracy. However, in highly complex styles with significant variability, such as brush pen writing, the model fails to capture style-specific features. These findings highlight a critical challenge in extending DPO to complex handwriting styles.

0.30