CS Paper Radar

Interpreting Classical Readability Formulae as Surprisal

Yo Ehara

cs.CL ·submitted 2026-10-31 23:58 UTC · v1

Abstract

Surprisal correlates closely with reading time and is widely used in the cognitive science of reading as a measure of processing difficulty. Classical readability formulae such as the Flesch-Kincaid Grade Level, which scores a text by the US school grade expected to read it, are the most widely used practical measures of text difficulty, but their coefficients are unitless regression constants and have never been related to surprisal. We theoretically show that these coefficients can be interpreted as surprisal: every formula of the classical additive shape is a sum of per-word and per-sentence cross-entropies under a memoryless model, and once that model is tied to the language through Zipf's law and the formula is read as its linearisation at a reference text, each coefficient factorises into a scale in grades per nat, a Zipf slope and a reference sentence length. A grade thereby becomes a quantity in surprisal. Because the formula outputs grades and children's vocabulary has been measured by grade, the interpretation recovers the vocabulary growth the formula implicitly assumes and lets it be checked against development: holding sentence length fixed, the word coefficient of the Flesch-Kincaid formula converts to 0.18 nat of unigram surprisal per word per grade (0.15 to 0.19 across the regression-derived formulae), and normative root-word vocabulary growth (Biemiller and Slonim, 2001), converted through Zipf's law, gives 0.16 nat per year at the frontier of the known vocabulary. The word coefficient is one year of vocabulary growth, and the grade is a vocabulary scale. The identification also yields a diagnostic (the sentence-length terms are not those of a memoryless surprisal) and extends to a Japanese formula.

Closest papers

Nearest neighbours by meaning across the whole index.

Revisiting the Estimation of Unigram Surprisal in Computational Psycholinguistics

cs.CL ·2026-10-13 ·Zachary Hopton, Francesco Ignazio Re, Samuel Kiegeland, Andreas Opedal, Eleanor Chodroff +1 more
top venue similarity 0.64
Abstract

Contextual and unigram surprisal are standard predictors of reading time in computational psycholinguistics. While contextual surprisal is typically derived from neural language models, unigram surprisal is usually obtained from independent corpus counts. Although both quantities are deterministic functionals of a single underlying distribution---capturing conditional and marginal structure respectively---this practice breaks their probabilistic link and may introduce estimation error from the mismatch. We propose three principles for estimating unigram surprisal in psycholinguistic research, including the requirement that both quantities be derived from the same language model. We operationalize this principle by introducing a Monte Carlo estimator of unigram surprisal from a language model, along with a marginalized variant that has lower variance empirically. When unigram surprisal is re-estimated under this unified framework, we find evidence that predictors change in their predictive contribution to reading time. We also confirm prior findings that frequency can be a stronger predictor of processing effort than contextual surprisal under this unified framework.

0.55

Flesch-Kincaid Readability Depends Only on the Topic Distribution in Long Texts under Topic Models

cs.CL ·2026-08-24 ·Yo Ehara
unusual similarity 0.61
Abstract

Flesch Reading Ease (FRE) and the Flesch-Kincaid Grade Level (FKGL) are widely used readability scores for English computed from the same two document statistics, yet their stability on long documents need not imply invariance to lexical composition. Surprisingly, under a topic model with an explicit sentence-boundary token, both scores converge almost surely to deterministic functions of the document topic distribution through just two scalar rates: in the long-text limit, all score variation is mediated by topical composition rather than any residual readability signal. The theory covers both formulae, while the experiments evaluate FKGL. In a fixed admixture with rank[1, q, s] = 3, fibres through interior topic vectors are locally (K-3)-dimensional, whereas regular iso-score level sets are locally (K-2)-dimensional and curved. In out-of-fold evaluation on two balanced corpora, Brown and the written BNC, a topic vector inferred from one document half's content words predicts the other half's FKGL at r = 0.779 and 0.884, respectively. On Brown, adding the topic prediction to genre and mean content-word syllable count yields $ΔR^2$ = 0.002, with a confidence interval spanning zero; on the BNC, the corresponding split-half increment is 0.024, positive in four of five K = 100 fits (median 0.021). Because inferred topics may also absorb genre, register, and style, we do not interpret these results as evidence about human readability or causal effects.

0.25

From Exposure to Expectation: Frequency, Surprisal, and Language Across Development in Spanish

cs.CL ·2026-08-23 ·Francisco Portillo López
similarity 0.60
Abstract · 30 pages; 4 figures; 5 tables

Surprisal, the negative log-probability a language model assigns to a word given its preceding context, reliably predicts adult reading times. Does it contribute as much to explaining when children acquire individual words? Frequency reflects a learner's cumulative exposure to a word, whereas surprisal reflects how predictable a single occurrence is given its context. We investigate this question across two corpus-based studies of Spanish. In Study 1, we modeled age of acquisition (AoA) for 225 Spanish nouns using lexical frequency and contextual diversity from child-directed speech, plus surprisal from three language models differing in architecture and training language (BETO, BERTIN, mGPT). Frequency strongly predicted AoA (r=-.597, p<.001); surprisal added little beyond frequency and word length, including in a naturalistic-context analysis. In Study 2, we modeled adult fixation durations in the Chilean Spanish subsample of the Multilingual Eye-movement Corpus (MECO Wave 2), using mGPT surprisal alongside two independent frequency measures. Surprisal robustly predicted longer fixation durations after controlling for frequency and word length, consistent across both frequency sources. A matched word-type-level comparison showed the surprisal-behavior association was stronger in reading than in acquisition (z=3.63, p<.001). The findings suggest cumulative lexical exposure and contextual predictability play different roles across the language trajectory: frequency is particularly informative about when early lexical representations are acquired, whereas surprisal captures moment-to-moment processing difficulty in an already-established linguistic system. We discuss this pattern in relation to usage-based and entrenchment-based accounts of lexical development and to the evaluation of language models as models of human language behavior.

0.34

Quantifying Meaning with SCAPE-T: Semantic Complexity from Encoder-Only Transformer Attention

cs.CL cs.AI ·2026-10-24 ·Maxim Khovansky, Lev Mukhanov
similarity 0.57
Abstract · 12 pages; semantic complexity, complexity estimation, encoder attention, attention analysis, pairwise accuracy, readability, text simplifica…

Automated semantic complexity estimation remains underexplored in NLP, despite important applications in linguistics and interpretability. This paper defines semantic complexity as the code length required to uniquely specify a text's semantic meaning, and operationalises it via SCAPE-T, an attention-derived metric based on encoder-only transformers. Existing measures either model reader effort (readability, lexical difficulty), which correlates with surface features (word length, syntax), or quantify surface features directly (type-token ratio, word frequency). SCAPE-T targets semantic integration directly by combining three submetrics: (i) inverse-weighted self-focus (token-to-self), (ii) CLS focus (token-to-CLS), and (iii) inter-head redundancy (cosine similarity across heads), using layer-weighted aggregation; data-driven learning is confined to hyperparameter tuning for calibration. Development uses philosophical prose vs. MCTest stories; evaluation is on children's stories vs. technical Wikipedia articles, with pairwise accuracy on contrasts as the primary criterion. Baselines include readability, syntactic/lexical, and LLM-derived measures. SCAPE-T substantially outperforms all tested baselines under word-length control, with a pairwise accuracy of 78.5% (95% bootstrap CI: 76.1-80.4, N=10k pairwise comparisons; +12.7 percentage points over the strongest baseline); without that control, readability heuristics overtake, consistent with their reliance on surface features. On a short-terms abstractness dataset, SCAPE-T also leads. Ablations show each submetric contributes materially. Overall, the results show SCAPE-T isolates conceptual structure from lexical/syntactic confounds better than existing methods, and generalises across two held-out domains.

0.26

A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese

cs.CL cs.AI ·2026-10-04 ·Hongao Zhu, Muxiaoqiao Xu, Yikang Liu, Siyuan Song, Yuxia Wang +2 more
similarity 0.56
Abstract · 17 pages, 3 figures

This study analyzes the predictive power of LM-derived, token-level surprisal on Mandarin Chinese reading times. Using a suite of Chinese-Pythia models (14M--1.4B) trained from scratch with 30B tokens, we examine how well surprisal predicts first fixation duration, gaze duration, and total reading time in three paragraph-level eye-tracking corpora of Mandarin Chinese (GECO-CN, HKP, and MECO). Contrary to previous null findings, our results show that surprisal is predictive of Chinese reading times. However, whether predictive power scales with model size and the amount of training is corpus-specific: bigger models predict reading times better in GECO-CN, whereas inverse scaling emerges in HKP and, at the largest sizes, in MECO. Subsequently, we find that checkpoints whose surprisal remains closer to $n$-gram statistics are better predictors of reading in HKP and MECO, providing one possible explanation for the inverse scaling behavior. All in all, the predictive power of surprisal on Chinese reading time measurements is corpus-specific, which cautions against drawing conclusions about language model scale from a single corpus.

0.14

The Cognitive Cost of LLM Writing: A Reading-Time Comparison with Human Text

cs.CL ·2026-10-16 ·Kamer Cekini, Elisabetta Biondi, Chiara Boldrini, Andrea Passarella, Marco Conti
similarity 0.55
Abstract

Although linguistic differences between human-written and Large Language Model (LLM)-generated texts have been extensively studied, their implications for the cognitive effort involved in reading remain less explored. In psycholinguistics, reading time is commonly used as an indirect measure of cognitive processing effort. Building on this perspective, we investigate whether human-written texts and their LLM-generated rewrites exhibit differences in predicted reading time that may reflect differences in processing demands for readers. To this end, we introduce RoBERTa-RT, a model trained on human eye-tracking data to predict word-level reading times. We evaluate the model on both internal and external test sets, construct a paired dataset of 496 human-written articles and their LLM-generated rewrites, and apply RoBERTa-RT to compare their predicted reading times. Our findings show that LLM-generated rewrites tend to be shorter and have slightly lower predicted total reading times, while exhibiting higher predicted reading times per word. These differences are associated with systematic linguistic changes, including longer and less frequent words, greater lexical diversity, and higher contextual predictability. From a cognitive perspective, our findings suggest that LLM-generated writing may redistribute reading demands: reducing the amount of text to process while increasing the predicted processing effort associated with individual words. This highlights the importance of considering not only the linguistic characteristics of machine-generated content, but also its potential implications for human cognitive processing.

0.12

Estimating Surprisal from the Speech Signal

cs.CL ·2026-10-13 ·Zachary Hopton, Samuel Kiegeland, Francesco Ignazio Re, Patrick Haller, Matthew B. Winn +2 more
top venue similarity 0.54
Abstract

Although language is predominantly spoken, empirical tests of surprisal theory rely almost entirely on written text. In practice, surprisal is estimated using language models (LMs) trained on text and applied to transcriptions, creating a text bottleneck that limits evaluation of the theory in the spoken modality. We remove this bottleneck by estimating surprisal directly from the speech signal using a speech LM. This raises a key question: should surprisal be defined over realization-specific acoustic inputs or over abstract linguistic units? Since a single unit can have many distinct acoustic realizations, each assigned a different probability by the model, defining surprisal over abstract units is difficult. To address this, we introduce an importance sampling estimator that approximates the probability of an abstract unit by marginalizing over realizations generated by a text-to-speech model. Using pupillometry data from a listening study, we evaluate surprisal theory in this text-free setting at multiple levels of granularity. Obtaining surprisal from several speech LMs, we consistently find that surprisal estimated via importance sampling better predicts pupil dilation than realization-specific surprisal for words. We also find that surprisal values for sub-lexical units are predictive of pupil dilation. While we find that surprisal can be estimated without recourse to text, and that marginalizing over distinct realizations of words is more predictive of lexical processing effort, surprisal as a whole is not a strong predictor of change in pupil size, regardless of its source.

0.34

Reader Proficiency Shapes Layer-wise Surprisal Profiles

cs.CL ·2026-09-29 ·Akio Hayakawa, Horacio Saggion
top venue similarity 0.54
Abstract

Reading behaviour varies not only with linguistic input, but also with reader proficiency. In this study, we investigate whether the layer-wise relationship between surprisal from large language models (LLMs) and human gaze behaviour differs across readers with different levels of proficiency and across gaze measures. Using eye-tracking data from the MECO L2 corpus, we compare readers with high and low vocabulary proficiency on first-pass gaze duration (FPGD) and total gaze duration (TGD). We quantify the distribution of the predictive power of surprisal across model layers using Predictive Depth. Across 12 tested LLMs, we find that readers with lower vocabulary proficiency tend to show deeper Predictive Depth for FPGD, while this difference is smaller for TGD. Also, TGD itself shows deeper Predictive Depth than FPGD in both proficiency groups. These patterns suggest that where predictive power is concentrated across LLM layers may be related to the timing and breadth of the reading processes captured by different gaze measures, and that this relationship can vary with reader proficiency. Our leave-one-out analysis further shows that the advantage of informative internal layers extends to unseen texts, although the practical improvements in prediction are limited. Overall, our results show that layer-wise LLM surprisal provides a useful perspective on variation in reading behaviour across both reader groups and gaze measures.

0.36