CS Paper Radar

The Depth of a Prediction: Where Language Models Resolve a Word Predicts Human Reading Time

Donato Riccio, Fabrizio Maturo

cs.CL cs.LG ·submitted 2026-11-19 11:53 UTC · v1

Abstract

Surprisal from a language model's final layer predicts how long people read a word, yet the model computes its prediction across many layers. We read each word's prediction at every layer with the tuned lens and summarize this layerwise surprisal trajectory by its resolution depth, the depth around which its change is centered. On eight English eye-tracking and self-paced reading corpora (81,957 words) and ten GPT-2, OPT and Pythia models, words that a model resolves later, and the words after them, are read more slowly than equally surprising words. Resolution depth improves held-out prediction beyond final-layer surprisal, entropy and the best single layer by 0.26 millinats per word (pooled p = 0.001), about a third of entropy's gain, and each model family gains, GPT-2 and Pythia through one corpus each. The net gain varies between corpora (random-effects mean 0.07, p = 0.37); against layer-shuffled depth, which adds as many coefficients, it is positive in seven of eight corpora (random-effects mean 0.23, p = 0.016), and untrained networks show no gain under the logit lens. Pooled over corpora, the effect holds with corrected word probabilities and beyond first-to-last-layer divergences, and a whole-trajectory functional regression adds no significant information beyond resolution depth.

Comment
17 pages, 4 figures, 7 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Reader Proficiency Shapes Layer-wise Surprisal Profiles

cs.CL ·2026-09-29 ·Akio Hayakawa, Horacio Saggion
top venue similarity 0.71
Abstract

Reading behaviour varies not only with linguistic input, but also with reader proficiency. In this study, we investigate whether the layer-wise relationship between surprisal from large language models (LLMs) and human gaze behaviour differs across readers with different levels of proficiency and across gaze measures. Using eye-tracking data from the MECO L2 corpus, we compare readers with high and low vocabulary proficiency on first-pass gaze duration (FPGD) and total gaze duration (TGD). We quantify the distribution of the predictive power of surprisal across model layers using Predictive Depth. Across 12 tested LLMs, we find that readers with lower vocabulary proficiency tend to show deeper Predictive Depth for FPGD, while this difference is smaller for TGD. Also, TGD itself shows deeper Predictive Depth than FPGD in both proficiency groups. These patterns suggest that where predictive power is concentrated across LLM layers may be related to the timing and breadth of the reading processes captured by different gaze measures, and that this relationship can vary with reader proficiency. Our leave-one-out analysis further shows that the advantage of informative internal layers extends to unseen texts, although the practical improvements in prediction are limited. Overall, our results show that layer-wise LLM surprisal provides a useful perspective on variation in reading behaviour across both reader groups and gaze measures.

0.36

A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese

cs.CL cs.AI ·2026-10-04 ·Hongao Zhu, Muxiaoqiao Xu, Yikang Liu, Siyuan Song, Yuxia Wang +2 more
similarity 0.68
Abstract · 17 pages, 3 figures

This study analyzes the predictive power of LM-derived, token-level surprisal on Mandarin Chinese reading times. Using a suite of Chinese-Pythia models (14M--1.4B) trained from scratch with 30B tokens, we examine how well surprisal predicts first fixation duration, gaze duration, and total reading time in three paragraph-level eye-tracking corpora of Mandarin Chinese (GECO-CN, HKP, and MECO). Contrary to previous null findings, our results show that surprisal is predictive of Chinese reading times. However, whether predictive power scales with model size and the amount of training is corpus-specific: bigger models predict reading times better in GECO-CN, whereas inverse scaling emerges in HKP and, at the largest sizes, in MECO. Subsequently, we find that checkpoints whose surprisal remains closer to $n$-gram statistics are better predictors of reading in HKP and MECO, providing one possible explanation for the inverse scaling behavior. All in all, the predictive power of surprisal on Chinese reading time measurements is corpus-specific, which cautions against drawing conclusions about language model scale from a single corpus.

0.14

The Cognitive Cost of LLM Writing: A Reading-Time Comparison with Human Text

cs.CL ·2026-10-16 ·Kamer Cekini, Elisabetta Biondi, Chiara Boldrini, Andrea Passarella, Marco Conti
similarity 0.64
Abstract

Although linguistic differences between human-written and Large Language Model (LLM)-generated texts have been extensively studied, their implications for the cognitive effort involved in reading remain less explored. In psycholinguistics, reading time is commonly used as an indirect measure of cognitive processing effort. Building on this perspective, we investigate whether human-written texts and their LLM-generated rewrites exhibit differences in predicted reading time that may reflect differences in processing demands for readers. To this end, we introduce RoBERTa-RT, a model trained on human eye-tracking data to predict word-level reading times. We evaluate the model on both internal and external test sets, construct a paired dataset of 496 human-written articles and their LLM-generated rewrites, and apply RoBERTa-RT to compare their predicted reading times. Our findings show that LLM-generated rewrites tend to be shorter and have slightly lower predicted total reading times, while exhibiting higher predicted reading times per word. These differences are associated with systematic linguistic changes, including longer and less frequent words, greater lexical diversity, and higher contextual predictability. From a cognitive perspective, our findings suggest that LLM-generated writing may redistribute reading demands: reducing the amount of text to process while increasing the predicted processing effort associated with individual words. This highlights the importance of considering not only the linguistic characteristics of machine-generated content, but also its potential implications for human cognitive processing.

0.12

Prediction Dynamics in Depth-Recurrent Language Models

cs.CL cs.LG ·2026-09-18 ·Xinyue Luo, Fei Yu
similarity 0.63
Abstract

Depth-recurrent language models refine predictions through repeated latent updates. Why can intermediate answers agree with the endpoint while their scores continue to change? We derive a sharp margin characterization that decomposes the conservatism of a magnitude bound into common translation, direction relative to the winner, and the pairing of each competitor's update with its score gap. Across Huginn-3.5B and Ouro-1.4B, accounting for update direction and competitor pairing reduces the mean earliest qualifying depth by a further 22.5-34.4% of the total depth beyond translation removal under full answer-text scoring. This retrospective comparison uses completed trajectories. Substantial contributions also occur under label scoring. For shared predictive distributions, we separate common and contrast motion orthogonally and express the common component through candidate-set mass and within-set concentration. Common and contrast energies can attenuate at different rates, allowing a growing preference-change share to coexist with shrinking absolute updates. These findings explain finite-depth answer preservation through the geometry and composition of observed score changes.

0.47

Reading Time Modeling with Pixel-Based Language Models

cs.CL ·2026-10-19 ·Junlin Li, Sara Alhajeri, Tatsuki Kuribayashi
top venue similarity 0.62
Abstract · accepted by EMNLP 2026 (Main)

Tokenization-free, pixel-based language models (PLMs) have recently been proposed as an alternative to conventional token-based language models (TLMs), typically motivated by practical considerations, e.g., the flexibility to support new languages or vocabularies. Nevertheless, PLMs also have the potential to serve as infrastructure for computational (psycho)linguistics, where tokenization has been a source of bias in estimating information-theoretic values, such as surprisal, and has pushed the bottom-up visual orthographic features outside the scope. In this study, we first integrate PLMs into the modeling of naturalistic human reading and, in particular, examine their use to estimate next-word predictability, which can also integrate orthographic and parafoveal information. Our results demonstrate that next-word prediction difficulty, computed in pixel space, has strong predictive power for gaze duration and word skipping behavior across various languages. In particular, their advantage is complementary to existing token-level surprisal and entropy estimates. Cross-lingual analyses with script variations also clarify their distinctive visual processing biases, facilitating research into how bottom-up visual information is incrementally integrated into expectation-based reading.

0.32

Revisiting the Estimation of Unigram Surprisal in Computational Psycholinguistics

cs.CL ·2026-10-13 ·Zachary Hopton, Francesco Ignazio Re, Samuel Kiegeland, Andreas Opedal, Eleanor Chodroff +1 more
top venue similarity 0.62
Abstract

Contextual and unigram surprisal are standard predictors of reading time in computational psycholinguistics. While contextual surprisal is typically derived from neural language models, unigram surprisal is usually obtained from independent corpus counts. Although both quantities are deterministic functionals of a single underlying distribution---capturing conditional and marginal structure respectively---this practice breaks their probabilistic link and may introduce estimation error from the mismatch. We propose three principles for estimating unigram surprisal in psycholinguistic research, including the requirement that both quantities be derived from the same language model. We operationalize this principle by introducing a Monte Carlo estimator of unigram surprisal from a language model, along with a marginalized variant that has lower variance empirically. When unigram surprisal is re-estimated under this unified framework, we find evidence that predictors change in their predictive contribution to reading time. We also confirm prior findings that frequency can be a stronger predictor of processing effort than contextual surprisal under this unified framework.

0.55

Emergent Inverse-Depth Scaling From Nonlinearity In Attention

cs.LG cs.AI ·2026-10-08 ·Zirui Peng, Yizhou Liu, Ziming Liu, Jeff Gore
theory similarity 0.61
Abstract · 30 pages, 14 figures, 5 tables

Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.

0.26

Lost in Read-out: Locating the Image Quality Gap in Vision-Language Models with a Closed-Form Edit

cs.CV ·2026-11-17 ·Zhuangzi Li, Shilv Cai, Jian Jin, Zhiyi Pan, Haihong Xiao +1 more
hot direction similarity 0.60
Abstract

Vision-language models (VLMs) are widely used as zero-shot image quality assessors, yet their good-vs-poor answers agree only moderately with human opinion. We ask whether these models fail to see quality or fail to say it. Layer-wise probes on five VLMs point to the second: quality is linearly decodable from the vision encoder and still from the last language layer, far better than the answer reveals, and about a hundred labels recover most of it. The loss lies mainly in the final read-out, which partly reads something else: relative to human opinion, it over-rewards colorfulness and under-rewards sharpness on all five models. We test this by intervention: if the read-out is at fault, a closed-form edit of a single last-layer neuron, fitted to many prompts at once, should close the gap and no more. Both hold. Under prompts never used for fitting, the edit recovers 86--100\% of what a ridge head on the last-layer state, fitted under the same prompts, finds beyond the answer on KonIQ. Where that gap is small, it gains little, too little for our transfer criterion on the two largest models; where the head itself is below the answer, it lowers the score; and it works only when written from the middle of the language model on. Unlike unconstrained fine-tuning, the edit leaves perception question answering intact. It does not carry over to yes-or-no or rating questions, and it reverses the good--poor contrast after sentiment-laden sentences. Code will be made available.

0.31