CS Paper Radar

Flesch-Kincaid Readability Depends Only on the Topic Distribution in Long Texts under Topic Models

Yo Ehara

cs.CL ·submitted 2026-08-24 14:41 UTC · v1
unusual

Abstract

Flesch Reading Ease (FRE) and the Flesch-Kincaid Grade Level (FKGL) are widely used readability scores for English computed from the same two document statistics, yet their stability on long documents need not imply invariance to lexical composition. Surprisingly, under a topic model with an explicit sentence-boundary token, both scores converge almost surely to deterministic functions of the document topic distribution through just two scalar rates: in the long-text limit, all score variation is mediated by topical composition rather than any residual readability signal. The theory covers both formulae, while the experiments evaluate FKGL. In a fixed admixture with rank[1, q, s] = 3, fibres through interior topic vectors are locally (K-3)-dimensional, whereas regular iso-score level sets are locally (K-2)-dimensional and curved. In out-of-fold evaluation on two balanced corpora, Brown and the written BNC, a topic vector inferred from one document half's content words predicts the other half's FKGL at r = 0.779 and 0.884, respectively. On Brown, adding the topic prediction to genre and mean content-word syllable count yields $ΔR^2$ = 0.002, with a confidence interval spanning zero; on the BNC, the corresponding split-half increment is 0.024, positive in four of five K = 100 fits (median 0.021). Because inferred topics may also absorb genre, register, and style, we do not interpret these results as evidence about human readability or causal effects.

Closest papers

Nearest neighbours by meaning across the whole index.

Interpreting Classical Readability Formulae as Surprisal

cs.CL ·2026-10-31 ·Yo Ehara
similarity 0.61
Abstract

Surprisal correlates closely with reading time and is widely used in the cognitive science of reading as a measure of processing difficulty. Classical readability formulae such as the Flesch-Kincaid Grade Level, which scores a text by the US school grade expected to read it, are the most widely used practical measures of text difficulty, but their coefficients are unitless regression constants and have never been related to surprisal. We theoretically show that these coefficients can be interpreted as surprisal: every formula of the classical additive shape is a sum of per-word and per-sentence cross-entropies under a memoryless model, and once that model is tied to the language through Zipf's law and the formula is read as its linearisation at a reference text, each coefficient factorises into a scale in grades per nat, a Zipf slope and a reference sentence length. A grade thereby becomes a quantity in surprisal. Because the formula outputs grades and children's vocabulary has been measured by grade, the interpretation recovers the vocabulary growth the formula implicitly assumes and lets it be checked against development: holding sentence length fixed, the word coefficient of the Flesch-Kincaid formula converts to 0.18 nat of unigram surprisal per word per grade (0.15 to 0.19 across the regression-derived formulae), and normative root-word vocabulary growth (Biemiller and Slonim, 2001), converted through Zipf's law, gives 0.16 nat per year at the frontier of the known vocabulary. The word coefficient is one year of vocabulary growth, and the grade is a vocabulary scale. The identification also yields a diagnostic (the sentence-length terms are not those of a memoryless surprisal) and extends to a Japanese formula.

0.25

Beyond Model Bias: Cross-Entropy Based Rankings in Literary and Academic Texts Remain Stable across Small Language Models

cs.CL ·2026-10-12 ·Ankit Bhattacharjee, Indrasabarna Hari, Krityapriya Bhaumik, Arka Das, Piyush Sarangi
similarity 0.60
Abstract

While language models provide powerful mechanisms for evaluating textual predictability, it remains unclear whether the relative extents of LM surprisal remain stable across models. This study presents empirical evidence of the invariance of ranking based on cross-entropy related measures (e.g. mean, variance of each data-point), across four distinct small scale transformer-based models, for a given dataset of text sources. We consider two stylistic environments: a corpus of contemporary poetry and a corpus of academic research papers. We demonstrate that mean and variance of cross-entropy maintain robust cross-model rank correlation ($ρ> 0.90$) across both regimes. To test whether this agreement merely reflects generic differences in textual complexity, we compare it against six parameter-free compression baselines. The extent of agreement among language models is found to be substantially higher than that among model-baselines, for most of the measures. Geometric analysis of the structurally constrained academic regime reveals that higher-order moments are sensitive to specific models, although strong agreement persists. The findings suggest that relative extents of textual unpredictability may be independent of the language model they are computed against.

0.27

From Word Counts to Context: Topic Models for Asset Pricing

cs.CE q-fin.PMq-fin.ST ·2026-09-28 ·Kevin Foley, Jonathan Hartadi, Shivesh Prakash, Swapnil Vatsal
benchmarkcode/data similarity 0.55
Abstract · Project page: shivesh777.github.io/bert_lda

News may reveal systematic risk, but whether its context enhances the construction of systematic risk factors is still unclear. We seek to test whether utilizing a sentence transformer represents an improvement over techniques such as Latent Dirichlet Allocation (LDA) in the coherence of topic term lists generated from unstructured text data. To test this, the same collection of unstructured text data comprising of 394,661 articles and the same downstream financial portfolio construction pipeline were applied with the text layer differing, including the length of article text each model used and how topic terms were ranked: we benchmark LDA against a frozen sentence transformer with k-means clustering. We find that the sentence transformer branch had higher observed scores both in terms of coherence (measured by NPMI) as well as financial performance (measured by Sharpe), although the available tests do not establish outperformance. Further exploratory specifications such as utilizing spherical clustering and multi-horizon exposures had an observed excess-return Sharpe of 1.03 for the combined model. We believe that there is some promise in applying context-aware techniques on unstructured news text, but stricter tests using only information available at each date and broader datasets may be required to enhance the confidence in the observed performance.

0.57

vMF Sentence LDA: A Spherical Topic Model over Sentence Embeddings

cs.CL cs.LG ·2026-10-04 ·Ryotaro Kobayashi, Yuri Murayama, Kiyoshi Izumi
similarity 0.55
Abstract

Latent Dirichlet Allocation (LDA) and models derived from it remain widely used topic models. LDA observes each document as a bag-of-words and models each topic by a categorical distribution over the vocabulary, so that it uses neither the internal structure of the document nor the similarity in meaning between words. Earlier work has responded to this limitation in two ways: many models have introduced embeddings, and some have assigned topics to sentences rather than to words. Their combination, a topic model that observes sentence embeddings, remains little explored. We propose vMF Sentence LDA (vSLDA), which keeps the admixture structure of LDA, observes each sentence as its L2-normalized embedding and models each topic by a von Mises-Fisher (vMF) distribution, which matches the cosine geometry of sentence embeddings. Its per-topic parameter count and per-iteration cost are linear in the embedding dimension, versus quadratic for the full-covariance Gaussian distribution in the existing model over sentence embeddings. We evaluate vSLDA where the limitation is expected to matter most, among topics that share much of their vocabulary: the topics that subdivide the one subject of a collection, and the narrow topics that result when a corpus is divided into a large number of topics. On two corpora, vSLDA attains the best mean rank against eight baselines when the fine categories within each coarse category are classified from the document-topic distributions. On the whole corpus, its advantage appears or widens as the number of topics grows. Weighting the word frequencies of each sentence by its topic posterior yields expected topic-word counts of the same form as those of LDA, so that the standard topic coherence and diversity measures apply to models that assign topics to sentences. On their product, topic quality, vSLDA leads in most within-category conditions of both corpora.

0.17

A Quantitative Approach to Asch Conformity in Frontier Large Language Models

cs.CL cs.AI ·2026-10-27 ·Senthil Rajasekaran, Stylianos Loukas Vasileiou
similarity 0.55
Abstract

The Asch conformity experiment has recently gained considerable attention as a way of measuring conformity in large language models (LLMs). While previous studies have focused on demonstrating conformity in LLMs, our work adds a further quantitative layer to this approach. Instead of sampling a frontier LLM on a suite of questions, our approach samples a single question repeatedly, leveraging the probabilistic nature of LLMs to produce a distribution of answers. In doing so, we can compare the distributions of answers before and after the model is presented with the manufactured consensus. Our experiments confirm that frontier LLMs do conform, reinforcing prior literature, and, through our distribution-to-distribution comparisons, provide additional quantitative data on the conditions that lead to conformity and the degree to which it occurs.

0.20

Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure

cs.CL ·2026-09-20 ·Shuyang Xiang
similarity 0.54
Abstract

Standard positional encodings treat position as a one-dimensional reading-order coordinate, but reading order alone does not determine hierarchical textual structure. We use a hierarchical rotary positional encoding (hRoPE) that represents paragraph, sentence, and token indices as separate channels, hold the token sequence fixed, intervene on the paragraph coordinate $p_1$, and measure cross-paragraph attention with a token-distance-exact estimator. Attention is compressed in every corpus, but compression alone is not diagnostic of true structure: an architecturally identical channel with density-matched random labels is compressed too. What distinguishes real structure is the depth of compression: the real-versus-random depth gap is resolvable in two of three corpora and not in the third, and real-structure depth varies more strongly across corpora than the control's. Comparing eight corpus-only quantities across three constructs (lexical persistence, paragraph length, embedding-based coherence), none fully reproduces the cross-corpus ordering of depth, though embedding-based coherence comes closest. At the paragraph level that relation transfers as a common slope, but with the opposite sign to the corpus-level ranking: controlling for length, diversity, and position, paragraphs with more similar neighbors compress less deeply, with no detectable slope difference between any pair of corpora, while length, lexical-diversity, and position effects remain corpus-specific. Compression depth, not its location, is the reproducible signature of genuine paragraph structure in our setting.

0.34

A Measurement Perspective on Benchmarking LLM-Generated Text Detectors using Item Response Theory

cs.CL ·2026-10-18 ·Yanbin Zhao, Zhipeng Chen, Guangsheng Bao, Guandong Zhang, Yan Li
benchmark similarity 0.54
Abstract

Benchmarking large language model (LLM)-generated text detectors commonly relies on aggregate detector-level metrics such as accuracy, F1 score, and AUC. Although useful for summarizing overall performance, these metrics provide limited insight into how benchmark composition shapes detector comparisons or which texts distinguish detectors with different response profiles. We propose a measurement-based benchmarking framework for jointly characterizing benchmark heterogeneity and detector behavior using multidimensional item response theory (IRT). By treating benchmark texts as measurement items and detectors as respondents, the framework estimates text-level discrimination and difficulty-location parameters alongside multidimensional detector abilities. We apply the framework to a benchmark of 32K human-written and machine-generated texts across four domains, evaluated by 14 detectors. The estimated IRT item-parameter space reveals distinct but overlapping human-machine distributions, with substantial variation across domains and text lengths. The human-machine label shows the strongest multivariate association with this space, while domain and text length further contribute to benchmark heterogeneity. Detector ability profiles reveal multidimensional and domain-dependent response patterns that are not fully reflected in aggregate performance rankings. As a practical application, clustering texts in the estimated IRT space yields a compact subset of 2,088 texts (6.5\% of the full 32K benchmark) that largely preserves the comparative detector-performance patterns observed on the full benchmark. Overall, the proposed framework provides a principled and interpretable methodology for benchmarking heterogeneous detection technologies while jointly characterizing the measurement properties of the texts used for their evaluation.

0.42

Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models

cs.CL cs.CRcs.LG ·2026-09-09 ·Arman Nik Khah
code/data similarity 0.53
Abstract · Corrects errors in v1: the composition of the mid-book non-members (Sec. 3.3, new App. B), the direction of the Sec. 7 mechanism (the compos…

When a language model finds a sentence unusually cheap to predict, it is tempting to conclude that the sentence was in its training data. Almost every published test of that inference has had to guess which sentences were in the training data, the members, and which were not. This paper removes the guessing. Two model families, OLMo-2 and Pythia, publish their pretraining corpora, and a public index over those corpora returns the exact number of times any sentence appeared in each. Those counts make three questions answerable directly. The answers form a pincer, closing from two sides. At the duplication levels ordinary text actually has, five models from 1B to 13B parameters carry at most a faint trace of their own exposure. We measure that trace with a design that reads the same sentence through two models, which cancels fluency and quality by construction, and it comes to a rank correlation near -0.08, where -1 would be a perfect relation and 0 none. Where the trace does become strong, above roughly a thousand copies, the two corpora agree on which sentences those are, because they are the famous ones, so exposure can no longer be told apart from fame. Two further measurements show how apparent membership signal gets manufactured. A common way to build a non-member is to change one word of a member. The model does prefer the original, but the gap is the same, within noise, whether the original appeared once or a hundred times, so what the model is rewarding is the author's word choice, not memory. Above a thousand copies the gap grows with model size on the twelve sentences we can test there, at the same boundary where the pincer closes. And swapping the controls for sentences that differ from the members in register moves a detector from 0.83 to 0.94 AUC, on a scale where 0.5 is a coin flip and 1.0 is perfect separation. We release the sentence banks, counts, and code.

0.35