CS Paper Radar

Contributions to the hierarchy of probabilistic languages

Lothar Sebastian Krapp, Remo Nitschke

cs.FL cs.CLmath.LOmath.PR ·submitted 2026-09-20 11:35 UTC · v1

Abstract

We reconsider the theory of probabilistic formal languages generated by n-gram models and by probabilistic context-free grammars (PCFGs). The expected hierarchy of probabilistic grammars is established by proving that every probabilistic language generated by an n-gram model is also generated by some PCFG, while some probabilistic languages generated by PCFGs cannot be generated by any $n$-gram model. We introduce the notion of fully connected PCFGs, namely PCFGs in Chomsky normal form where every production rule only involving non-terminals has non-zero probability. Our main result shows that any probabilistic language generated by an $n$-gram model differs from any probabilistic language generated by a fully connected PCFG. Therefore, the class of probabilistic languages generated by $n$-gram models is not a subset of the class generated by fully connected PCFGs.

Closest papers

Nearest neighbours by meaning across the whole index.

Non-Terminal Complexity of Simple Semi-Conditional Grammars

cs.FL math.LO ·2026-09-12 ·Henning Fernau, Sanjay Jain, Linus Richter, Frank Stephan, Dan Turetsky
theory similarity 0.62
Abstract · 26 pages

We study the complexity of simple semi-conditional grammars (SSCGs) in terms of the number of their terminals and non-terminals. We show that SSCGs with three non-terminals can generate all RE languages; two non-terminals suffice for linear languages; and one suffices for unary regular languages. We also determine both upper bounds and fundamental limitations of SSCGs: while certain one-non-terminal SSCGs can generate non-context-free languages, every language generated by a one-non-terminal SSCG is in non-deterministic linear space. We also prove that there exists a regular language of alphabet size at least 3 which cannot be generated by any one-non-terminal SSCG. Finally, we prove that membership testing for SSCGs of two non-terminals is already NP-hard.

0.16

A Data-free Universal Prior over Syntactic Structures

cs.CL cond-mat.dis-nn ·2026-09-15 ·Fermín Moscoso del Prado Martín
similarity 0.59
Abstract · 30 pages, 4 figures

The probabilities of syntactic structures in human languages are assumed to emerge fully from language-specific experience. Here, I show that a universal prior over syntactic structures emerges from a model of human language production, in which words are progressively integrated into syntactic structure. Without fitting any parameters to specific language data, the resulting prior assigns higher probabilities to attested than to random dependency trees in all 138 typologically diverse languages examined. These prior probabilities correlate positively with those estimated from corpora in 33 of 34 languages. The results indicate that part of the probability structure of syntax can arise independently of language-specific learning. This identifies human language production as a possible cognitive source of universal statistical structure in language, while providing a data-independent structural bias for probabilistic models, including large language models.

0.21

On the Kanazawa--Salvati Conjecture

cs.FL ·2026-09-12 ·Takao Yuyama
theory similarity 0.57
Abstract · 39 pages

The language $\mathrm{MIX}$ consists of all words over a three-letter alphabet that have an equal number of occurrences of each letter. It is also the word problem of $\mathbb{Z}^2$ with respect to a suitable choice of generators. The Kanazawa--Salvati conjecture states that $\mathrm{MIX}$ is not a well-nested multiple context-free language. Every well-nested multiple context-free language is an indexed language. We reduce the conjecture to an explicit combinatorial problem about tuples of words, which is easier to state than the original formulation in terms of arbitrary well-nested multiple context-free grammars. More generally, for every surjective monoid homomorphism $ψ\colon Σ^* \to \mathbb{Z}^d$, we define a family of well-nested multiple context-free grammars $G_ψ[r]$ for $r \geq 1$, each of which generates a sublanguage of $ψ^{-1}(\mathbf{0})$. We prove that every well-nested multiple context-free sublanguage of $ψ^{-1}(\mathbf{0})$ is contained in $L(G_ψ[r])$ for some $r \geq 1$. Using this family, we prove that the four-letter analogue $\mathrm{MIX}_4$, which is a word problem of $\mathbb{Z}^3$, is not a well-nested multiple context-free language. The proof reduces this claim to a result of Bishop--Elder--Evetts--Gallot--Levine stating that the two-letter analogue $\mathrm{MIX}_2$ is not generated by any non-branching multiple context-free grammar. The Kanazawa--Salvati conjecture itself remains open.

0.27

A strengthening of the MCFL-ness of $O_2$

cs.FL cs.AIcs.LOmath.LO ·2026-08-19 ·Marco B. Caminati
similarity 0.57
Abstract · MCU 2026

In the last years, a number of proofs of the fact that $O_2$ is a multiple context-free grammar (MCFG) were given. Such results can be exploited in the fields of both computational linguistics and of computational algebra. Here, we focus on a recent such proof spelled in terms of factorizations of string tuples, and give a new result with a stronger characterization of such factorizations than in existing theorems.

0.24

On the equivalence between generating functions computed by memory transducers and enumerating functions produced by indexed grammars

cs.FL ·2026-09-04 ·Vincent Ghigo
theory similarity 0.56
Abstract

We consider the sequences of natural integers that can be computed by a deterministic transducer, with input in a structure A, output in N. and with memory the set of stacks of stacks of A. We show that these sequences are, exactly, the counting sequences of formal languages generated by unambiguous context-free indexed grammars (equivalently, the counting sequences of derivation trees of arbitrary context-free indexed grammars), with indexes in A. This general theorem applies, notably, to the set of natural integers endowed with the operation -1 and the non-zero predicate, showing that the polynomial recurrences count exactly the index-languages (where the parameter used for counting is the index itself).

0.25

Formal Grammars and Generating Functions: From Algebraic Structure to Non-Algebraic and Non-Derivable Series

cs.FL ·2026-10-22 ·Zhijun Wang
theory similarity 0.56
Abstract

This paper investigates how the algebraic and analytic properties of ordinary length generating functions constrain the grammatical and computational complexity of formal languages. We review the Chomsky-Schutzenberger representation, DSV translations for unambiguous context-free grammars, and their stack-indexed extensions, distinguishing finite algebraic systems from more general constructions. Examples and a diagonal argument demonstrate that even a rational generating function need not determine a language's grammatical class. Our central computability result establishes that, for a recursively enumerable language, computability of its length-counting sequence is equivalent to decidability of membership. Consequently, if a recursively enumerable language has a D-finite length generating function over Q(z), it is decidable; equivalently, undecidable recursively enumerable languages cannot have D-finite length generating functions. We then examine coefficient growth and singularities through Pringsheim's theorem, the Polya-Carlson dichotomy, Hadamard gaps, and Siegel G-functions. The resulting picture is asymmetric: analytic properties provide useful one-way constraints on language computability, but length enumeration loses too much information to determine grammatical or computational complexity without additional assumptions.

0.12

Latent Coherence: Do Frontier Language Models' Probability Judgments Obey the Law of Total Probability?

cs.CL ·2026-10-22 ·Peter Potash
code/data similarity 0.54
Abstract · 15 pages

LawZero's Scientist AI trains a Predictor toward the Bayesian posterior, with an objective that penalizes inconsistent probabilities among latent explanations. We ask whether current frontier language models, with no such training, are already consistent in this sense zero-shot: do their probability judgments satisfy the law of total probability, $P(X)=\sum_z P(z)P(X|z)$, when the latent $z$ must be inferred from records in the context? In separate single-turn calls, models are asked for $P(X)$, for $P(z)$ over the latent's states, and for $P(X|z)$; a causal family applies the same test to $P(y|do(x))$ and the back-door adjustment. Every query has an exact ground-truth probability, and a subset of items is asked in three wordings to separate persistent incoherence from rewording noise. Across seven models, GPT-6.1 Sol and Claude Opus 5.5 are the most coherent (mean absolute residual 0.08 and 0.12 log-odds, about 2 percentage points). Claude Haiku 5.5, Gemini 3.8 Flash and DeepSeek V4-Pro show persistent incoherence (0.27-0.35 log-odds) that is mostly item-specific rather than a shared offset. Within every model, larger residuals go with larger errors against the truth (Spearman 0.40-0.62), so the residual, which needs no ground truth, flags unreliable answers. Reasoning length does not predict coherence, across models or within a model across rewordings of the same question. Simple consistency checks pass almost everywhere; the failures are specific to reasoning through a latent variable. Code and data: https://github.com/ppotash/latent-coherence

0.38

Compositional Information Geometry of Probabilistic Programs

cs.PL cs.CG ·2026-10-13 ·Jacek Karwowski, Frank Nielsen, Sam Staton
highly cited similarity 0.53
Abstract

We study the compositional, intensional information geometry of probabilistic programs. For a first-order probabilistic programming language with finite random variables, we construct a contravariant lax monoidal functor that, for a program of type $X \to Y$, pulls back a (degenerate) Riemannian metric on $Y$ to $X$. It specialises to the Fisher metric in the case of standard statistical models and to ordinary pullback in the case of deterministic programs. We further justify this construction in two ways. First, we show that this functor arises as a pullback of geometry from a universal space of programs over a type $Y$ with a fixed trace space. Second, we show that it is a special case of a more general pullback construction over weighted measures. Our compositional geometry depends on the presentation of the program, and we study how it relates to the (non-compositional) geometry of the realised measures by establishing a three-level inequality relating our score-velocity metric for the program, the program's corresponding output-conditioned metric, and the standard Wasserstein-Fisher-Rao metric of the realised family. Finally, we show how the theory can be extended to accommodate continuous random variables using diffeological quasi-Borel spaces. We also discuss several concrete directions for future work.

0.43