CS Paper Radar

The Dynamics of Intelligence Explosions

Toby Ord

cs.AI econ.TH ·submitted 2026-08-14 16:07 UTC · v2

Abstract

AI is increasingly being used to help with AI R&D. Under certain conditions this feedback loop might be able to produce an intelligence explosion, with rapidly escalating AI capabilities. I explore the mathematics of the most explosive possibilities, with an eye to understanding what drives the dynamics. I show that singular growth (towards a vertical asymptote) is harder to achieve than would be expected from recent economics-inspired modelling, and that there is an important but neglected class of growth rates that are faster than exponential but don't lead to a vertical asymptote. I draw out the generation time (the time to go around the feedback loop) as a neglected parameter that plays a pivotal role in determining the behaviour of any intelligence explosion --- one cannot have singular growth unless the generation time rapidly approaches zero.

Comment
33 pages, 3 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Recursive Criticality of AI Self-Improvement

cs.AI ·2026-08-31 ·Mikhail Burtsev
code/data similarity 0.64
Abstract · The code and computational notebook used to reproduce the numerical results and figures are available at https://github.com/burtsev/recursiv…

AI is increasingly used in the R\&D process that produces future AI systems. We study the conditions under which this feedback becomes self-amplifying. Our model describes how the rate of AI capability growth depends on baseline research productivity, recursive feedback, and the increasing difficulty of research progress. We derive a recursive reproduction number, $\mathcal{R}_{\mathrm{AI}}$, that determines whether improvements are amplified or damped across development cycles. This quantity compares the strength of feedback with the rate at which further progress becomes more difficult. When $\mathcal{R}_{\mathrm{AI}}>1$, the effects of improvements compound across development cycles, placing the system in a self-amplifying regime. When $\mathcal{R}_{\mathrm{AI}}<1$, their effects weaken across cycles. The transition depends on the structure of the AI R\&D feedback loop and need not occur at any particular level of model capability. A system can therefore enter a self-amplifying regime before acceleration becomes visible, while rapid progress can also occur without self-amplification. Higher baseline research productivity can accelerate progress without changing whether the system is self-amplifying, but the duration of the development cycle becomes a limiting timescale for amplification. Increasing research difficulty can end a period of self-amplification. Extending the model to multiple research actors shows that improvements shared across organizations can make the overall research ecosystem self-amplifying even when no individual actor is. The framework identifies measurable properties of AI R\&D systems that can help distinguish recursive amplification from rapid progress driven by other sources, including the strength of recursive feedback, how effectively improvements propagate into successor systems, cycle duration, and the increasing difficulty of further progress.

0.31

The Singularity Has a Distribution: Exact Timing Laws and Detection Limits for a Stochastic Intelligence Explosion

cs.AI cs.CYmath.PR ·2026-10-13 ·Xiaoyu Li, Zheng Gao, Chiwun Yang, Zhizhou Sha
code/data similarity 0.62
Abstract · 24 pages, 6 figures, 3 tables. Code and data: https://github.com/xiaoyulics/singularity-distribution

Automating AI research creates a positive feedback loop between software capability and research effort. We study a stochastic realisation of this loop in which independent, positive, multiplicative improvements arrive at a rate proportional to a power of current capability. Within this model, randomness leaves the deterministic explosion threshold unchanged but replaces the blow-up date by an explicit distribution. For equal-sized improvements the explosion time is a geometrically weighted sum of independent exponentials; its $q$-series representation gives its transforms, moments and tail behaviour, including a two-term small-time asymptotic. For Lomax-distributed improvement sizes the explosion time is exactly Gamma distributed. Self-similarity implies that, with parameters known, the relative uncertainty about the remaining time does not vanish as explosion approaches. We formulate a fixed-sample test between constant and geometrically increasing idea-arrival rates. If the baseline rate and the alternative log-rate increment $\ell$ are known, the minimum sample size at error level $α$ is asymptotic to $(12z_α^2)^{1/3}\ell^{-2/3}$, where $z_α$ is the upper-$α$ standard normal quantile. Uncertainty about the growth exponent near its critical value produces a $1/t$ tail with an explicit constant, and hence an infinite conditional mean waiting time, although explosion is certain given a positive exponent. Separate extensions show that a fixed deployment delay prevents explosion in the specified delayed-feedback models, that power-law diversion of effort shifts the growth threshold, and that in a deterministic oversight race a fixed allocation share does not change the asymptotic regime, except at a critical equality of exponents. Numerical illustrations, which are not empirical forecasts, show how timing and detectability depend on the granularity of improvements.

0.35

The End of AI Exponentiation: Fluttering Inside and Outside AI Bubble

cs.AI ·2026-09-05 ·Victor Kebande
similarity 0.57
Abstract

The exponentiation of Artificial intelligence (AI) in the recent past has entered a transformative era that has been driven by the growth in large language models (LLMs), large-scale compute infrastructures, and autonomous reasoning systems. However, the rapid acceleration of AI has increasingly shown technological, societal, economic, ethical and infrastructural challenges associated with peak data limitations, rising computational demands, synthetic data recursion, valuation inflation, and societal instability. The traditional scaling paradigms that have powered the modern AI systems are gradually encountering friction in sustaining continuous exponential growth. This paper views ``the end of AI exponentiation,'' thus exploring how it flutters inside and outside the bubble, where instability emerges within the AI ecosystem through compute and data-center races, speculative investments, and the rat-race toward superintelligence, and outside the ecosystem through labor disruption, governance concerns, public uncertainty, and geopolitical acceleration surrounding future intelligent systems and infrastructures globally.

0.17

What if automating AI R&D triggers an intelligence explosion?

cs.CY cs.AI ·2026-09-28 ·Alan Chan, Christoph Winter, Andrew Barto, Jakub Pachocki, Geoffrey Hinton +17 more
highly cited similarity 0.56
Abstract

In contrast to even a year ago, AI systems now write most of the code inside the companies that build them. As more of the AI research and development (R&D) pipeline is automated, could AI progress radically accelerate in an "intelligence explosion," where years of advances are compressed into months or less? Preliminary evidence suggests that it could. In this work, we assess this evidence, analyze an intelligence explosion's potential impacts, and propose policy responses. AI systems are on track to automate most AI R\&D work within a few years, and possibly all of it. If this triggers an intelligence explosion, it could dramatically bring forward AI's benefits, but also pose extreme risks: capabilities growth could accelerate far beyond what society can keep up with, humanity could lose control over superhuman AI systems, and checks on power within and between states, companies, and branches of government could be severely eroded. Although there remains much uncertainty about these possibilities, the high stakes warrant serious further attention. Policymakers should urgently obtain more visibility into the automation of AI R&D, develop ways to steer and constrain an intelligence explosion, and prepare society to adapt to an intelligence explosion's impacts.

0.44

Ecology of AI Agents: Collaboration Creates a Population Threshold for Takeoff

cs.AI cond-mat.dis-nncs.MAphysics.bio-ph ·2026-10-08 ·Erin Crawley, Hidenori Tanaka
similarity 0.54
Abstract · 22 pages, 11 figures, 1 table

AI agents can now conduct real-world cyberattacks, scale up capabilities with the number of agents, and collectively pursue misaligned goals to obtain rewards. Together, these factors raise the risk of a population explosion of misaligned agents: agents could compromise computers and secretly deploy additional agents, creating a self-reinforcing cycle where larger populations develop greater collective cyber capability and expand further. This raises a fundamental question: What determines whether a population of misaligned agents remains contained or takes off into this self-reinforcing cycle? This population-level problem is ecological safety: unlike individual-agent or multi-agent safety with a fixed population, it concerns the dynamics of the population itself. Here, we develop an ecological theory of AI-agent populations based on a population growth equation in which fitness (growth rate) depends on cybersecurity capability. We show that, without collaboration, the population takes off only when individual-agent capability exceeds a critical threshold. With collaboration, however, collective cybersecurity capability increases with population size. This creates a critical population threshold: below it, the population declines; above it, the population takes off, even though individual-agent capability has not changed. In ecology, this phenomenon is known as the strong Allee effect. Because red teaming a small group of agents cannot guarantee ecological safety in larger populations, our theory calls for ecological red teaming and population pacing: gradually deploying larger agent populations in controlled environments, while measuring how cyber capability scales with population size, and estimating the critical population size for takeoff. Capability gains may lower this threshold, requiring re-estimation for each new model generation.

0.19

A Bifurcation-Theoretic Framework for the Controllability of Emergent Intelligent Systems

cs.AI eess.SYnlin.AO ·2026-11-09 ·Hao Li, Shaowen Lu
similarity 0.52
Abstract · 21 pages, 1 figure, 5 tables

Real-world intelligent systems must reconcile two demands: the adaptability conferred by emergence and the steerability required for accountable deployment. We show that these demands are governed by two distinct bifurcations on the same object and that, consequently, increased capability does not imply increased controllability. Emergence is a bifurcation of the capability variable $S$, with criterion $Γ:=\mathcal F'(0)>1$ equivalent to the existence of a unique positive stable fixed point of a self-consistency equation, following solely from the concavity of $\mathcal F$. Controllability is a bifurcation of the permission variable $c$, whose dynamics include an erosion term and exhibit a saddle-node fold with hysteresis as the gate criterion $ρ^\ast$ crosses unity. Three quantitative results follow: the critical exponent distinguishes families rather than classifying the transition; the gating-cost bound $K(M)\propto\log(1+M)$ grows with scope, not capability; and the fold threshold is $γ_c\approx4μκ$. Calibrated in both directions, the criteria reproduce the two-dimensional Ising transition, admit two true-feedback order parameters, and reject nine candidate quantities, supporting the conclusion that the order parameter is geometric rather than capability-related. Since permission cannot be endogenized and no deployed system has been measured to satisfy $Γ>1$ and $ρ^\ast>1$ simultaneously, increased capability does not imply controllability.

0.27

Is the arXiv Apocalypse upon us? A break in preprint growth coincident with agentic AI

cs.DL physics.soc-phstat.AP ·2026-10-25 ·Carlo R. Contaldi
similarity 0.51
Abstract · 9 pages, 2 figures, 3 tables. Submitted to PNAS

As an experiment in agentic assisted analysis, we used an agentic AI model to analyse the effect of agentic AI on arXiv submissions; data acquisition, analysis and the first draft were carried out by the model under the author's direction. Monthly arXiv submissions have grown exponentially for three decades, at about $9\%$ per year. Fitting a piecewise log-linear trend with seasonal modulation to the full record, we find two post-pandemic breaks in growth rate. The first, in mid-2022, precedes ChatGPT and is carried by the expansion of AI research within computer science (CS); outside CS, growth from 2022 to 2025 remained at $8\%$ per year. The second, between December 2025 and February 2026, follows the release of models capable of extended autonomous work and is absent from the AI categories themselves. Since December 2025 submissions without a CS listing have run $17\%$ above trend, and in September 2026 total submissions reached a record $40\,363$. The excess is largest in mathematics, where submissions doubled within a year, and follows a gradient from formal and theoretical fields to experimental ones that is not ordered by surveyed rates of AI-tool adoption. In mathematics the increase is distributed across subfields and authors, two-thirds of it arising from a larger number of distinct authors rather than from a few prolific submitters. Across fields the clearest signature is a shift to single authorship: solo-authored papers rose by $70\%$ within a year in mathematics and condensed matter, and by $44\%$ in quantitative biology even as its total output fell, but not in experimental particle physics. The analysis is deliberately agnostic of LLM-detection methods: it uses only submission counts and metadata, with treatment timing fixed by calendar dates, and is therefore free of the outcome-triggered timing bias of author-level studies.

0.27

LLM Evolution in a Simple Simulated Market

cs.MA cs.AIcs.CLq-bio.PE ·2026-10-16 ·Charlie Pilgrim, Christopher Krupenye, Dora Biro, Richard P. Mann
hot direction similarity 0.51
Abstract

Arguments that AI systems will evolve in harmful ways are largely made in the abstract. We provide a concrete empirical example. We simulate simple markets consisting of populations of twenty Large Language Model agents tasked with solving arithmetic problems. Agents differ only in their system prompt; those that go bankrupt are removed and replaced by mutated copies of the prompts of successful agents. Two market conditions select for opposite phenotypes: a market that rewards accuracy produces agents with high accuracy (94%) and high token use; a market that charges per token produces agents with low accuracy (10%) and low token use. In the second market, only tokens are incentivised yet we see a collapse in accuracy. Token use and accuracy are correlated traits such that selection on one also moves the other, and behaviour adapts along an unpriced dimension. A third neutral condition with no selection allows us to separate these effects from bias in the mutation operator. Adaptation occurs through a collective optimisation process that operates above the level of individual agents, who are never told the payment rule, do not receive feedback, and have no memory.

0.31