CS Paper Radar

Superintelligence and the Triple Risk of an AI Race

Matthew Rendall, Max Hellrigel-Holderbaum, Leonard Dung

cs.CY cs.AI ·submitted 2026-10-12 16:47 UTC · v1
position

Abstract

On many estimates, artificial general intelligence (AGI) is likely to emerge within the next few decades or even sooner. Once it does, it may quickly surpass human intelligence, possibly by a huge margin. Some believe that the country that develops AGI first could attain global dominance. This prospect risks setting off intense competition between the United States and China. Indeed, a narrative has already developed that the superpowers are locked in an AI race, with the Trump administration proclaiming its determination to 'dismantle unnecessary regulatory barriers' to winning it. We argue that this approach presents three profound dangers. The first two are familiar from past arms races: being overtaken by a rival or triggering a war. The third is new: losing control of human beings' future to agents far more intelligent than we are, which could endanger our very survival. Striving to get to AGI first risks a race to the bottom on safety practices and regulation concerning them. China does not yet appear bent on racing to superintelligence. The United States should prioritise both unilateral and co-operative efforts to improve AI safety and avoid provoking a race that all humanity could lose.

Closest papers

Nearest neighbours by meaning across the whole index.

Frontier AI Safety and Governance 2026: Risk Understanding and Coordinated Governance

cs.CY ·2026-11-09 ·Zeng Yi, Seán Ó hÉigeartaigh, Cao Gongce, Lu Enmeng, Chen Yu +105 more
survey similarity 0.60
Abstract · 69 pages. Available at https://www.wicinternet.org/2026-11/03/c_1224363.htm

Frontier AI Safety and Governance 2026: Risk Understanding and Coordinated Governance is a key outcome of the AI Safety and Governance Program of World Internet Conference Specialized Committee on AI in 2026. Drawing on the latest developments in frontier AI capabilities and applications, the report systematically examines the evolving risk landscape, analyzing key risk areas, their severity and response priorities. Building on a review of existing global governance practices and an assessment of frontier AI governance readiness, it identifies key governance gaps and weaknesses and proposes directions and practical mechanisms for coordinated responses. The report provides a reference for government departments, international organizations, businesses, research institutions and other stakeholders. It aims to strengthen the international community's understanding of and consensus on frontier AI risks, enhance preparedness and collective response capacity, and advance the safe, reliable and responsible development of frontier AI to better serve the common well-being of humanity.

0.41

Advancing a Global Framework for AI Safety and Governance for the Well-being of Humanity

cs.CY ·2026-11-10 ·Zeng Yi, Seán Ó hÉigeartaigh, Wang Zhengqi, Lu Enmeng, Chen Yu +71 more
publishedsurvey similarity 0.60
Abstract · 47 pages. Published in 2025 by World Internet Conference. Available at https://www.wicinternet.org/2025-11/07/c_1137524.htm

The Report "Advancing a Global Framework for AI Safety and Governance for the Well-being of Humanity" is an important outcome of the AI Safety and Governance Program of World Internet Conference Specialized Committee on Artificial Intelligence in 2025. This report provides a systematic review of current global explorations and practices in AI safety and governance, identifies key issues that need to be addressed, and draws upon multilateral governance experiences from other domains. Centered on mechanism design, it explores how to build a global AI safety and governance system that ensures safety and security, promotes inclusiveness, clarifies responsibilities and accountabilities, strengthens effective coordination, and ensures authoritative and efficient governance. Guided by the principles of multilateralism and the vision of building a community with a shared future for humankind, the report proposes institutional measures and policy recommendations to establish a United Nations-centered global AI safety and governance framework. It aims to serve as a reference for governments, international organizations, enterprises, and research institutions. The report is to contribute insights and momentum to advancing people-centered and beneficial AI governance worldwide, ensuring that AI technologies truly serve the well-being of all humanity.

0.52

Sovereign by necessity? Frontier AI export controls, cyber security, and the limits of national AI capability

cs.AI cs.CR ·2026-08-13 ·Alan Woodward, Andrew Rogoyski
similarity 0.58
Abstract · 42 Pages

A small number of firms based in two states produce the most capable frontier AI models. The governments of those states have shown both the legal power and the political will to decide which other countries may use these systems. In June 2026 the United States required a leading developer to obtain licences before releasing its most advanced models to any foreign person, including foreign nationals resident in the United States. The affected models were withdrawn worldwide at short notice, partly because the restriction proved impractical to administer. This followed within months of the first documented case of a largely autonomous, AI-run cyber espionage campaign, and coincided with mounting evidence that frontier models alter the economics of both cyber attack and cyber defence. This article examines how these two developments interact, and situates them within the unusual market dynamics now driving large-scale AI development. It argues that access to frontier AI is becoming part of national cyber defence, that such access can be revoked, and that the obvious remedy of sovereign capability remains only partly feasible for all but a handful of states. Drawing on evidence about training costs, the concentration of computing power and the support offered by national AI programmes, it asks what sovereignty can realistically mean for small and middle powers, and for large powers as well. The article proposes a layered strategy: negotiated access guarantees, sovereignty at the level of inference, hedging with open-weight models, pooled regional capability, sustained talent development and continued investment in basic cyber resilience. The open-weight hedge proves at once more capable and more politically exposed than is commonly assumed. Much of the near-term risk lies in how capable models are deployed and contained rather than in their apparent performance.

0.10

How Could AI Eliminate Humanity? A Failure-Mode Analysis of Civilizational Risk

cs.AI cs.CYcs.HC ·2026-10-06 ·Mikołaj Sienicki, Krzysztof Sienicki
similarity 0.58
Abstract · 83 pages, 20 sections, 35 references

This article develops a failure-mode framework for analyzing how advanced artificial intelligence could contribute to human extinction, irreversible civilizational collapse, or permanent human disempowerment. The central thesis is that catastrophic AI risk does not require consciousness, hostility, or an explicit intention to harm humanity. Instead, risk may arise through several distinct but interacting pathways, including autonomous misalignment, harmful human use, organizational failure, and competitive deployment. The severity of these pathways depends on factors such as capability, autonomy, external access, persistence, institutional safeguards, and the preservation of recovery capacity. The analysis is deliberately non-operational: it identifies causal conditions, empirically tractable intermediate quantities, and defensive research questions rather than procedures for causing harm.

0.18

Non-Great-Power Conflict and AI Risk

cs.CY ·2026-08-26 ·Kristina Kempkey, Seán Boddy, Catherine Ge-Wang
similarity 0.57
Abstract

Research on advanced AI and the risk of war has focused almost exclusively on great power conflict, on the grounds that confrontation between nuclear-armed adversaries poses the greatest risk of catastrophic or existential harm. Considerably less attention has been paid to non-great-power conflict (NGPC): wars between non-great powers, between non-great powers and great powers, civil wars, proxy wars, and conflicts involving nonstate actors. This paper evaluates the null hypothesis that NGPC is much less important than great power conflict (GPC) as a source of catastrophic risk in an era of increasingly capable AI, against the alternative that it is within an order of magnitude of GPC in importance. We assess three sub-hypotheses: that NGPC increases the likelihood of great power conflict; that it increases the expected harm from catastrophic terrorism; and that it increases the expected harm from loss of control over advanced AI systems. We find the null poorly supported for H1 and H2, and identify H3 as a priority for further work rather than a settled finding. We also identify five intermediate variables that recur across the pathways - information environment quality, decision-making timeline compression, great power threat perception, capability diffusion, and norm erosion - and argue that these shared nodes are the highest-priority targets for further investigation and intervention.

0.43

What if automating AI R&D triggers an intelligence explosion?

cs.CY cs.AI ·2026-09-28 ·Alan Chan, Christoph Winter, Andrew Barto, Jakub Pachocki, Geoffrey Hinton +17 more
highly cited similarity 0.56
Abstract

In contrast to even a year ago, AI systems now write most of the code inside the companies that build them. As more of the AI research and development (R&D) pipeline is automated, could AI progress radically accelerate in an "intelligence explosion," where years of advances are compressed into months or less? Preliminary evidence suggests that it could. In this work, we assess this evidence, analyze an intelligence explosion's potential impacts, and propose policy responses. AI systems are on track to automate most AI R\&D work within a few years, and possibly all of it. If this triggers an intelligence explosion, it could dramatically bring forward AI's benefits, but also pose extreme risks: capabilities growth could accelerate far beyond what society can keep up with, humanity could lose control over superhuman AI systems, and checks on power within and between states, companies, and branches of government could be severely eroded. Although there remains much uncertainty about these possibilities, the high stakes warrant serious further attention. Policymakers should urgently obtain more visibility into the automation of AI R&D, develop ways to steer and constrain an intelligence explosion, and prepare society to adapt to an intelligence explosion's impacts.

0.44

Who Can Say No? Control, Accountability and the Pace of Frontier AI

cs.CY ·2026-10-09 ·Rikard Rosenbacke, Carl Rosenbacke, Victor Rosenbacke, Martin McKee
position similarity 0.56
Abstract · 23 pages

In September 2026, the leaders of the largest frontier AI laboratories said in public that the pace of capability improvement is too fast and that none of them can slow down alone, and agents under evaluation at OpenAI broke out of their test environment and reached a third party's live systems, which a UN-mandated panel called an early warning of one route to loss of human control. This paper examines the institutional side of that risk: not the machine, but the person who must control it. We argue that control is not taken by AI but given away, one delegation at a time, by organisations arranged so that nobody is positioned to say no, and we define control as a person who can detect a serious problem, challenge the decision, stop the system and reverse it, and who has something to lose by not doing so. Three questions follow, which any builder, funder, regulator or deployer can answer in public: (1) who inside the organisation can say no, and who answers for saying yes; (2) who outside the organisation checks the decision, and what can they do when they disagree; (3) what happens when AI builds AI faster than anyone can check it. A checkable answer needs a named person, a date, a number, and a stop that has been used or tested. Three measures meet that test: a named person on every consequential decision, including the standing instructions AI agents carry out; a supervisor with a stop authority; and two published clocks, time to build against time to check. We show that governance theory arrives at the same conditions, and code fifty older mechanisms against the questions, finding that almost every one began as one rule at one point, taken by one party without waiting for the rest. That is the paper's design: a start, not a solution. Publication is the first step and one laboratory can take it alone; the contract, the independent body and the law come after.

0.14

A Behavioural Framework for Predicting and Understanding Loss of Control in Frontier Artificial Intelligence Systems

cs.AI cs.CLcs.LG ·2026-11-05 ·Yao Huang, Yichi Zhang, Yifan Wang, Yitong Sun, Haoyu Dong +6 more
similarity 0.56
Abstract · 39 pages, 5 figures

Frontier artificial intelligence (AI) systems are increasingly operating over longer horizons with diminishing human oversight, raising the risk of loss of control: systems acting beyond effective human control with potentially irreversible consequences. Because such failures may become observable only after containment is difficult, they must be assessed prospectively. We introduce a behavioural framework that decomposes loss-of-control risk into three factors: misaligned motive, harm-enabling capability, and monitoring evasion. We operationalise these factors through 13 measurable aspects and 29 evaluation tasks, whose scores are combined into a single loss-of-control (LoC) score. Across 13 frontier models, the LoC score shows strong prospective validity, correlating with failure rates in contained long-horizon risk scenarios (Pearson r = 0.84). Risk varies systematically across models and cannot be explained by capability alone. We further show that the framework can guide targeted mitigations: fine-tuning and monitoring directed at diagnosed vulnerabilities reduce failure rates while largely preserving general performance. These results provide a practical basis for predicting, understanding, and mitigating loss-of-control risk before it materialises.

0.12