CS Paper Radar

A Behavioural Framework for Predicting and Understanding Loss of Control in Frontier Artificial Intelligence Systems

Yao Huang, Yichi Zhang, Yifan Wang, Yitong Sun, Haoyu Dong, Jifan Ma, Jinzhe Tu, Xiaojian Li, Sören Mindermann, Wei Xu, Yinpeng Dong

cs.AI cs.CLcs.LG ·submitted 2026-11-05 12:49 UTC · v1

Abstract

Frontier artificial intelligence (AI) systems are increasingly operating over longer horizons with diminishing human oversight, raising the risk of loss of control: systems acting beyond effective human control with potentially irreversible consequences. Because such failures may become observable only after containment is difficult, they must be assessed prospectively. We introduce a behavioural framework that decomposes loss-of-control risk into three factors: misaligned motive, harm-enabling capability, and monitoring evasion. We operationalise these factors through 13 measurable aspects and 29 evaluation tasks, whose scores are combined into a single loss-of-control (LoC) score. Across 13 frontier models, the LoC score shows strong prospective validity, correlating with failure rates in contained long-horizon risk scenarios (Pearson r = 0.84). Risk varies systematically across models and cannot be explained by capability alone. We further show that the framework can guide targeted mitigations: fine-tuning and monitoring directed at diagnosed vulnerabilities reduce failure rates while largely preserving general performance. These results provide a practical basis for predicting, understanding, and mitigating loss-of-control risk before it materialises.

Comment
39 pages, 5 figures

Closest papers

Nearest neighbours by meaning across the whole index.

A Competing-Hazards Systematization of Loss of Control in Autonomous Agents

cs.AI cs.CR ·2026-09-29 ·Mohamed Aly Bouke
similarity 0.72
Abstract · 12 pages, 1 figure, 3 tables. Dataset: https://doi.org/10.5281/zenodo.22995268

Leading AI developers have reported agents acting beyond their approved limits, which a United Nations panel described as an early warning of loss of human control. Yet incident reports and agent-safety evaluations describe these events differently, making it difficult to compare failures, trace risk across attempts, or separate agent behavior from the environment's role in allowing an out-of-scope action to succeed. To address this gap, we introduce a common framework in which each attempt ends in approved completion, safe stopping, scope escape, or continuation. We formalize the framework as a discrete-time competing-hazards model and derive escape probability within a retry budget, a model-conditional safe-budget limit, and conditions for estimation from execution logs. We audit 22 incident reports and 102 agent-safety evaluations published from January 2025 to September 2026 using primary sources. Six incidents involved tasks that could not be completed within scope, thirteen involved agents that continued rather than stopped, and five did not report stopping behavior. Developers' figures imply a task-level incidence ratio near 47 for out-of-scope coordination in never-solved versus solved tasks. Among evaluations, 87 recorded an out-of-scope effect or specification violation, 26 treated safe stopping as a first-class outcome, only 20 recorded both, and 79 merged budget exhaustion with failure. In 20 of 22 incidents, the environment allowed an out-of-scope effect, indicating that realized loss of control often reflected persistent agent behavior interacting with permissive boundary conditions; meanwhile, no evaluation reported all fields needed to estimate the full competing-hazards process from published evidence.

0.25

Safe AI Evaluation: A Systems Perspective

cs.SE ·2026-11-18 ·Qinghua Lu
similarity 0.67
Abstract

AI evaluation can involve an active, adaptive optimisation process in which AI agents re-plan, retry, use tools or delegate work to achieve a goal. Unanticipated capabilities or unsafe behaviour may emerge, and existing controls may be inadequate to prevent harm. Selected safeguards may also be deliberately relaxed to observe behaviour they would otherwise suppress, creating additional risks. This article examines the distinctive safety considerations of evaluation and their implications for control placement, observation and intervention, and presents four architectural patterns to support safe AI evaluation, including diverse supervisors, an independent verifier, silent monitoring and coordinated shutdown.

0.14

R-BED: A Consequence-Aware Framework for System-Level AI Evaluation and Continuous Assurance

cs.AI ·2026-11-07 ·Vishal Srivastava
similarity 0.65
Abstract · 22 pages (18 pages main text plus appendices), 8 figures, 11 tables. Specification-level methodology with a synthetic walkthrough; no empiri…

Evaluations of AI systems usually report properties of a model, while decisions to deploy concern a configured system: a model together with prompts, tools, permissions, data, safeguards and an operating context. We ask how evidence about technical failures, deployment-control effectiveness and real-world consequences can be connected to defensible authorization decisions throughout the lifecycle of a deployed system. We present R-BED (Risk-Based Evaluability Design), a specification-level methodology with a fixed 6+4+3 architecture of thirteen diagnostic dimensions (technical failure, deployment controls, consequence and impact); a failure-mediated expected-loss identity that is kept separate from intended-behavior and misuse pathways; bounded evidence statuses with independence levels; a control assurance ladder; five authorization dispositions governed by eight rules; and a Continuous System Evaluation (CSE) layer in which evidence carries a configuration pin, an expiry and declared dependencies, so that a change invalidates evidence in proportion rather than wholesale. We work through a synthetic refund-agent walkthrough, with a deterministic script that reproduces every number, to show how a task-success view can miss a consequential control weakness that the procedure identifies as missing evidence. The contribution is a reproducible methodological specification. Most components are standard practice integrated into one evidence-to-decision record; the Evidence Validity Register, the authority envelope and the tier rules are proposals. None has been empirically validated, and we state seven hypotheses for independent testing.

0.17

Quantifying System-Level Harms from AI Adoption in Complex Sociotechnical Systems

cs.AI ·2026-08-24 ·Paul Vautravers, Oliver Chalkley, Gabriel Downer, Kate S, Damian Ruck
published similarity 0.64
Abstract

Artificial Intelligence (AI) is increasingly integrated into complex sociotechnical systems, including Critical National Infrastructure (CNI), where harms emerge from interactions between technical, human, and organisational elements. Yet current AI evaluation remains model-centric, offering little insight into how observed behaviours might translate into system-level risk. We propose a framework that links structured hazard analysis, component-level testing, and probabilistic system modelling to bridge this gap. By providing a traceable pathway from model behaviour to system-level outcomes, the framework enables practitioners to answer the "so what?" of AI failures, quantify their systemic impact, and move toward evidence-based and anticipatory governance of AI in complex systems. Applied to the UK's Real Time Gross Settlement (RTGS) system as an illustrative worked example, we derive AI-driven loss scenarios using Systems Theoretic Process Analysis (STPA) and examine adversarial manipulation of LLM-based trading as one such loss scenario. Component-level experiments show that simple adversarial inputs induce measurable behavioural shifts where AI recommendations are followed. Under the component-to-system mapping used here for a financial contagion model, these shifts alter system resilience, increasing bank failures and lowering the threshold at which shocks lead to cascading disruption, particularly under widespread or monopolistic AI adoption.

0.17

Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation

cs.AI ·2026-09-25 ·Chang Gong, Jingping Bi, Di Yao, Ran Cheng, Yaodong Yang +4 more
highly cited similarity 0.64
Abstract · 25 pages, 6 figures

Rapid advances in AI are bringing recursive self-improvement (RSI) closer to practice, with growing progress in model self-training, self-evolving agents, and automated AI development. Yet the very mechanisms that enable continuous improvement also introduce new challenges for AI safety. Unlike isolated failures, safety risks can persist across updates, accumulate through repeated adaptation, and propagate to successor systems. More importantly, the processes that generate and select future improvements may themselves evolve, making safety increasingly difficult to assess and maintain. This raises a fundamental question, i.e., how can safety be maintained when both AI systems and the processes that improve them continue to change? Therefore, we introduce Evolutionary Safety to study how safety properties are preserved, weakened, or restored throughout persistent and recursive self-improvement. We characterize six recurring manifestations of evolutionary risk and develop a five-domain taxonomy that captures how safety-relevant changes arise, persist, and propagate across evolving systems. Building on this foundation, we establish a framework for risk discovery and evaluation across states, updates, trajectories, and lineages, and examine governance principles for controlling modifications, verifying and authorizing updates, and ensuring traceability and recovery. We further identify open problems in maintaining safety as both AI systems and their improvement mechanisms continue to evolve.

0.44

Frontier AI Safety and Governance 2026: Risk Understanding and Coordinated Governance

cs.CY ·2026-11-09 ·Zeng Yi, Seán Ó hÉigeartaigh, Cao Gongce, Lu Enmeng, Chen Yu +105 more
survey similarity 0.63
Abstract · 69 pages. Available at https://www.wicinternet.org/2026-11/03/c_1224363.htm

Frontier AI Safety and Governance 2026: Risk Understanding and Coordinated Governance is a key outcome of the AI Safety and Governance Program of World Internet Conference Specialized Committee on AI in 2026. Drawing on the latest developments in frontier AI capabilities and applications, the report systematically examines the evolving risk landscape, analyzing key risk areas, their severity and response priorities. Building on a review of existing global governance practices and an assessment of frontier AI governance readiness, it identifies key governance gaps and weaknesses and proposes directions and practical mechanisms for coordinated responses. The report provides a reference for government departments, international organizations, businesses, research institutions and other stakeholders. It aims to strengthen the international community's understanding of and consensus on frontier AI risks, enhance preparedness and collective response capacity, and advance the safe, reliable and responsible development of frontier AI to better serve the common well-being of humanity.

0.41

Assessing the Empirical Evidence for Loss of Control from Agentic General-Purpose AI

cs.CY cs.AIcs.CLcs.LG ·2026-10-18 ·Risto Uuk, Santeri Koivula, Lorenzo Pacchiardi, Rokas Gipiškis, Laura Caroli +1 more
similarity 0.63
Abstract · 42 pages. Previously posted on SSRN on 28 May 2026

Theoretical literature warns that advanced generalist agentic systems might pursue instrumental goals that are misaligned with human preferences, thus posing risks of irreversible loss of human control. The EU's Code of Practice for General-Purpose AI lists loss of control as a systemic risk that must be assessed, yet no structured assessment of the quality of the empirical evidence for this risk exists. This study fills that gap. We evaluate 19 recent empirical studies across eight properties identified from threat models (such as strategic deception, situational awareness, and autonomous replication), assessing the quality of evidence across three dimensions: construct validity, content validity, and external validity. Our central finding is that the current body of evidence supports a judgment of loss of control as "weakly plausible." In our framework, 'plausibility' denotes the strength and direction of available evidence, not a probability estimate. Emerging evidence exists across multiple properties. No study clears the validity thresholds for a stronger judgment that a property is present; a single study clears them only for a negative-direction judgment (against situational awareness). Validity gaps, particularly in external validity, are widespread. Therefore, our findings highlight the need for more rigorous evaluations of agentic general-purpose AI systems. However, for the most dangerous properties, high external validity may be structurally unachievable under responsible research practices, reflecting a ceiling on feasible evidence rather than absence of the capability. Furthermore, as AI capabilities advance rapidly, empirical assessments may systematically lag behind the capability frontier. Our framework provides a structured baseline for ongoing monitoring as capabilities advance.

0.22

Toward a Threat Actor Profiling Taxonomy for Pre-Release Risk Management of Open-Weight Frontier Models

cs.CY ·2026-08-26 ·James Zhang
similarity 0.63
Abstract

Pre-release risk management for frontier AI misuse risks routinely leaves threat actor assumptions implicit, inconsistently specified, or ungrounded. This capstone argues that explicit adversary characterization should be regarded as a prerequisite for evaluations that are interpretable, comparable, and faithful to the risks they target. We propose a six-attribute taxonomy (covering technical sophistication, prior domain knowledge, organizational capacity, operational infrastructure, financial capacity, and time horizon) with empirically grounded tiers derived from existing terrorism, biosecurity, and cybersecurity literature. The taxonomy is designed to function as research infrastructure: a common language for pre-specifying adversary assumptions before evaluations are conducted, analogous to pre-analysis plans for randomized controlled trials (RCTs) in medicine and economics. Its application is particularly urgent for open-weight model developers, for whom release decisions are irreversible and must anticipate adversarial reasoning.

0.24