CS Paper Radar

The Normalization of Deviance in AI Development

Emilio Barkett, Alexander Kimpton, Daniel Graham, Yusuf Kundgol

cs.AI cs.ETeess.SY ·submitted 2026-09-04 22:00 UTC · v2

Abstract

Work on the risks of artificial intelligence has focused predominantly on capability risk: the danger that systems become too powerful, too autonomous, or too misaligned with human values. Far less attention has been paid to the organizational level---to whether the institutions building these systems are themselves predisposed to drift toward failure. This paper argues that they are. Regardless of how capable AI systems become, the organizations building them face the same structural dynamics that preceded past major technological disasters. Drawing on case studies of the Space Shuttle Challenger, the Three Mile Island accident, and the Boeing 737 MAX crashes, this paper identifies the common structural mechanisms preceding each failure and maps them onto contemporary AI development. The findings suggest that existing safety infrastructure may provide less protection than it appears, as organizations can complete safety processes in full compliance and still produce catastrophic outcomes. The pre-disaster period of AI development is still underway; the purpose of this paper is to make these dynamics legible while they can still be interrupted.

Closest papers

Nearest neighbours by meaning across the whole index.

Between Rigor and Reality: How AI Safety Benchmark Developers Understand Benchmark Use and Maintenance

cs.AI cs.CYcs.HCcs.SE ·2026-11-13 ·Megan Li, Ningjing Tang, Christopher Stewart, Hoda Heidari, Hong Shen
publishedbenchmarkposition similarity 0.72
Abstract · Poster at the Eval Eval Workshop at ACL 2026. 11 pages. https://openreview.net/forum?id=3QAmXJ8A5j

Safety benchmarks are cited as the primary tool to understand the risks posed by generative AI (GenAI) systems, yet growing evidence suggests they often fail to meet the needs of real-world safety evaluation. We present findings from five interviews with AI safety benchmark developers in academia and industry. We find a disconnect between perceptions of the intended usefulness of safety benchmarks and their practical usefulness in safety evaluations of deployed systems. Our participants offered perspectives on two challenges contributing to this disconnect: (1) the difficulty of achieving inter-rater reliability on safety constructs, and (2) a lack of clarity regarding how to address persistent threats to external validity. Based on our analysis, we argue that safety benchmarks must not only be grounded in deployment contexts, but actively integrated with them. To facilitate this, we outline pathways for more transparent communication between academic and industry AI safety stakeholders.

0.33

How Could AI Eliminate Humanity? A Failure-Mode Analysis of Civilizational Risk

cs.AI cs.CYcs.HC ·2026-10-06 ·Mikołaj Sienicki, Krzysztof Sienicki
similarity 0.71
Abstract · 83 pages, 20 sections, 35 references

This article develops a failure-mode framework for analyzing how advanced artificial intelligence could contribute to human extinction, irreversible civilizational collapse, or permanent human disempowerment. The central thesis is that catastrophic AI risk does not require consciousness, hostility, or an explicit intention to harm humanity. Instead, risk may arise through several distinct but interacting pathways, including autonomous misalignment, harmful human use, organizational failure, and competitive deployment. The severity of these pathways depends on factors such as capability, autonomy, external access, persistence, institutional safeguards, and the preservation of recovery capacity. The analysis is deliberately non-operational: it identifies causal conditions, empirically tractable intermediate quantities, and defensive research questions rather than procedures for causing harm.

0.18

Advancing a Global Framework for AI Safety and Governance for the Well-being of Humanity

cs.CY ·2026-11-10 ·Zeng Yi, Seán Ó hÉigeartaigh, Wang Zhengqi, Lu Enmeng, Chen Yu +71 more
publishedsurvey similarity 0.68
Abstract · 47 pages. Published in 2025 by World Internet Conference. Available at https://www.wicinternet.org/2025-11/07/c_1137524.htm

The Report "Advancing a Global Framework for AI Safety and Governance for the Well-being of Humanity" is an important outcome of the AI Safety and Governance Program of World Internet Conference Specialized Committee on Artificial Intelligence in 2025. This report provides a systematic review of current global explorations and practices in AI safety and governance, identifies key issues that need to be addressed, and draws upon multilateral governance experiences from other domains. Centered on mechanism design, it explores how to build a global AI safety and governance system that ensures safety and security, promotes inclusiveness, clarifies responsibilities and accountabilities, strengthens effective coordination, and ensures authoritative and efficient governance. Guided by the principles of multilateralism and the vision of building a community with a shared future for humankind, the report proposes institutional measures and policy recommendations to establish a United Nations-centered global AI safety and governance framework. It aims to serve as a reference for governments, international organizations, enterprises, and research institutions. The report is to contribute insights and momentum to advancing people-centered and beneficial AI governance worldwide, ensuring that AI technologies truly serve the well-being of all humanity.

0.52

Negotiating Control and Accountability among Stakeholders in High-Risk AI Use Cases: Development and Evaluation of the Method ALIGN

cs.HC cs.AI ·2026-11-18 ·Lena Schneider, Daniel Boos, Gudela Grote
similarity 0.68
Abstract

Characteristics of Artificial Intelligence (AI) severely restrict human control. This creates new challenges for socio-technical systems design especially in safety-critical domains, where control over risks is essential along with the accountability for any harm caused. Drawing from human factors and AI governance literature, expert interviews and participant observation in a complex high-risk AI use case from the railways, we developed ALIGN, a scenario-based method that supports the governance of and stakeholder involvement in AI development and implementation in three ways: raising awareness for potential accountability-control misalignments in socio-technical systems and their negative consequences, offering a structured approach to assessing the alignment between accountability and control for stakeholders in a given use case, and supporting effective design decisions to mitigate AI risks. We describe the iterative development and evaluation of ALIGN and discuss how this method promotes AI governance by highlighting the relevance of control-accountability alignment throughout the AI life cycle.

0.06

Understanding as an Explicit and Assessable Component of Frontier AI Safety Decisions

cs.CY ·2026-08-20 ·Stephen Barrett, Robin Bloomfield, Alexandra Chirilă, Mamoon Masud, David Meredith Hardy +1 more
similarity 0.66
Abstract · Changed first line of abstract

Decision makers need sufficient understanding to make good decisions about training or deploying frontier AI systems. However, such decisions are increasingly made under time-pressure, and this combined with the use of AI generated artefact creation, can mean that the existence of safety cases and system cards may no longer demonstrate that sufficient understanding exists. Our provisional methodology for making understanding explicit and assessable requires the production of an explicit description of 4 objects of understanding (decision, decision-frame, safety justification, system-in-context) and a justification for the adequacy of this understanding. In addition, the methodology provides a mechanism for describing and evaluating the adequacy of the decision-maker representation of this understanding. It builds on recent developments in safety cases using the Assurance 2.0 framework to operationalise the philosophical basis of understanding from Elgin and Arendt. To assess the methodology we trialled two different scenarios. One scenario, which we investigated through role-based analysis, concerned the risk of scheming in the deployment of an AI coding agent in a robotics company and the other scenario was for the higher uncertainty, more decision-critical argument of 'If Anyone Builds It, Everyone Dies' (Yudkowsky and Soares). The trial's central finding, for these two scenarios, is that the methodology could be applied and was found to be generative: we found the analyses that justify sufficiency of understanding (internal coherence, tethering, felicitous falsehoods, external coherence) drives the engineering.

0.25

Frontier AI Safety and Governance 2026: Risk Understanding and Coordinated Governance

cs.CY ·2026-11-09 ·Zeng Yi, Seán Ó hÉigeartaigh, Cao Gongce, Lu Enmeng, Chen Yu +105 more
survey similarity 0.66
Abstract · 69 pages. Available at https://www.wicinternet.org/2026-11/03/c_1224363.htm

Frontier AI Safety and Governance 2026: Risk Understanding and Coordinated Governance is a key outcome of the AI Safety and Governance Program of World Internet Conference Specialized Committee on AI in 2026. Drawing on the latest developments in frontier AI capabilities and applications, the report systematically examines the evolving risk landscape, analyzing key risk areas, their severity and response priorities. Building on a review of existing global governance practices and an assessment of frontier AI governance readiness, it identifies key governance gaps and weaknesses and proposes directions and practical mechanisms for coordinated responses. The report provides a reference for government departments, international organizations, businesses, research institutions and other stakeholders. It aims to strengthen the international community's understanding of and consensus on frontier AI risks, enhance preparedness and collective response capacity, and advance the safe, reliable and responsible development of frontier AI to better serve the common well-being of humanity.

0.41

Quantifying System-Level Harms from AI Adoption in Complex Sociotechnical Systems

cs.AI ·2026-08-24 ·Paul Vautravers, Oliver Chalkley, Gabriel Downer, Kate S, Damian Ruck
published similarity 0.65
Abstract

Artificial Intelligence (AI) is increasingly integrated into complex sociotechnical systems, including Critical National Infrastructure (CNI), where harms emerge from interactions between technical, human, and organisational elements. Yet current AI evaluation remains model-centric, offering little insight into how observed behaviours might translate into system-level risk. We propose a framework that links structured hazard analysis, component-level testing, and probabilistic system modelling to bridge this gap. By providing a traceable pathway from model behaviour to system-level outcomes, the framework enables practitioners to answer the "so what?" of AI failures, quantify their systemic impact, and move toward evidence-based and anticipatory governance of AI in complex systems. Applied to the UK's Real Time Gross Settlement (RTGS) system as an illustrative worked example, we derive AI-driven loss scenarios using Systems Theoretic Process Analysis (STPA) and examine adversarial manipulation of LLM-based trading as one such loss scenario. Component-level experiments show that simple adversarial inputs induce measurable behavioural shifts where AI recommendations are followed. Under the component-to-system mapping used here for a financial contagion model, these shifts alter system resilience, increasing bank failures and lowering the threshold at which shocks lead to cascading disruption, particularly under widespread or monopolistic AI adoption.

0.17

AI Alignment Is Neither Solvable nor Sufficient

cs.AI ·2026-11-09 ·Gerrit Großmann
position similarity 0.64
Abstract

Humans and AI systems increasingly act within the same social and economic environments. Alignment, the goal of ensuring that AI systems act in accordance with human values, is widely regarded as a central challenge for ensuring that AI benefits society while avoiding harm. We argue that this endeavor is misguided. First, alignment is not well defined, as it presumes a sufficiently stable and identifiable set of human intentions or values. Second, where a concrete notion of alignment can be specified, it is not solvable. Third, where alignment appears solvable, solving it would not necessarily mitigate the negative consequences of AI. A key reason is that alignment is typically framed as a property of individual AI systems, while the resulting hybrid societies of humans and machines are complex systems shaped by interactions and feedback loops. Emergent risks, and not only those posed by individual misaligned systems, are therefore likely to become a major source of harm. We conclude that AI safety is an ongoing challenge and not a problem that can be solved once and for all by verifying a particular model property.

0.18