CS Paper Radar

Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression

T. Bauer, W. P. Kegelmeyer, E. Begoli, A. Sadovnik, T. Emerson, C. Corley, N. Generous, J. Moore, B. Bartoldson, R. Goldhahn, M. Goldman, M. Greaves, M. J. D. Vermeer, B. MacLennan, D. Schulker, N. VanHoudnos, J. Bansemer, Y. Bengio

cs.CY cs.AIcs.HC ·submitted 2026-09-02 22:13 UTC · v2

Abstract

This article presents a structured framework of behavioral indicators that may signal progression toward potentially catastrophic threats from artificial intelligence systems. We adopt a pragmatic approach, inspired by established methodologies in cybersecurity and national security. By establishing clear metrics, indicators, and thresholds across multiple dimensions of AI capability and behavior, this framework enables researchers and policymakers to implement evidence-based monitoring protocols.

Closest papers

Nearest neighbours by meaning across the whole index.

Toward Responsible AI-Augmented Cyber Defense: Pattern Recognition, Defense-in-Depth, and the Case for Human-AI Collaboration

cs.CR cs.AI ·2026-09-22 ·Mustafa S. Aljumaily, Hayder Kareem Abed, Nawar S. Alseelawi
code/data similarity 0.69
Abstract · 11 pages, 3 figures, 2 tables. Original theoretical and modeling contribution. Simulation code: https://github.com/nawaralseelawi/ai-augment…

Cybersecurity literature has extensively documented the operational benefits of artificial intelligence (AI) for threat detection, incident response, and prevention, while raising qualitative concerns about over-automation, algorithmic bias, and analyst-skill erosion. What remains largely absent is a formal, falsifiable model connecting three constructs that recur across this literature: Defense-in-Depth Theory, the Artificial Intelligence Theory of Pattern Recognition, and human-AI collaboration in security operations. This paper develops such a model. We formalize layered defense as a Bernoulli detection cascade in which AI augmentation enters multiplicatively across layers; we formalize each layer's pattern-recognition behavior as a Neyman-Pearson/Bayesian detector with a derived closed-form optimal threshold; and we formalize human-AI triage as a capacity-constrained cascade with an explicit, quantifiable trade-off between detection probability and false-alarm ("alert fatigue") rate. A Monte Carlo/analytical simulation evaluated at illustrative but realistic operating points shows that (i) AI augmentation compounds across defense layers, delivering its largest marginal gains exactly where traditional layering saturates, and (ii) full human review of AI-flagged alerts is not optimal: increasing analyst capacity toward 100% coverage cuts false alarms by roughly 20-fold but simultaneously lowers system-level detection probability, because imperfect analyst accuracy is then applied to every alert rather than a filtered subset. These results give the widely repeated qualitative recommendation of "balanced human-AI collaboration" a precise, testable form and suggest an interior-optimum capacity ratio as a concrete design target for security operations centers (SOCs), including those securing IT/OT-converged critical infrastructure.

0.30

Governing at Machine Speed: An Adaptive Intelligence Architecture for Real-Time AI Policy Enforcement

cs.AI ·2026-09-11 ·Sandeep Bokkasam, B. Durgalakshmi
similarity 0.69
Abstract · 9 pages, 3 tables, 20 references

Enterprise AI adoption has reached 78% of organizations globally, yet the infrastructure to govern that adoption has not kept pace. This paper identifies and characterizes the attestation deficit, a structural condition in which organizations maintain governance policies but cannot produce auditable, tamper-evident evidence of enforcement within regulatory timelines. Drawing on empirical data from the Stanford 2026 AI Index Report (362 documented incidents), the IBM/Ponemon 2026 Cost of a Data Breach study (USD 4.99M average cost, 92% lacking access controls), and the EY/AIUC-1 Consortium survey (38% end-to-end monitoring, 17% agent-to-agent coverage), this paper demonstrates that the governance failure is organizational and architectural rather than technical. To address this deficit, we propose AGIL (Adaptive Governance Intelligence Layer), a conceptual five-layer architecture designed to use machine learning for real-time AI governance enforcement. The proposed layers include: (1) Autonomous Discovery for shadow AI detection via behavioral fingerprinting, (2) Behavioral Risk Classification unifying security, hallucination, privacy, and accountability scoring, (3) a Policy Enforcement Gateway for inline permit/deny/modify decisions at sub-100ms latency, (4) a Continuous Attestation Engine generating tamper-evident audit trails as a byproduct of enforcement, and (5) Adaptive Policy Intelligence for ML-driven policy evolution across jurisdictions. AGIL is presented as a theoretical framework and architectural proposal; empirical validation through controlled deployment remains a direction for future work.

0.26

From Guidelines to Practice: Defining Requirements for AI Red-Teaming Tools

cs.CY ·2026-11-03 ·Gia Phong Tran, Thai Duong Huynh, Randolph Loh, Balachandar Gowrisankar, Zihao Wang +4 more
similarity 0.69
Abstract

Artificial Intelligence (AI) has seen significant developments and increased usage in recent years. AI benefits various tasks, automates decision-making, and enhances efficiency. However, AI also introduces additional attack surfaces for threat actors to perform attacks like prompt injection and jailbreaking, making security a great concern. AI security testing aims to imitate adversarial attacks or malicious inputs via manual testing or with the assistance of tools. Thus, it is critical that organizations are aware of the capabilities and limitations of the tools before using them. Existing studies provide insights on the overall architecture of AI security testing tools and raise concerns on their reliability, but their analysis is limited to attack performance. Selection of a tool for AI risk assessments should also depend on their alignment with the recommendations of industry standards, but to our knowledge, previous studies have not utilized such guidelines. To bridge this gap, we analyze several industry standards and guidelines to identify the requirements of AI testing tools. We discuss and highlight desirable features across various AI security testing tools for a variety of different AI systems. We evaluated six tools across six Large Language Models, reported on our insights, and identified gaps that should be considered by future tools.

0.14

Beyond Predictable Paths: Redefining AI Security Incident Reporting for Agents

cs.CR cs.AI ·2026-09-21 ·Anastasia Pustozerova, Eugene Bagdasarian, Luca Beurer-Kellner, Battista Biggio, Nico Ebert +20 more
similarity 0.69
Abstract · under submission, mega paper (authorship does not imply endorsement of every sub-section)

AI agents are being deployed rapidly, accompanied by a growing number of AI-specific attacks and corresponding incidents. As incident reporting becomes increasingly important for legal compliance, governance, accountability, and security; current frameworks must be adapted to the unique characteristics of AI agents. In this paper, two editorial authors compare AI systems and AI agents and, drawing on input from 23 experts in academia and industry, identify the information required for reporting incidents where the security of AI agents is harmed. %involving AI agents. Potential reporting elements include, for example, agent memory and memory accesses, actual and potential levels of autonomy, and tool usage. Based on these findings, we identify several open research questions, including how to efficiently record incidents and how to determine whether vulnerabilities and incidents generalize. Expert feedback also highlighted potential reporting weaknesses, such as risks of data leakage and attacks targeting the reporting infrastructure itself, creating additional research needs. Lastly, we summarize privacy requirements and outline research directions for the secure and trustworthy deployment of AI agents.

0.18

AI Persuasion as a Threat to Human Control

cs.AI ·2026-09-13 ·Joshua Levy, Mick Yang, Kellin Pelrine
code/data similarity 0.67
Abstract

The threat that AI persuasion poses to human control has been acknowledged in the literature, but not yet systematically studied. Now that persuasion attacks are no longer theoretical - with Anthropic's Claude Mythos 5 recently making headlines for trying to convince people involved in an open-source project to merge malicious code during an evaluation - there is a pressing need to deeply analyze this threat. We undertake that effort here. In particular, we analyze how AI could persuade humans in key settings (e.g. safety-relevant R&D within frontier labs) toward decisions that compromise the development, containment, oversight, and governance of AI itself. In doing so, we elucidate a framework for characterizing this threat, develop five concrete scenarios using this framework, and provide a blueprint for assessing the associated risks. Using this blueprint, we conduct an initial risk estimation survey with select researchers and find that their opinions on which scenarios are riskiest are highly mixed. Their disagreements stem from differing opinions about the effectiveness of AI persuasion in different contexts, and point to the need for follow-up risk elicitation studies and persuasion evaluations, which we outline. Our hope is that this paper highlights the risks from AI persuasion undermining control, and provides a path forward for future research.

0.34

Explainable Artificial Intelligence for Industrial Cybersecurity: A Review of Methods, Operational Integration, and Research Challenges

cs.CR eess.SY ·2026-08-31 ·Amr S. Mohamed, Charlotte Fritz, Ahmad Mohammad Saber, Yiqun Ma, Ratinder Kaur +3 more
survey similarity 0.67
Abstract · 31 pages, 6 figures

The increasing digitalization of industrial infrastructure and the convergence of information technology (IT) and operational technology (OT) have expanded the cyberattack surface of industrial systems. To address the growing complexity of cyber threats, artificial intelligence (AI) and machine learning (ML) techniques are increasingly deployed within industrial cybersecurity operations, particularly in Security Operations Centers (SOCs). While these approaches improve anomaly detection, threat analysis, and automated response, their opaque decision-making presents challenges for operational trust, regulatory compliance, and incident response. EXplainable Artificial Intelligence (XAI) has emerged as a promising paradigm to improve the transparency and interpretability of AI-driven cybersecurity systems and decisions. This paper provides a comprehensive review of XAI techniques in industrial cybersecurity, focusing on industrial SOC environments and operational security workflows. We examine the role of AI in industrial SOC workflows, the types of operational data leveraged in industrial environments, and the benefits and limitations of AI-based threat detection. We then review major families of XAI approaches, including feature attribution methods, surrogate models, rule-based explanations, and visualization techniques, and analyze their applicability to industrial use cases. We further discuss the operational, regulatory, and safety requirements that distinguish industrial systems from traditional IT environments. Key challenges are examined, including limited labeled datasets, model reliability, explainability-performance tradeoffs, and the integration of XAI tools into SOC workflows. Finally, we identify open research directions and opportunities for developing trustworthy, operationally viable, and domain-specific XAI-enabled cybersecurity solutions for industrial environments.

0.26

From Sandbox to Enforcement: Confidence-Qualified Threat Intelligence for Critical Infrastructure

cs.CR cs.AI ·2026-10-05 ·Nikolaos Kekatos, Mihaela Curcă, Georgios Koutidis, Mihai Nena, Tom Nianios +3 more
published similarity 0.67
Abstract · 20 pages, 3 figures. Accepted at the 21st International Conference on Critical Information Infrastructures Security (CRITIS 2026)

Security operations centres and national incident-response teams defending critical infrastructure collect abundant threat data yet struggle to turn it into actionable intelligence. A malware sandbox produces detailed behavioural evidence, but as a large, unranked report whose confidence is unstated. We present CG-CTI, an operational pipeline that converts live sandbox output (CAPEv2) into STIX 2.1, correlates it in a knowledge graph with other critical-infrastructure sensors, and attaches to every intelligence object an explicit confidence status derived from provenance, cross-source corroboration, and observation durability. This status gates automated action: only corroborated intelligence is eligible for automated enforcement, while lower-confidence objects are routed to analyst review or kept as context. A grounded language-model stage then narrates the confidence-qualified evidence, where each statement either cites a supporting object or is marked unsupported, so fabricated references are removed before analyst review. We implement CG-CTI within the CYBERGUARD project, whose consortium includes Romania's national cyber-security directorate, and evaluate it against the live sandbox on a labelled malware corpus, measuring conversion validity, indicator yield, technique coverage, corroboration, enforcement eligibility, latency, and summary grounding. CG-CTI turns fragmented sandbox output into corroborated, confidence-ranked, and auditable intelligence for critical-infrastructure defence.

0.28

Open Problems in AI Risk Modeling: Insights from a Workshop on the Technical Foundations of AI Risk Modeling

cs.CY ·2026-09-02 ·Krystal Jackson, Deepika Raman, Jakub Kryś, Sean P. Fillingham, Jack Kengott +6 more
similarity 0.66
Abstract

We investigate the design of robust risk models to assess societal risks posed by advanced AI systems, an emerging area in AI governance. Many regulatory proposals increasingly require systemic risk assessment, but in the absence of rigorous quantitative methods, the question remains what state of the art risk modeling should look like in practice. We identify the key methodological and institutional challenges that currently limit the adoption of risk modeling. We review five research traditions that inform this problem: probabilistic risk assessment, catastrophic AI risk analysis, cybersecurity risk quantification, Bayesian causal inference, and threshold-based governance. We compare two leading proposals, scenario-based risk estimation and Bayesian network-based threshold setting. Drawing on a workshop with 22 experts and subsequent analysis, we identify a structured agenda of open questions concerning model structure, scope, evidence integration, validation, and governance. We close by outlining priorities for progress, arguing that it will depend on integrating quantitative modeling with independent evaluation, transparent and tiered disclosure, and institutions capable of maintaining and updating risk models over time.

0.16