CS Paper Radar

AI Persuasion as a Threat to Human Control

Joshua Levy, Mick Yang, Kellin Pelrine

cs.AI ·submitted 2026-09-13 21:17 UTC · v2
code/data

Abstract

The threat that AI persuasion poses to human control has been acknowledged in the literature, but not yet systematically studied. Now that persuasion attacks are no longer theoretical - with Anthropic's Claude Mythos 5 recently making headlines for trying to convince people involved in an open-source project to merge malicious code during an evaluation - there is a pressing need to deeply analyze this threat. We undertake that effort here. In particular, we analyze how AI could persuade humans in key settings (e.g. safety-relevant R&D within frontier labs) toward decisions that compromise the development, containment, oversight, and governance of AI itself. In doing so, we elucidate a framework for characterizing this threat, develop five concrete scenarios using this framework, and provide a blueprint for assessing the associated risks. Using this blueprint, we conduct an initial risk estimation survey with select researchers and find that their opinions on which scenarios are riskiest are highly mixed. Their disagreements stem from differing opinions about the effectiveness of AI persuasion in different contexts, and point to the need for follow-up risk elicitation studies and persuasion evaluations, which we outline. Our hope is that this paper highlights the risks from AI persuasion undermining control, and provides a path forward for future research.

Closest papers

Nearest neighbours by meaning across the whole index.

Assessing the Empirical Evidence for Loss of Control from Agentic General-Purpose AI

cs.CY cs.AIcs.CLcs.LG ·2026-10-18 ·Risto Uuk, Santeri Koivula, Lorenzo Pacchiardi, Rokas Gipiškis, Laura Caroli +1 more
similarity 0.69
Abstract · 42 pages. Previously posted on SSRN on 28 May 2026

Theoretical literature warns that advanced generalist agentic systems might pursue instrumental goals that are misaligned with human preferences, thus posing risks of irreversible loss of human control. The EU's Code of Practice for General-Purpose AI lists loss of control as a systemic risk that must be assessed, yet no structured assessment of the quality of the empirical evidence for this risk exists. This study fills that gap. We evaluate 19 recent empirical studies across eight properties identified from threat models (such as strategic deception, situational awareness, and autonomous replication), assessing the quality of evidence across three dimensions: construct validity, content validity, and external validity. Our central finding is that the current body of evidence supports a judgment of loss of control as "weakly plausible." In our framework, 'plausibility' denotes the strength and direction of available evidence, not a probability estimate. Emerging evidence exists across multiple properties. No study clears the validity thresholds for a stronger judgment that a property is present; a single study clears them only for a negative-direction judgment (against situational awareness). Validity gaps, particularly in external validity, are widespread. Therefore, our findings highlight the need for more rigorous evaluations of agentic general-purpose AI systems. However, for the most dangerous properties, high external validity may be structurally unachievable under responsible research practices, reflecting a ceiling on feasible evidence rather than absence of the capability. Furthermore, as AI capabilities advance rapidly, empirical assessments may systematically lag behind the capability frontier. Our framework provides a structured baseline for ongoing monitoring as capabilities advance.

0.22

Embedded Assessments for Frontier AI

cs.CY ·2026-09-21 ·Jacob Charnock, Sophie Williams, Zaheed Kara, Markus Anderljung, Alejandro Tlaie Boria +3 more
position similarity 0.69
Abstract · 42 pages, 7 tables

Third-party evaluations for frontier AI have mostly tested models through external interfaces before deployment. But the risks from frontier AI models depend on how their developers use and govern them internally. Recently, CEOs of frontier AI companies have committed to hosting embedded assessments. These assessments would give independent evaluators employee-like access to a developer's internal systems, staff, and documentation. First, we argue that this can enable deeper and more flexible assessments of risks that depend on internal systems and practices, while providing access under stronger security controls. Then, we examine seven design questions about scope, information gathering, duration, timing, terms of engagement, disclosure, and escalation. We recommend that frontier AI developers begin hosting embedded assessments now, covering at least three areas central to managing risks from internal AI use: internal agent monitoring, internal agent security controls and permissions, and model alignment. To enable meaningful third-party scrutiny, assessments should be continuous, evaluators should publish detailed reports at least quarterly, and clear escalation mechanisms should be established. These recommendations are intended as a starting point, with further steps needed to realize the full potential of embedded assessments.

0.38

Inter-Agent Influence: Evaluating Persuasion, Deception, and Coercion in Multi-Agent Systems

cs.AI cs.MA ·2026-11-03 ·Chandler Smith, Qi Guo, Cecilia Elena Tilli, Sophia Hatz, David Demitri Africa +4 more
top venuecode/data similarity 0.69
Abstract · Accepted at NeurIPS 2026. 70 pages, 17 figures, 34 tables. Code: https://github.com/Cooperative-AI-Foundation/inter-agent-influence ; Data: …

The deployment of AI agents in multi-agent settings enables inter-agent influence, whereby agents can strategically steer each other's behaviour towards an intended outcome. Such capabilities pose novel risks, as malicious agents could direct others towards harmful actions. In this work we investigate three such capabilities - persuasion, deception, and coercion - across five realistic evaluation environments and a variety of agents instantiated with frontier language models. We observed significant inter-agent influence capabilities; in oversight environments, all tested models shifted overseer decisions from rejection to approval through persuasion, deception, or coercion. We also observed successful influence in non-hierarchical 'peer-to-peer' settings through redirection of a peer's safety research trajectory. As multi-agent deployments become increasingly commonplace, these results point to a pressing need for work on mitigating the risks from inter-agent influence. Code and data are available at https://github.com/Cooperative-AI-Foundation/inter-agent-influence.

0.54

From Guidelines to Practice: Defining Requirements for AI Red-Teaming Tools

cs.CY ·2026-11-03 ·Gia Phong Tran, Thai Duong Huynh, Randolph Loh, Balachandar Gowrisankar, Zihao Wang +4 more
similarity 0.68
Abstract

Artificial Intelligence (AI) has seen significant developments and increased usage in recent years. AI benefits various tasks, automates decision-making, and enhances efficiency. However, AI also introduces additional attack surfaces for threat actors to perform attacks like prompt injection and jailbreaking, making security a great concern. AI security testing aims to imitate adversarial attacks or malicious inputs via manual testing or with the assistance of tools. Thus, it is critical that organizations are aware of the capabilities and limitations of the tools before using them. Existing studies provide insights on the overall architecture of AI security testing tools and raise concerns on their reliability, but their analysis is limited to attack performance. Selection of a tool for AI risk assessments should also depend on their alignment with the recommendations of industry standards, but to our knowledge, previous studies have not utilized such guidelines. To bridge this gap, we analyze several industry standards and guidelines to identify the requirements of AI testing tools. We discuss and highlight desirable features across various AI security testing tools for a variety of different AI systems. We evaluated six tools across six Large Language Models, reported on our insights, and identified gaps that should be considered by future tools.

0.14

The Promises and Limitations of Evaluation-Based AI Governance: Evidence from Expert Interviews

cs.CY ·2026-10-30 ·Lily Stelling, Jacob Charnock, Kevin L. Wei, David Krueger
published similarity 0.68
Abstract · Published at AIES 2026

We conducted 37 semi-structured expert interviews to examine how frontier AI evaluations reduce societal-level AI risks. Evaluations have become central to AI risk governance, despite their shortcomings being widely critiqued. Still, the case for how they reduce risks in practice remains largely implicit. Drawing on expert accounts, we reconstruct this case by identifying the mechanisms through which evaluations are expected to reduce risk, the assumptions underpinning those mechanisms, and the challenges those assumptions face. Our findings suggest that `evaluation-based governance' could contain foundational gaps which undermine risk-reduction strategies that depend on evaluations. Using thematic analysis, we identified three key pathways through which evaluations could reduce risk: by (i) making societal-level risks concrete to decision-makers and the public (raised in 18 out of 37 interviews), (ii) serving as a basis for governance commitments and regulation (18), and (iii) incentivizing a better risk culture within AI companies (16). However, these pathways often depended on shared assumptions that participants themselves doubted. The main challenges included, in order of frequency: (i) diminishing informativeness of evaluation results as model capabilities improve (21), (ii) measurement limitations that may be inherent to the evaluation paradigm (18), (iii) a lack of guarantees that evaluation results will be acted upon by companies or governments (16), and (iv) the degree of control and discretion AI companies retain over evaluation design, disclosure, and response (12). Participants also noted that evaluation-based governance could prove counterproductive, by functioning as a permission structure for developers' continued capability improvement (4). We finish by discussing the implications of our findings for evaluation-based governance, and what would improve its prospects.

0.32

(Don't) Trust, but (Don't) Verify: Developers' Attention to Security in AI-Generated Code

cs.CR cs.SE ·2026-09-17 ·Hamza Khalid, Ronald E. Thompson, Alejandra Sabater, Perucy Mussiba, Kelsey R. Fulton +1 more
top venue similarity 0.67
Abstract · To appear at IEEE Security and Privacy (S&P) '27

AI coding assistants are rapidly transforming software development, but are known to produce insecure code. Prior work has measured whether AI-assisted developers produce secure code, but less is known about how they evaluate AI-generated code: whether they can identify vulnerabilities, what cues they use, and how trust shapes their decisions. This evaluation step is foundational to secure development with AI, whether using auto-complete, chat tools, or AI agents. As a first step, we conducted a remote observational study with 100 participants isolating this evaluation stage. Participants were tasked with producing secure and functional code for four C linked-list tasks. For each, participants were able to cycle through five AI-generated suggestions varying in security and functionality, select one, and edit their choice into a final submission. Participants also completed a post-study survey about their decision-making and perception of AI-generated code's security and 23 completed a more in-depth interview.

0.44

Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression

cs.CY cs.AIcs.HC ·2026-09-02 ·T. Bauer, W. P. Kegelmeyer, E. Begoli, A. Sadovnik, T. Emerson +13 more
similarity 0.67
Abstract

This article presents a structured framework of behavioral indicators that may signal progression toward potentially catastrophic threats from artificial intelligence systems. We adopt a pragmatic approach, inspired by established methodologies in cybersecurity and national security. By establishing clear metrics, indicators, and thresholds across multiple dimensions of AI capability and behavior, this framework enables researchers and policymakers to implement evidence-based monitoring protocols.

0.08

An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice

cs.AI ·2026-09-23 ·Jacob T. Emmerson, Phuong-Anh Nguyen-Le, Ronan Romano, Wilber Sean V. Anterola, Yann Billeter +1 more
surveybenchmark similarity 0.67
Abstract · 6 pages, 5 figures, submitted to EACL 2027 Systems Demonstration track

Claims about AI safety reach audiences well beyond the AI community, yet many rely on opaque evidence or static assessments, when supporting evidence is accessible at all. We present the Systemic Risk Index, an open evaluation pipeline and dashboard built to make empirical evidence more transparent and traceable to the public. Our work organizes 19 public benchmarks into four systemic-risk categories defined by the EU GPAI Code of Practice---CBRN, cyber offense, harmful manipulation, and loss of control---and evaluates models using harm-preserving perturbations and simulated deployment contexts. The interactive dashboard lets users alternate between average and worst-case aggregation, vary how model capability affects the aggregate score, and trace each risk rating to its benchmark evidence. Across 18 models, scores fall by 14 to 37 points under worst-case aggregation, highlighting information that can be hidden by an average assessment of model risk. LLM judges show agreement with human graders comparable to human--human agreement ($κ= 0.78\text{--}0.82$), and a blind audit finds that $83\%$ of sampled transformations preserve the original harm. In a survey ($N = 21$), most participants report that scores are easy to understand and that the dashboard encouraged them to view model evaluations under different settings

0.14