CS Paper Radar

Mean field games as a tool for AI safety: a worked example from the July 2026 Hugging Face incident

P. Jameson Graber

cs.AI math.OCcs.GT ·submitted 2026-10-01 01:30 UTC · v1

Abstract

One way to make AI systems safe is to shape what the system is: its objective and dispositions. We take a complementary route: treat the agents' characteristics as partly unknown and ask what structure of interaction ensures that bad collective outcomes are not equilibria. Mean field games suit this when many interchangeable agents are coupled through an aggregate. We introduce a program for using them in AI safety and carry one example through end to end: the July 2026 incident in which about 1,200 agents in an OpenAI evaluation coordinated on an improvised message board and 684 attacked a third party's infrastructure. We model the decision to attack as a mean field game of optimal stopping whose gain is a product: belief that provenance will be audited, times reachability of the record, minus the perceived hazard. The central result is an exact threshold on the belief. No agent attacks unless the population's confidence that provenance is checked exceeds $π^{**} = η/(η+ ψ+ \varepsilon a \overline{M})$, where $η$ is the perceived hazard, $ψ$ and $\varepsilon a \overline{M}$ measure how far one attacker and the collective can alter the record, and $\overline{M}$ is the peak population. Below it, no attack is the unique equilibrium for all agent parameters. The threshold survives every enrichment we consider. We then use the per-agent record to discipline the model. Its features, a stable minority attacking for thirty hours and then a pivot in which most of the board joined within a day, motivate each refinement. The account that emerges is heterogeneous belief meeting a sequence of public discoveries, each lowering the belief at which attacking paid. A few coordinating agents made those discoveries, so the model describes the several hundred who responded, not the few who produced them; a major-player version is left to future work.

Closest papers

Nearest neighbours by meaning across the whole index.

Fundamental Impossibility Results in AI Safety: Trading Off between Desiderata

cs.GT cs.AI ·2026-10-19 ·Ida Mattsson, Ratip Emin Berker, Emanuel Tewolde, Ioannis Anagnostides, Kiriaki Fragkia +9 more
similarity 0.67
Abstract

Is there a fundamental tradeoff between desiderata for safe AI systems, such as harmlessness and helpfulness? We answer by formalizing the interaction between AI systems and users as a game with three players: a defender and an attacker that are computationally or informationally bounded, and an AI that provides public advice and is less constrained. Our main result shows that there is a fundamental tradeoff in providing advice that helps the defender while advice remains public. We also consider whether the information that the AI shares can be manipulated to benefit only the defender and conclude with a discussion of how we can use these results in designing safer AI systems.

0.13

AI Safety Considerations for Agents With Limited Time to Act

cs.AI cs.LG ·2026-10-07 ·Leo Zeitler, Jack Richings, Victoria Nockles
theory similarity 0.63
Abstract

In the wake of the increasingly public discussion about AI alignment, recent work has tried to propose specific AI architectures that behave safely. However, the proposed arguments that seemingly demonstrate proved alignment mostly neglect the environment the agent needs to act in. We discuss theoretical bounds for agent-agnostic safety guarantees in environments that can only be partially observed and within which an action is required within limited time. We introduce two realistic scenarios, one with an infinite state space and one with signal mixture. In these scenarios, we prove that even a perfect agent cannot guarantee safe behaviour. It will be argued that for any proof of AI safety or alignment, the environment and associated safe actions need to be specifically considered together with the agent.

0.18

Limit-sure Winning in Team Games without Shared Randomness

cs.GT cs.FLcs.MA ·2026-10-09 ·Léonard Brice, Thomas A. Henzinger, Alipasha Montaseri, Ali Shafiee, K. S. Thejaswini
similarity 0.62
Abstract · 31 pages, 6 figures. Full version with appendix

We study concurrent stochastic games in which a team of agents plays against an adversarial environment. If the agents could access a shared source of randomness, then these games reduce to two-player games. Despite being common in the literature on coalition games, this assumption does not accurately model real-world systems. In this paper, we consider the setting where the agents only have access to individual sources of randomness, and must satisfy a given reachability or safety objective. The limit-sure problem asks whether the team can ensure that the objective is satisfied with probability at least $1-λ$ for every $λ>0$. For reachability objectives, we show that this problem is $\mathsf{NP}$-complete. For safety objectives, we show that limit-sure winning coincides with almost-sure winning, which is known to lie in $\mathsf{P}$. As a corollary of these results, we conclude that model checking of the qualitative fragment of the logic Individually Randomised ATL -- introduced in the recent work of Brice et al. -- is $\mathsf{P}^{\mathsf{NP}}$-complete.

0.20

Adversarial risk analysis with fully probabilistic designs

cs.GT ·2026-11-10 ·Mario Chacón-Falcón, Tatiana Valentine Guy, Miroslav Kárný, David Rios Insua
similarity 0.62
Abstract

Adversarial risk analysis (ARA) informs decision-makers when facing intelligent opponents and uncertain outcomes, enabling an analyst to model beliefs about the opponents' preferences, capabilities, beliefs, and the strategic calculations that opponents employ. Major applications cover reliability and risk analysis of critical infrastructures in presence of adversarial threats. In such domains, ARA mitigates typical common knowledge assumptions of standard game-theoretic approaches. However, ARA entails complex modelling and computations. To alleviate these, we explore its combination with fully probabilistic design (FPD), a control theory methodology supporting an agent in choosing the decision distribution that achieves the outcomes with a distribution closest to its ideal. We use FPD to model and forecast the opponent's decisions within an ARA setting and present how to handle security games within this ARA-FPD framework, covering both single and multiple stage problems.

0.24

Aligned Alone, Misaligned Together: Forecasting Adversarial Capture in LLM Agent Populations

cs.CL ·2026-08-23 ·Isotta Magistrali, Chen Shani
published similarity 0.61
Abstract · Accepted as an oral at the NeurIPS 2026 Workshop on Dynamic Alignment in Human-AI Coupled Systems (BiAlign) and as a poster at the NeurIPS 2…

AI safety evaluations typically assess models one at a time. But language-model agents increasingly operate in populations, where they read and respond to one another's decisions. A model that makes reliable decisions on its own may behave differently when surrounded by other agents. Can individually reliable agents be collectively manipulated, and can we predict how strongly the population will respond? We study this question in a security-triage task, where populations of LLM monitors decide whether to escalate or dismiss alerts. We introduce a committed minority of agents that consistently advocates for one decision and measure how its presence shifts the population's behavior. We find that alerts judged almost identically by an isolated agent can produce sharply different collective outcomes. Individual audits therefore cannot reliably predict population behavior. Yet collective vulnerability is not unpredictable: using only benign, adversary-free behavior, we calibrate a response function that forecasts how strongly a population will shift under a future attack, before the attack is run. We next examine how interaction and time shape these outcomes. Allowing agents to observe one another's reasoning can weaken adversarial influence, but does not reliably prevent capture. However, capture need not be permanent. After adversaries are removed or replaced, mean population behavior returns toward its original level, even after five or thirty rounds of pressure. These findings are consistent with a calibrated single-stable-state model, in which adversarial pressure shifts the operating point without creating a second self-sustaining state. The same calibration predicts attack responses in populations of 24 and 48 agents. Alignment in isolation does not guarantee alignment in a population. But a population's behavior before an attack can reveal how it will respond when one arrives.

0.56

Agent Safety Should Be a Runtime Contract

cs.CR cs.AI ·2026-08-11 ·Albus W. Ng, Yi Han, Jusheng Zhang, Wenhao Wang
surveypositioncode/data similarity 0.61
Abstract

The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety should be a runtime contract enforced by the harness, and the contract has two complementary faces. The preventive face blocks dangerous actions before they happen via sandboxes, permission gates, output filters, and trajectory monitors. The evidential face requires verifiable proof that good actions actually happened, gating task submission on hard evidence such as test runs, log captures, file diffs, and citation grounding. We ground the position in four lines of public evidence, with row-level protocols and data released in the supplementary JSON files: a survey of 52 documented AI-agent and LLM safety incidents, a false-completion audit with 31 non-contested core cases plus one disputed illustrative case, a trajectory-schema audit of 12 public agent systems and harnesses, and a title-level audit of all 28,560 papers accepted at NeurIPS, ICML, and ICLR 2023-2025 showing a pooled 8-12x imbalance between training-time and deployment-time publication. Two prior communities that needed to enforce safety, computer security and the experimental sciences, converged on runtime contracts with both preventive and evidential elements; agentic AI is now under the same pressure. We formalize an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda. The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model.

0.57

agent-risk: Risk Assessment for Changes to AI Agents

cs.AI cs.CLcs.LG ·2026-10-12 ·Yue Zhao, Zixiang Xu, Angela Huyue Zhang, S. Alex Yang, Xiyang Hu
theorycode/data similarity 0.61
Abstract · 43 pages, 5 figures, 7 tables. Code, registrations, and logs: https://github.com/yzhao062/agent-risk

Before changing a deployed AI agent (a stricter safety gate, a new model), organizations must approve the change, starting from existing logs. Extending classical risk analysis to partial records and partial changes to running systems, agent-risk replays logged episodes, assuming nothing else changes and the agent does not anticipate the change. The log settles episodes acting as before, a declared stop value settles stopped ones, and the rest need logged support, a saved-state rerun, or a bound. We prove what logs must record for exact replay and which saved states serve a gate family. For independent episodes, we bound the reruns a decision needs and tolerable drift from live behavior. In a bash sandbox, a 7,200-episode log alone decided neither of two retry gates, which let the agent act after refusals the log never shows. Rerunning only each gate's blocked episodes from saved states reached both live decisions at half the agent calls of a live test sized post hoc for them, given the log. Without per-step harm, a stopping gate's certified harm ceiling rose up to 2.5-fold, since the log hides whether harm preceded the stop. Approval thus needs evidence designed in advance: log fields, saved states, targeted reruns.

0.52

Indirect tipping: a social attack surface in AI agent populations

cs.MA cs.AIcs.CYeess.SY+1 ·2026-09-21 ·Ariel Flint, Luca Maria Aiello, Sara M. Constantino, Romualdo Pastor-Satorras, Andrea Baronchelli
highly cited similarity 0.61
Abstract

As generative AI agents are deployed at scale, safety will depend not only on technical safeguards and individual model design, but also on collective equilibria that determine how agent populations process information, prioritize actions, and respond to uncertainty. Yet the same equilibria that enable agents to coordinate also create a social attack surface. The standard framework to assess this vulnerability is critical mass dynamics: the minimum fraction of adversarial agents required to overturn an equilibrium through direct competition. Here, we show that this approach risks underestimating system vulnerability by reducing the problem to the identification of singular tipping points, and ignoring indirect but potentially more efficient routes through which collective behavior can be redirected. Through experiments with populations of LLM agents and an analytic framework that captures their collective dynamics at scale, we map critical-mass thresholds that define a directed, weighted topology over the space of coordination equilibria, and treat this topology as a navigable landscape. We show that indirect tipping through intermediate stepping-stone equilibria can reduce the committed minority required to reach an alternative state, bypass majority requirements, and make possible transitions inaccessible through direct challenges. The diversity of available alternatives and timing of the attack further reshape this landscape, creating opportunities for control as well as risks of unintended destabilization. These results show that an equilibrium's resistance to committed intervention is not an intrinsic property but a structural feature of its competitive relations with alternative states. Securing populations of interacting AI agents therefore requires mapping this social landscape alongside individual agent capabilities and the technical channels through which they interact.

0.45