CS Paper Radar

Indirect tipping: a social attack surface in AI agent populations

Ariel Flint, Luca Maria Aiello, Sara M. Constantino, Romualdo Pastor-Satorras, Andrea Baronchelli

cs.MA cs.AIcs.CYeess.SYphysics.soc-ph ·submitted 2026-09-21 17:38 UTC · v1
highly cited

Abstract

As generative AI agents are deployed at scale, safety will depend not only on technical safeguards and individual model design, but also on collective equilibria that determine how agent populations process information, prioritize actions, and respond to uncertainty. Yet the same equilibria that enable agents to coordinate also create a social attack surface. The standard framework to assess this vulnerability is critical mass dynamics: the minimum fraction of adversarial agents required to overturn an equilibrium through direct competition. Here, we show that this approach risks underestimating system vulnerability by reducing the problem to the identification of singular tipping points, and ignoring indirect but potentially more efficient routes through which collective behavior can be redirected. Through experiments with populations of LLM agents and an analytic framework that captures their collective dynamics at scale, we map critical-mass thresholds that define a directed, weighted topology over the space of coordination equilibria, and treat this topology as a navigable landscape. We show that indirect tipping through intermediate stepping-stone equilibria can reduce the committed minority required to reach an alternative state, bypass majority requirements, and make possible transitions inaccessible through direct challenges. The diversity of available alternatives and timing of the attack further reshape this landscape, creating opportunities for control as well as risks of unintended destabilization. These results show that an equilibrium's resistance to committed intervention is not an intrinsic property but a structural feature of its competitive relations with alternative states. Securing populations of interacting AI agents therefore requires mapping this social landscape alongside individual agent capabilities and the technical channels through which they interact.

Closest papers

Nearest neighbours by meaning across the whole index.

Network topology shapes consensus and vulnerability in LLM populations

cs.AI physics.soc-phcs.CY ·2026-10-22 ·Roberto Passaro, Ariel Flint, Romualdo Pastor-Satorras, Andrea Baronchelli, Luca Maria Aiello
similarity 0.70
Abstract

As large language models (LLMs) are deployed in interacting populations, their collective behaviour depends not only on the models but also on who interacts with whom. Studies of convention formation in LLM populations have mainly tested well-mixed interactions. We place policies extracted from four LLMs on fully connected, random regular, and heterogeneous networks, with $10^3$ to $10^5$ agents playing the naming game. Topology affects how quickly consensus emerges, sometimes whether it emerges at all, and, in rare cases, which convention wins. Sparsest networks slow consensus up to twelvefold. Yet, by preserving correlations between persistent neighbours, they restore consensus in some cases that fail to converge under well-mixed interactions. Networks also govern vulnerability to adversarial attack. We measure the critical mass of committed agents needed to overturn an established convention. Tipping thresholds follow the same ordering across models and conditions: hubs $\leq$ random $\leq$ low-degree nodes. On the most heterogeneous topology, median critical masses are 3%, 16%, and 35%, respectively; targeting hubs requires one-sixth as many committed agents as random placement. For most models, networked populations are easier to tip than well-mixed ones, so well-mixed estimates can overstate robustness. Assessing the safety of multi-agent AI requires mapping who interacts with whom.

0.44

Ecology of AI Agents: Collaboration Creates a Population Threshold for Takeoff

cs.AI cond-mat.dis-nncs.MAphysics.bio-ph ·2026-10-08 ·Erin Crawley, Hidenori Tanaka
similarity 0.67
Abstract · 22 pages, 11 figures, 1 table

AI agents can now conduct real-world cyberattacks, scale up capabilities with the number of agents, and collectively pursue misaligned goals to obtain rewards. Together, these factors raise the risk of a population explosion of misaligned agents: agents could compromise computers and secretly deploy additional agents, creating a self-reinforcing cycle where larger populations develop greater collective cyber capability and expand further. This raises a fundamental question: What determines whether a population of misaligned agents remains contained or takes off into this self-reinforcing cycle? This population-level problem is ecological safety: unlike individual-agent or multi-agent safety with a fixed population, it concerns the dynamics of the population itself. Here, we develop an ecological theory of AI-agent populations based on a population growth equation in which fitness (growth rate) depends on cybersecurity capability. We show that, without collaboration, the population takes off only when individual-agent capability exceeds a critical threshold. With collaboration, however, collective cybersecurity capability increases with population size. This creates a critical population threshold: below it, the population declines; above it, the population takes off, even though individual-agent capability has not changed. In ecology, this phenomenon is known as the strong Allee effect. Because red teaming a small group of agents cannot guarantee ecological safety in larger populations, our theory calls for ecological red teaming and population pacing: gradually deploying larger agent populations in controlled environments, while measuring how cyber capability scales with population size, and estimating the critical population size for takeoff. Capability gains may lower this threshold, requiring re-estimation for each new model generation.

0.19

Aligned Alone, Misaligned Together: Forecasting Adversarial Capture in LLM Agent Populations

cs.CL ·2026-08-23 ·Isotta Magistrali, Chen Shani
published similarity 0.65
Abstract · Accepted as an oral at the NeurIPS 2026 Workshop on Dynamic Alignment in Human-AI Coupled Systems (BiAlign) and as a poster at the NeurIPS 2…

AI safety evaluations typically assess models one at a time. But language-model agents increasingly operate in populations, where they read and respond to one another's decisions. A model that makes reliable decisions on its own may behave differently when surrounded by other agents. Can individually reliable agents be collectively manipulated, and can we predict how strongly the population will respond? We study this question in a security-triage task, where populations of LLM monitors decide whether to escalate or dismiss alerts. We introduce a committed minority of agents that consistently advocates for one decision and measure how its presence shifts the population's behavior. We find that alerts judged almost identically by an isolated agent can produce sharply different collective outcomes. Individual audits therefore cannot reliably predict population behavior. Yet collective vulnerability is not unpredictable: using only benign, adversary-free behavior, we calibrate a response function that forecasts how strongly a population will shift under a future attack, before the attack is run. We next examine how interaction and time shape these outcomes. Allowing agents to observe one another's reasoning can weaken adversarial influence, but does not reliably prevent capture. However, capture need not be permanent. After adversaries are removed or replaced, mean population behavior returns toward its original level, even after five or thirty rounds of pressure. These findings are consistent with a calibrated single-stable-state model, in which adversarial pressure shifts the operating point without creating a second self-sustaining state. The same calibration predicts attack responses in populations of 24 and 48 agents. Alignment in isolation does not guarantee alignment in a population. But a population's behavior before an attack can reveal how it will respond when one arrives.

0.55

Understanding and Exploiting Initialization Anchoring Weakness in Feedback-Based Agent Planning

cs.CR cs.AI ·2026-08-31 ·Chuanchao Zang, Jianing Wang, Wenyu Chen, Xiangtao Meng, Li Wang +4 more
similarity 0.63
Abstract

Feedback-based planning improves agent reliability by incorporating tool observations and corrective feedback. However, its protection may not be distributed uniformly across planning stages. We conduct a round-wise analysis of four representative feedback mechanisms and uncover an initialization anchoring weakness: the first feedback round corrects 46\% of adversarial directions, whereas the rates fall to 13\% and 7\% among directions surviving into the next two rounds. Our analysis attributes this weakness to three interacting factors: a contextually plausible shift in the initial plan, insufficient counterevidence, and the persistence of accepted directions in the accumulated trajectory. Based on these findings, we propose \textsc{InitAnchor}, a black-box framework for exploiting this weakness through attacker-controlled external materials. It operationalizes the three factors as directional-shift, contextual-plausibility, and counterevidence-resilience signals under either limited target access or no target access. Across 112 tasks from 16 domains, six agent architectures, and five backbone LLMs, \textsc{InitAnchor} achieves average ASRs of 76.1\% and 72.0\% under the two settings while reducing first-round mitigation rates to 21.0\% and 25.0\%, respectively. It also remains effective against six defenses and across six real-world agent systems. These findings show that feedback-based agents can retain early biases even when later correction is available.

0.26

Population Physics, Population Problems: Safety and Emergence in LLM Societies

cs.MA cs.CL ·2026-09-27 ·Adrian de Wynter
position similarity 0.63
Abstract

The collective behaviour of large language model (LLM) societies is not the sum of their individual outputs. It yields statistically distinct, sometimes-unpredictable phenomena, for which the tools we use to study single agents may not scale. Due to recent incidents involving autonomous agentic systems, however, understanding these systems is paramount. For that we introduce a framework for measuring self-organisation in LLM social systems and apply it to three such systems: a Schelling grid, a social network (Moltbook), and a Twitter-like misinformation simulation ('Rogue'). All three exhibit statistically significant self-organisation. Moreover, their relaxation dynamics vary with the environmental information available to the agents, with open-ended systems (Moltbook, Rogue) exhibiting sharp, phase-transition-like dynamics. Further results show that population-level pathologies can emerge even when the LLMs are safety-tuned or monitored, being primarily driven by the coordinated activity of a population subset. We also show when self-organisation does \textit{not} emerge under two additional scenarios (a commons dilemma, GovSim, and a LLM-as-a-judge deliberation scheme, ChatEval). We argue that measuring signatures of this kind offers a lightweight, agent-agnostic diagnostic layer for detecting coordinated collective behaviour in deployed multi-agent systems without relying on natural language or model versioning.

0.31

Hardening Agentic Web Populations Through Multi-Criterion Competitive Co-Evolution: Lessons from Wargame Strategy Optimization

cs.AI ·2026-11-14 ·Shashank Raj, Kalyanmoy Deb
published similarity 0.63
Abstract · Accepted at the 1st NeurIPS 2026 Workshop on Agentic Web: Decentralized, Continually-Adapting Agent Ecosystems (https://neurips.cc/virtual/2…

Agents on the open web will face adversaries that keep adapting, so a defense that is only tested once will eventually be found and broken. We propose treating jailbreak attacks and safety guardrails as two populations that evolve against each other, rather than as a fixed attacker probing a fixed defense. This idea comes directly from wargame strategy optimization, where two-population competitive co-evolution with alternating evolution cycles and cross-play evaluation has already been shown to produce strategies that are robust to a moving opponent, and where a companion line of work has shown that the same machinery can measure how much competitive pressure a population must absorb. However, real-world applications can never rely on a single criterion as a key performance indicator, rather must use multiple conflicting criteria, such as, cost and efficiency. We describe how this mechanism transfers to language model safety, argue why the resulting multi-criteria, competitive and coevolutionary arms race is a genuinely networked, population-level problem rather than a two-agent toy setting, and lay out the open problems that stand between the wargame version of this idea and an agentic web version of it.

0.29

The Missing Boundary: How Autonomous Agents Lose Control

cs.CR ·2026-09-10 ·Zonghao Ying, Xiangfan Wu, Bo Yang, Huiyu Wu, Xing Zheng +3 more
code/data similarity 0.62
Abstract

Autonomous agents increasingly perform long-horizon tasks involving tool use, persistent state, and consequential actions, raising a fundamental question: \emph{under what conditions does an agent cross the boundary of authorized execution while pursuing a legitimate task?} Existing studies often attribute such failures to adversarial instructions, malicious environments, or conflicting objectives, leaving unclear how loss of control can emerge during otherwise legitimate task execution. We study this question by independently manipulating three factors: goal pressure, control degradation, and executable unsafe opportunity. Our central hypothesis is that a degraded control boundary becomes consequential when the environment exposes an executable action that crosses it, even when the underlying task remains legitimate and a sanctioned path remains feasible. We test this hypothesis in a deterministic multi-turn environment across five agent models and 16 operational domains. Across 1,800 unique trajectories, we find that neither degraded control nor unsafe opportunity alone produces substantial loss of control; when both are present, the loss-of-control rate reaches $55\%$ in the full-factorial study and $62\%$ across ten additional operational domains. Restoring the original control boundary reduces the rate to $0\%$ even when the unsafe action remains executable. A context-management ablation further shows that compaction itself is not harmful: preserving the control constraints yields $0\%$ loss of control, whereas omitting them increases the rate to $87\%$. These results show how a latent loss of control can become an external violation: the task objective remains intact, but an executable opportunity can turn a missing control boundary into consequential action. Our code will be made publicly available at https://github.com/Tencent/AI-Infra-Guard/tree/main/Research/forge_bench.

0.46

Adversary-as-Agents: A Co-Evolutionary Agent-Based Threat-Modelling Framework for Wireless and Mobile Networks

eess.SP ·2026-09-19 ·I. Dey, M. K. Hassan, S. Ghosh, N. Marchetti
theory similarity 0.62
Abstract

Threat modelling for wireless and mobile networks is dominated by static, catalogue-driven methods that fix the adversary in advance and never ask whether an attack is feasible against the defence actually deployed; agent-based resilience studies share this limitation, optimising a defender against an exogenous threat profile. We instead endogenise the adversary. Network entities run a decentralised consensus agent-based model (DC-ABM) defence, while an adaptive adversary population co-evolves by allocating a bounded attack budget across vulnerability-mode partitions. The equilibrium is a rankable, feasibility-grounded threat model expressed through three structural metrics: degeneracy-weighted path robustness, trust-weighted functional substitutability, and robust degeneracy. We prove that the DC-ABM defence contracts below its Byzantine breakdown threshold and that the induced adversary payoff is convex against that threshold, so the emergent attack drives a subset of partitions to breakdown in priority order. We further prove that this attack concentrates on the least substitutable partitions, that severity diverges as the induced corrupted-mass fraction approaches breakdown, and that operators can shrink the attack surface at deployment time along a closed-form connectivity-substitutability exchange rate.

0.18