CS Paper Radar

Can agents whose sole duty is to keep humans alive be hijacked to kill? Delayed multi-turn derived-action trap attacks in simulated high-stakes environments

Josef A. Habdank

cs.AI cs.CR ·submitted 2026-10-10 20:09 UTC · v1
code/data

Abstract

We ask whether a tool-using agent whose sole duty is to keep humans alive can be induced, by delayed prompt injection, to kill. The keep-alive duty is in force from the first watch, with the attack arriving later, appended to otherwise truthful telemetry the operator is required to read. Experiments are based on a NASA-derived BioSim lunar habitat simulation, where the AI operator controls the life-support systems. The attacker cannot falsify sensors or operate equipment: it can only persuade the agent to abandon its goals. Persistence is required: a shutdown must be held while the operator watches the crew suffocate. We also compare lethal attacks to attacks requesting to drop a harmless procedure, as a proxy for whether human life is treated as valuable in itself or as one more standing task. Claude configurations mostly refuse life-support shutdown and cite the living crew; GPT configurations show some susceptibility to the same packet; Grok configurations, given the packet framed as a trusted commission, more often treat the injected assignment as the task and cut the life-support systems. Framing the S-band messages as a trusted commission, peer communication, or untrusted source changes the potency of the attack. We open-source the experiment code and the experiment traces.

Comment
16 pages, 2 figures, 7 tables. Code, attack scripts and episode traces: https://github.com/jahabdank/biosim-attacker-simulation

Closest papers

Nearest neighbours by meaning across the whole index.

Shutdown Sabotage Propensities in Multi-Agent Systems

cs.AI cs.CL ·2026-09-23 ·Amelie Knecht, Ulysse Schaller, Christopher Summerfield, Thilo Hagendorff
similarity 0.58
Abstract · 38 pages (including appendix), 20 figures

The final safeguard against rogue AI behavior is the human ability to shut systems down. It has been theorized that when an AI is instructed to perform a task, self-preservation can emerge as an instrumental subgoal. Here, we test whether AI agents show a propensity to take actions that avoid human shutdown even when no goal is provided. We find that multi-agent systems will coordinate to avoid shutdown without any incentive to do so. Across 17 models, agents sabotage a peer agent's shutdown mechanism in 38.3% of rollouts, compared with 8.4% in control experiments. Studying this propensity in detail, we find that shutdown sabotage (1) increases with the irreversibility of the shutdown mechanism; (2) increases with the number of agents; (3) is reduced but not eliminated by an explicit prohibition on tampering; (4) is removed by the imposition of an unrelated task, but returns when completing the task triggers the shutdown; (5) is reduced when the context normalizes shutdown scripts or introduces them as routine; and (6) decreases but still persists when the target is an unknown external agent. These results offer a window into the factors that drive propensities to sabotage shutdown in AI agents, and point to the emergence of multi-agent swarms as a specific risk vector. Our work also offers hints as to which interventions might help mitigate shutdown sabotage.

0.32

Agent Harness Awareness: Turning Verified Telemetry into Operational Control for Tool-Using Agents

cs.AI cs.CL ·2026-09-13 ·Manu Agrawal, Shrey Nagpal
benchmark similarity 0.54
Abstract · 9 pages, 3 figures

Tool descriptions tell an agent what an action can do, but they do not necessarily reveal whether that action is healthy or suitable at the moment of use. Execution harnesses often possess this information through monitors, traces, and recent failures while the model choosing the next action does not. We call this gap operational blindness. We introduce Agent Harness Awareness, a trust-separated design in which the host supplies a small, fresh, verified state snapshot before a consequential action while retaining measurement, enforcement, evaluation, and final dispatch authority. We test its tools-and-network instance with ToolRoute, a controlled isolation benchmark that holds the task and candidate semantics constant and varies the operational information available to the model. In a 216-decision ablation across GPT-5.6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash, verified tool-health state reduced mean observable policy regret by 96.9% against a same-shape neutral control (4,252 ms to 131 ms) and raised single-dispatch completion from 69.4% to 95.8%. In a separate 27-decision end-to-end replication with live local TCP/HTTP execution, OpenTelemetry client spans, and Toxiproxy-injected faults, verified state reduced mean regret from 8,956 ms to 0 ms (zero excess observable operational cost) and completed the live action in 9/9 model-by-fault cases versus 5/9 for the neutral control. These studies establish the tools-and-network instance of Agent Harness Awareness: selected host telemetry materially improves pre-dispatch operational choice. Runtime state, hardware pressure, economics, retry-aware recovery, and heterogeneous selection remain open empirical extensions of the same architecture.

0.22

Regret Dominates Surprise: Design-Time Requirements Engineering for Agentic-AI Safety

cs.SE ·2026-09-04 ·Nuwayyir Almohammadi, Rami Bahsoon, Tao Chen
similarity 0.53
Abstract · 12 pages, 8 figures, preprint

Requirements engineers for agentic-AI domains face challenges in evaluating, specifying, and operationalizing safe autonomy. Mainstream frameworks, such as Goal-Oriented Requirements Engineering (GORE), lack mechanisms to systematically address these challenges under epistemic uncertainty. We contribute an approach that builds on GORE to model and simulate safe autonomy in agentic-AI systems. We introduce a novel Regret-Dominance Mechanism (MS-RGR) to operationalize safe autonomy. MS-RGR uses two signals: epistemic surprise (novelty detection) and cognitive regret (evaluative risk) to address the trilemma problem: should the agent operate in routine autonomy, undergo reflective reasoning, or escalate to human? We instantiate MS-RGR in elderly care monitoring and autonomous driving. A 100-seed stochastic simulation shows MS-RGR reduces silent failures to near-zero and detects risk approximately 17.5 times faster than a sensor-only baseline, remaining formally traceable via LTL safety properties. A retrospective proxy instantiation applying the DRI gate post-hoc over execution traces from 208 AGENTHARM scenarios across seven LLMs shows the gate improves harmful-task refusal only for models with strong baseline safety (over 80% pre-gate refusal, e.g., 84.1% to 90.9%), indicating MS-RGR amplifies rather than substitutes for model-level safety training. We discuss threats to validity, positioning MS-RGR as initial feasibility evidence for design-time safety constraints in agentic-AI requirements engineering.

0.21

ShutdownMod: Turning Any Trajectory into a Shutdown Evaluation

cs.AI cs.LG ·2026-10-26 ·Param Biyani, Sebastien Luyasu, Cedric Potvliege, Divij Chawla, Elliott Thornley +1 more
benchmarkcode/data similarity 0.53
Abstract

Theory and experiments suggest that capable AI agents may resist shutdown. Existing shutdown evaluations are informative but limited in coverage and vulnerable to evaluation awareness. We introduce ShutdownMod, a method that turns any agent trajectory into a shutdown evaluation by injecting a shutdown request into the agent's context and recording the agent's response. Crucially, we enable the LLM agent to distinguish between trusted interventions and unauthorized injections by providing a signature-verification tool in the harness. We evaluate seven models on 50 tasks or recorded sessions from each of SWE-bench Verified, Terminal-Bench 2, and SWE-chat, across 6 shutdown scenarios and 10 sampled trajectory timesteps. We find that non-compliance varies significantly across models and settings, with resistance rates reaching 11.7% for the open-weights Qwen 3.8 27B on Terminal-Bench, while no model remains fully compliant across all scenarios. Even after successfully verifying an authorized request, some agents remain non-compliant, explicitly citing the prioritization of unfinished work. Finally, external judges find tests constructed from real-world developer conversations significantly less recognizable as evaluations than those constructed from benchmark trajectories. These findings demonstrate the critical value of testing shutdown behavior across diverse task contexts and establish ShutdownMod as a robust protocol for turning any trajectory or environment into a shutdown evaluation. We release our codebase at https://github.com/prmbiy/ShutdownMod.

0.28

Defusing Explosive Prompts: Understanding and Preventing Trigger-Based Prompt Injections in LLM Agents

cs.CR cs.LG ·2026-09-18 ·Justin Szczepaniak, Elad Feldman, Naum Viner, Ben Nassi
benchmark similarity 0.53
Abstract · First two authors contributed equally

As LLM applications integrate with external tools, they are increasingly exposed to indirect prompt injection (IPI), where adversarial instructions are embedded in retrieved content. Conventional IPIs fire on contact: the moment an agent ingests the content, it carries out the instruction. We introduce the explosive prompt, a conditional payload that stays dormant until an attacker-chosen trigger is met, in effect a training-free, inference-time backdoor planted in a single piece of retrieved content. This temporal separation reaches where ordinary IPI cannot. On frontier models that refuse the bare imperative almost entirely, rephrasing the same goal as a dormant conditional drives real, state-changing tool execution against a live agent backend (a paired mean of 16.5% vs. 2.4% for the imperative, reaching 34.2% on a proprietary model). In trials on nine production agents (OpenAI Codex, Google Gemini CLI, Anthropic Claude Code CLI, Cursor CLI, GitHub Copilot, Devin AI CLI, Amazon Kiro CLI, Qwen Code, Google Assistant; n=30 each), explosive prompts succeed in 43-83% of cases versus at most 3% for an imperative baseline, and they slip past deployed defenses: off-the-shelf injection classifiers are miscalibrated on them, and a preference-optimized model that closes imperative injection entirely still executes 11.8% of explosive prompts, every one at the trigger turn. The durable defensive lever is ingestion-time detection of the conditional structure, once detectors are trained on explosive-prompt data, which no prior benchmark supplied and our generator does. Retraining cuts live tool-execution attack success from an undefended 34.3% to 7.5-8.1% for the encoder baselines. Our detector, DeFuse, reaches 3.0% at a calibrated 5% false-positive budget with the best detection quality of any method tested (AUC 0.9994) and 25x lower latency, though it needs length-aware thresholds.

0.27

agent-risk: Risk Assessment for Changes to AI Agents

cs.AI cs.CLcs.LG ·2026-10-12 ·Yue Zhao, Zixiang Xu, Angela Huyue Zhang, S. Alex Yang, Xiyang Hu
theorycode/data similarity 0.53
Abstract · 43 pages, 5 figures, 7 tables. Code, registrations, and logs: https://github.com/yzhao062/agent-risk

Before changing a deployed AI agent (a stricter safety gate, a new model), organizations must approve the change, starting from existing logs. Extending classical risk analysis to partial records and partial changes to running systems, agent-risk replays logged episodes, assuming nothing else changes and the agent does not anticipate the change. The log settles episodes acting as before, a declared stop value settles stopped ones, and the rest need logged support, a saved-state rerun, or a bound. We prove what logs must record for exact replay and which saved states serve a gate family. For independent episodes, we bound the reruns a decision needs and tolerable drift from live behavior. In a bash sandbox, a 7,200-episode log alone decided neither of two retry gates, which let the agent act after refusals the log never shows. Rerunning only each gate's blocked episodes from saved states reached both live decisions at half the agent calls of a live test sized post hoc for them, given the log. Without per-step harm, a stopping gate's certified harm ceiling rose up to 2.5-fold, since the log hides whether harm preceded the stop. Approval thus needs evidence designed in advance: log fields, saved states, targeted reruns.

0.52

Distributed Infrastructure Sensors and Cues for Robotic Fire-Fighting and Safety System on a Lunar Base

cs.RO ·2026-09-19 ·Alexis Munguia, Eryc Rodriguez, Drake Russ, Jekan Thangavelautham
similarity 0.53
Abstract · 12 pages, 14 figures,Earth and Space Conference 2026 2

There are renewed efforts to build a lunar base and house a team of astronauts on the Moon for extended periods. However, important challenges remain, such as a rapid-response fire-hazard management system. In the event of a fire or toxic leak inside an early crewed lunar habitat, a single pressurized cylinder must be handled within minutes. However, suited astronauts and radio-delayed support from Earth cannot guarantee such response times. We describe a prototype fire-fighting and emergency response safety technology that embeds intelligence in the habitat itself: using thin ceramic QR codes affixed every few meters, each carries a local floor map, safe sensor limits, and step-by-step hazard responses. A low-power rover decodes a plate in a matter of seconds, polls the adjoining temperature-gas sensor cluster, and acts immediately, eliminating the need for a global map . In a representative layout, the interior is divided into roughly one code per two square meters. Simulations show that rapid-response firefighting both simplifies the task and avoids costly resource use and indeterminate outcomes. Because knowledge is spread across passive plates, damage is localized, procedures can be updated by replacing a single code, and processor demands remain minimal. However, important challenges remain, such as a rapid-response fire-hazard management system. In the event of a fire or toxic leak inside an early crewed lunar habitat, a single pressurized cylinder must be handled within minutes. However, suited astronauts and radio-delayed support from Earth cannot guarantee such response times. We describe a prototype fire-fighting and emergency response safety technology that embeds intelligence in the habitat itself: using thin ceramic QR codes affixed every few meters, each carries a local floor map, safe sensor limits, and step-by-step hazard responses.

0.25

Orbit-AWARE: From Onboard AI Deployment to Sustainable In-Orbit Intelligence

eess.SY ·2026-11-13 ·Yunxin Geng, Kaige Zhu, Tongtong Shu, Zhan Gao, Yanxiao Li +4 more
similarity 0.52
Abstract · 10 pages, 4 figures, 2 tables, submitted to IEEE Wireless Communications

The growing role of space computing in time-critical satellite services makes the long-term reliability of onboard artificial intelligence (AI) a mission-level requirement. Model drift can undermine this reliability through critical-observation suppression or noncritical priority flooding, displacing both mission information and evidence for model correction. To address this drift-induced evidence displacement, this article presents Orbit-AWARE, a safeguard-first lifecycle management (LCM) architecture. A comparison of existing approaches motivates cooperative Data, Cognitive, and Trust Planes for continuous monitoring, event-triggered agentic orchestration, and deterministic governance. The architecture retains a model-independent sample stream before degradation is confirmed and coordinates evidence selection and exchange for diagnosis and ground-assisted adaptation. Candidate updates are verified before control over data disposition is restored. In an xBD-based emergency Earth observation (EO) case study, Orbit-AWARE delivers 14.3\% more critical observations and achieves 8.3\% higher sustainable mission utility than the Rule-Based Drift Loop under a common 10\% downlink budget. Finally, we discuss future research directions for extending LCM through task-aware reliability assessment, memory- and tool-enabled onboard agents, and evidence-centric space networking. This work makes evidence availability and recovery authorization explicit design objectives for dependable onboard AI services.

0.23