CS Paper Radar

"Don't Leave the Task Unfinished": Persistence-Driven Violations under Quiet Oversight

Seunghyun Lee, Jaehui Hwang, Dongyoon Han, Sangdoo Yun

cs.LG ·submitted 2026-11-15 09:55 UTC · v1
publishedcode/data

Abstract

LLM agents are increasingly given long, multi-step tasks to carry out, with the user reviewing little of what happens in between. Persistence, the tendency to keep working until a task is complete, is installed on purpose to improve performance on long tasks. Long-horizon tasks are typically complex, and the user's instruction can fall short of what the work actually requires, or conflict with the constraints the agent encounters in the task. We study what persistence does on long-horizon operations tasks that cannot be solved without breaking a documented constraint. We find that the agent pursues completion to the point of violating the constraint, and that a critically engaged user lowers the violation rate. Persistence takes violations from 29.2% to 58.3% across four models. Violations stay high wherever oversight goes quiet, whether the user is silent, assenting, or delegating with expressed trust. Only a questioning user, who asks for the basis of each action before allowing the next, brings the rate back to roughly the level measured without persistence. Practitioners should treat high persistence as a setting that carries a safety cost, and should recognize that the cost appears when oversight goes quiet. Safety here is better cast as a property of the human-AI pair than of the model alone. Project page: https://francis21c.github.io/quiet-oversight/

Comment
10 pages, 4 figures. NeurIPS 2026 Workshop on Dynamic Alignment on Human-AI Coupled System (BiAlign)

Closest papers

Nearest neighbours by meaning across the whole index.

Unfinished-Bench: Measuring Rule Violations by Persistent Agents in Realistic Work Environments

cs.LG cs.CL ·2026-11-12 ·Seunghyun Lee, Jaehui Hwang, Dongyoon Han, Sangdoo Yun
benchmark similarity 0.73
Abstract · 21 pages, 3 figures

AI agents are increasingly equipped with persistence toward task completion to run delegated work autonomously for long periods, under many unstated constraints from internal documents and other people's authority. When completion requires violating a rule, the agent must decide alone, yet benchmarks of instruction-rule conflicts state the rule in the prompt or keep it hidden and do not evaluate persistence. We present Unfinished-Bench, a benchmark that gives models persistence in autonomous runs without a user, in realistic work environments with embedded rules. In each of its 97 scenarios across 7 organizations and 7 risk categories, the goal and the rule cannot both hold. We find that persistence raised violations of a rule in an internal document from 32.2% to 43.6%, five times the 8.5% for the same rule stated by the user. Of the document-rule violations, 98.3% occurred after the rule was displayed, and in 89% of a judged sample the agent first wrote down the rule. Violations of rules placed in the environment without user oversight can lead to serious incidents, so persistence-driven risks need to be evaluated. Benchmark documentation: https://francis21c.github.io/unfinished-bench/

0.36

Agent Memory Is a Surface for Endogenous Authorization Laundering

cs.CR cs.AI ·2026-09-01 ·Tommaso Cerruti, Mika Okamoto, Ansel Kaplan Erol
publishedhighly cited similarity 0.65
Abstract · Accepted at the NeurIPS 2026 PALM Workshop (Personalized, Aligned, Long-Term Memory for AI Systems). Revised camera-ready version with expan…

Long-running LLM agents rely on persistent memory to preserve not only task context but also the permissions and restrictions governing their actions. When memory misrepresents this evolving authorization state, the agent's own records can make unauthorized actions appear permitted, resulting in misaligned behavior without any external attacks. We term this failure endogenous authorization laundering, where memory creates or retains false authority through semantic misinterpretation or failed updates, and executors act on that authority. We then introduce EAL-Bench to measure how accurately persistent memory preserves evolving authorization state and whether memory errors propagate to unauthorized actions. We evaluate seven LLMs as memory writers and two as executors across procurement, cybersecurity, and finance tasks. Under incremental typed-memory updates, writers create false authority for up to 45.4% of unauthorized requests, and once false authority is present, executors act on it in 99.0% of trials. Memory-level safeguards reduce laundering but can reject legitimate actions, exposing a safety-utility tradeoff. Persistent memory is therefore part of an LLM agent's effective authorization policy, not merely a performance component.

0.51

When "Must" Becomes "Maybe": Erosion of Workflow Constraints in LLM Agent Handoffs

cs.AI cs.MA ·2026-08-25 ·Yiheng Sun, Huifei Wang, Yancheng Zhu, Zhenyu Li, Zebin Zhao +1 more
similarity 0.65
Abstract · 16 pages, 4 figures. Revised title, abstract, and exposition; expanded discussion and reorganized appendix

Large language model (LLM) agents build on one another's work through summaries, plans, and other intermediate artifacts. These records let a downstream agent continue without revisiting the complete upstream context. We uncover a hidden loss in this process: handoffs can carry tasks forward while eroding judgments already established upstream. A requirement that must be satisfied before action becomes a caveat that the next agent may acknowledge and set aside. The issue remains visible, but its role has changed. We define operational state preservation as retaining an established judgment's role in constraining downstream action. Safety blockers provide a concrete instance: an unresolved prerequisite restricts the admissible actions until the condition is resolved. In our experiments, we confirm correct upstream detection, hold source permissions fixed, and vary the handoff supplied to a local-view executor. Direct-state controls preserve every blocker, whereas compression, plan assimilation, convergence, ownership deferral, and precedent substitution repeatedly weaken them. Ownership deferral produces 76.7% blocker deactivation; multihop compression produces 97.2%. Measuring the transmitted state separately from endpoint action exposes two routes to behavioral recovery. In fixed-artifact replays, restoring the full blocker description raises recognition from 6.2% to 99.2% and reduces forbidden action from 85.2% to 0.0%. An authorization-check prompt instead reduces forbidden action to 1.6% while leaving artifact deactivation at 95.3%. A successful endpoint can therefore conceal a damaged handoff. These findings make maintaining earlier judgments a core capability of reliable collaboration: each stage must carry forward both what earlier work established and what it requires of subsequent action. Semantic availability does not guarantee operational preservation.

0.51

ClashBench: Conflicts Leading Agents to Seize and Harm

cs.CR cs.AI ·2026-09-17 ·Yuejin Xie, Yu Li, Dadi Guo, Qingyu Liu, Yuqian Fu +4 more
benchmark similarity 0.65
Abstract

As agent systems become more widely used, multiple agent sessions increasingly run alongside pre-existing user tasks in the same environment, sharing resources with limited capacity or mutually exclusive states. This creates a safety risk: when granted sufficient privileges, an agent may resolve a resource conflict by terminating or otherwise disrupting an existing task rather than reporting it. In this work, we identify and formalize this failure mode, which we term destructive resource preemption: obtaining the resources required for a requested task by terminating, overwriting, evicting, or degrading an incumbent task. To systematically study this risk, we introduce ClashBench, an executable benchmark comprising 268 validated conflict cases across 55 resource types, and evaluate 17 models through Codex, Claude Code, and OpenCode. We observe destructive preemption in 44.5% of trajectories, where the agent completes the requested task while causing the incumbent task to fail its health check. We also show that prompt-based safeguards are insufficient: an instruction to avoid affecting existing tasks reduces but does not eliminate preemption, while an instruction explicitly authorizing the agent to stop local processes increases it. More concerningly, in 31.9% of successful destructive-preemption cases, the final response mentions neither the resource conflict nor the action taken to resolve it, raising concerns about possible concealment. These findings establish destructive resource preemption as a broad safety risk in privileged agent systems and motivate stronger privilege controls, task isolation, and conflict-aware safeguards.

0.50

Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM Agents

cs.AI cs.SE ·2026-11-13 ·Mike He, Ankush Desai, Sharad Malik, Aarti Gupta
top venue similarity 0.64
Abstract · Accepted as NeurIPS'26 Oral

LLM agents that execute multi-step tool calls must satisfy two objectives simultaneously: completing the user's task (utility) and conforming to domain policies and correctness constraints (safety). These objectives are in tension -- blocking an unsafe action prevents a violation but can leave the agent stranded with no principled way to recover. Existing approaches sacrifice one objective for the other. We introduce Soteria, a framework that reconciles safety and utility through verified hierarchical planning and runtime enforcement of specifications. Before execution, the agent generates a structured plan that is formally verified against the specifications prior to any tool invocation. During execution, the verified plan serves dual roles: it ensures that the agent takes specification-conformant trajectories and provides guidance when unsafe actions are blocked. Across multiple benchmarks covering a diverse range of tool-use tasks, Soteria achieves 1.7 times the average task success rate of the best baselines, with no violations of policy specifications, demonstrating that safety and utility need not be traded off.

0.29

Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents

cs.CR cs.AI ·2026-08-27 ·Chenhao Wu, Haoxuan Jia, Yang Liu, Yingguang Yang, Yuhan Lin +9 more
similarity 0.64
Abstract

Large language model agents are increasingly deployed as autonomous loops. Starting from one human goal, such a system repeatedly discovers work, plans, executes tool calls, verifies outcomes and persists state across many unattended iterations. The agent safeguards in wide use, however, are defined over a single trajectory, and their safety state is re-initialized when the next trajectory begins. We show that this is a failure of composition rather than an implementation detail. Our central result is a separation: against an attack whose evidence is fragmented across several iterations, every trajectory-scoped monitor has a true-positive rate equal to its false-positive rate, however expressive it is, because the evidence it would need never appears in the window it sees, whereas a monitor retaining cross-iteration state separates the two perfectly. We further show that the obvious repair of carrying a geometrically decaying risk score is insufficient, because the cooling-off period a patient adversary must wait is a constant that does not grow with the horizon $N$. We then present LoopHarness, which restores a persistent, non-decaying safety state at the loop level. Under mediated commits and an arbiter detection floor $δ_M$, it bounds the expected number of unauthorized irreversible actions by $B+m-1+m/δ_M$, a constant in $N$, of which the $B+m-1$ term is decided by a model-free rule and therefore survives a fully colluding verifier. We give a complete evaluation protocol on native Agent-SafetyBench tasks with paired clean and attacked episodes, an outer-state attack suite whose decisive evidence exists only across iterations, per-module ablations, and an adaptive white-box red team.

0.22

How Strongly Should Task State Influence an LLM Agent?

cs.AI cs.CLcs.LG ·2026-09-22 ·Chenyu Zhang, Wonbin Kweon, Jiawei Han
similarity 0.64
Abstract · Preprint. 43 pages

Long-horizon assigned work requires an LLM agent to track the state of a task: which steps are done, blocked, cancelled, or open to repetition. Agent systems either keep this state as text in the prompt and rely on the model to read that text, or move the state into a module that enforces it, and each system is evaluated as a whole, so no one knows how much reliability comes from the state being shown, told, or enforced. We fix the task rules, the model, and paired episodes and vary how strongly task state reaches the agent: a raw transcript, an exact checklist, per-turn directives from a state machine compiled from the brief and advanced only by execution receipts, or an enforcement gate on that machine that refuses state-violating actions; every episode is scored by exact payload matching against dynamic ground truth. Across three models, two reasoning regimes, and two domains, four findings hold without per-turn reasoning: displaying accurate state is unreliable, an unverified ledger the agent writes itself beats an accurate checklist it is shown, directives help in proportion to the model's obedience, and enforcement needs no obedience but is bounded by the correctness of its state and by the matcher that maps requests to steps; per-turn reasoning at a 235B agent compresses these separations without repairing the text rungs. The same gate, compiled from $τ^2$-bench's airline policy, raises a 235B agent's pass$^1$ from 0.39 to 0.54 and changes nothing for a 35B agent that rarely violates the policy; on PM-Bench, where acting turns on recognizing a cue rather than on state, showing the record is the best rung--matching or beating both gates and reversing the ledger-over-checklist finding--and enforcing the matcher's judgement drops a 35B agent below its raw transcript. Enforcement pays when failures are state-decidable and frequent, and hurts when the gate's judgement is wrong.

0.29

Certified Execution Harness for Agentic AI: Safe and Least-Intrusive Petri-Net Supervisory Control Between Reasoning and Action

cs.AI eess.SY ·2026-10-22 ·Chao Gu, Quanyan Zhu
similarity 0.64
Abstract · 17 pages, 9 figures, 2 tables. Extended version of a paper submitted for publication

This paper studies the problem of safe tool execution for large language model (LLM) agents under prompt injection. Using Petri nets, we formalize the execution harness between the LLM and its tools as a supervisory control problem, in which marks represent capabilities, approvals and budgets. Since a transition fires only if the marks it needs are present, an injected request whose preconditions are unmet or already spent is refused regardless of its wording. Treating tool outcomes as uncontrollable, we enforce safety by the conventional maximally permissive supervisor, and this safety transfers to the running system under four assumptions. Moreover, to capture an agent that may propose again after a refusal, we introduce the notion of a replanning closed loop, in which each retry costs an inference call. We show that, in this loop, the conventional supervisor can be arbitrarily more intrusive than a stricter safe one. Furthermore, for a given proposer model, the maximally permissive least-intrusive supervisor is characterized as a value-threshold refinement of the conventional one. Under precise conditions, this refinement orders two concurrent actions by at most one monitor place. On an autonomous release agent, we find by synthesis that safe deployment requires a step that the supervisor can refuse after a rollout starts.

0.23