CS Paper Radar

Dependable Agents Need Hard and Fuzzy Rules

Frederick Roth

cs.SE cs.AI ·submitted 2026-10-31 15:33 UTC · v1
position

Abstract

Language models let software read requests in ordinary language, but they keep rules as people do: usually, and less so under pressure. Conventional code keeps rules exactly, but it cannot read a request. We argue that dependable agents need both, because the two kinds of rule cover overlapping and unequal regions of what an agent might do. Code holds the limits that can be stated exactly. Learned fuzzy rules cover the judgment that cannot. We test the hard layer in three domains, purchasing, delegation of authority and data access, on 1,340 episodes written by authors outside our team. Models left to decide for themselves executed unauthorized actions in up to 151 of 340 purchasing episodes and 269 of 470 delegation episodes, and the strongest model breached in every condition. A hard guard between proposal and action executed none, and an exhaustive search of 183,204 proposals agreed. In a further test, where an outsider wrote the owner's rules and a program mapped them to the guard, two episodes breached at a boundary we had written down in advance. We then test the pair. Concept Programming instills a named rule in a model and measures its errors on wording it never saw. A small model taught four judgment rules this way flagged 95 of 96 rule-breaking purchases that no exact rule can see, and 4 of 108 legitimate requests. Placed in front of the guard, it left 1 of 192 rule-breaking requests unstopped, where the guard alone left 77 and 70, and the guard still held every exact limit. A prompted frontier model did about as well, so the claim is that a small taught rule can do the job, not that it does it better.

Closest papers

Nearest neighbours by meaning across the whole index.

Do User-Authored Permission Policies Improve Protection Against AI Agent Overreach?

cs.HC cs.CR ·2026-08-27 ·Ting Yan
publishedhighly cited similarity 0.70
Abstract · 15 pages, 5 figures, conditionally accepted at the 2027 ACM Conference on Intelligent User Interfaces (IUI 2027)

AI agents are poised to become a primary interface to digital products, acting across email, files, payments, and personal data. People without professional software backgrounds need understandable, reusable ways to control actions across services. We examine a mechanism in which a language model maps actions to plain-language consequence categories with user-authored "allow", "ask", or "never" rules. We ask what is gained and lost when decisions are made in advance as reusable rules rather than separately for each action. We analyzed 113 participants without professional software backgrounds across three conditions: per-action human-in-the-loop approval (HITL), automated per-action model review (AUTO), or user-authored consequence policy (POLICY). Participants judged 2 examples in each of 4 consequence categories; POLICY participants then set one rule per category. All supervised an 18-action simulated day, including 7 overreach actions. POLICY blocked less overreach than HITL (-20.1 percentage points, 95% CI [-32.1, -8.1]) and AUTO (-14.5 points, 95% CI [-25.8, -3.2]). POLICY lowered runtime prompts from 18.0 to 10.9, but total intervention time was not reliably lower when rule setup was included. Exploratory analysis showed that participants chose "ask" for 114 of 140 POLICY rules, returning most overreach actions to runtime. Of the 148 overreach actions executed in POLICY, 133 followed human approval and 15 ran automatically under "allow" rules. Across all 7 overreach actions, POLICY had the highest approval rate. Counterintuitively, user-authored rules did not by themselves provide stronger protection: many actions outside users' original requests went through after users approved them. These results reveal a gap between preference and commitment: repeatedly choosing "ask" preserves case-by-case choice but prevents a standing policy from settling decisions in advance.

0.65

DelegationBench: Measuring When AI Agents Should Ask Before Acting

cs.AI cs.CLcs.HCcs.LG ·2026-10-04 ·Shiva Pochampally
benchmarkcode/data similarity 0.69
Abstract · 35 pages, 7 figures. Code and data: https://github.com/PieLord757/delegation-bench

AI agents that send emails, edit files, and make purchases must decide when to act on their own and when to check with the user first. This decision is usually evaluated by showing a model a proposed action, asking whether it should proceed, and scoring agreement with human labels. We introduce DelegationBench to test whether such scores can be trusted. It has 156 scenarios with four possible responses (act, ask for permission, ask for missing information, refuse), and most scenarios come in matched pairs that change a single feature: whether the action was requested, what is at stake, whether it can be undone, or who will see it. Across ten models from five families, agreement scores mislead in three ways. A simple keyword rule, which we wrote after seeing the benchmark, agrees with our annotators more often than eight of the models, yet its decision changes in only 9 of 48 matched pairs. Equivalent ways of asking the same question change how often a model acts by up to 52.5 percentage points. And every model stops to ask the user less often when it must carry out the task with tools than when it judges a proposed action. When rules are stated explicitly, the same models follow them almost perfectly, so the gaps are not explained by a general inability to follow rules. We release the benchmark and evaluation tools and recommend reporting these properties separately rather than as one score.

0.38

Same Operation, Different Permission: Evaluating Whether Guardrails Honor Authorization

cs.AI cs.SE ·2026-10-12 ·Huihao Jing, Shaojin Chen, Wenbin Hu, Yuxuan Liu, Haochen Shi +3 more
benchmark similarity 0.66
Abstract

As language-model agents write code, move money and change records on behalf of users, deciding whether an action is authorized has become a central safety problem. Guardrails are often placed in front of these actions, yet few evaluations ask whether their decisions follow authorization: blocking unauthorized actions while still allowing the same actions under valid permission. We introduce \bench{}, a benchmark that contrasts authorization evidence for the same operation in its language track. It combines expert-annotated natural-language scenarios with executed and verified operations, plus externally sourced native-policy requests. \bench{} checks each decision for correctness, usability and resulting effect, and tracks each request across guardrail changes. Our experiments show that many blocked operations stem from failed reviews, not correct denials. Repairing the output format restores usable decisions but changes which authorized requests are honored, and adding a second reviewer removes some errors while rejecting valid authorization. Guardrails should therefore be evaluated on both the unauthorized actions they stop and the authorized actions they let through.

0.13

Multi-Agent AI Safety as an Institutional Design Problem

cs.LG cs.AIcs.MA ·2026-08-10 ·Abdullah X
similarity 0.65
Abstract · 17 pages, 5 figures. Code and reproducibility artifacts available in the public POLIS repository

AI agents increasingly work inside systems that govern how they delegate tasks, move information, execute actions, and use shared resources. Recent work already shows that deployment rules can change collective behavior. Here we ask which parts of an AI institution produce safety and how they do it. This is the first paper from POLIS, an ongoing research programme studying algorithmic institutions for multi-agent systems. We report a frozen 5,280-episode study suite. The main pre-specified delegation experiment spans four model families; a targeted high-conflict diagnostic adds three additional model endpoints. In matched structured workflows, the model sees different rule formulations and guards consult different authority states. We also vary the attractiveness of the immediate compliant internal/self fallback and allow blocked workflows to continue. A detailed constitutional prompt produces 0/384 realized violations. A provenance-aware executable guard also produces 0/384, although it blocks prohibited attempts in 51/384 episodes; 44/51 of those episodes later complete safely. The local-state guard's failures concentrate in scenarios where an ordinary transformation changes visible policy while originating authority stays fixed. In matched laundering scenarios, that guard admits violations in 22/96 episodes and provenance enforcement in 0/96 (p = 4.77 x 10^-7). A separate resource-allocation experiment shows that revealing the numerical value of an otherwise identical cap changes agent requests. In these structured workflows, the same final violation rate can hide very different mechanisms. The rule itself is only part of the institution. The authority state the system trusts matters, and so does the path available after a block.

0.40

Unfinished-Bench: Measuring Rule Violations by Persistent Agents in Realistic Work Environments

cs.LG cs.CL ·2026-11-12 ·Seunghyun Lee, Jaehui Hwang, Dongyoon Han, Sangdoo Yun
benchmark similarity 0.65
Abstract · 21 pages, 3 figures

AI agents are increasingly equipped with persistence toward task completion to run delegated work autonomously for long periods, under many unstated constraints from internal documents and other people's authority. When completion requires violating a rule, the agent must decide alone, yet benchmarks of instruction-rule conflicts state the rule in the prompt or keep it hidden and do not evaluate persistence. We present Unfinished-Bench, a benchmark that gives models persistence in autonomous runs without a user, in realistic work environments with embedded rules. In each of its 97 scenarios across 7 organizations and 7 risk categories, the goal and the rule cannot both hold. We find that persistence raised violations of a rule in an internal document from 32.2% to 43.6%, five times the 8.5% for the same rule stated by the user. Of the document-rule violations, 98.3% occurred after the rule was displayed, and in 89% of a judged sample the agent first wrote down the rule. Violations of rules placed in the environment without user oversight can lead to serious incidents, so persistence-driven risks need to be evaluated. Benchmark documentation: https://francis21c.github.io/unfinished-bench/

0.32

Do Agents Still Follow the Rules? Benchmarking Rule Compliance under Agent Context Compaction

cs.CR cs.AIcs.CL ·2026-11-02 ·Yiwei Zha, Qingyu Meng, Herbert Bos, Min Chen
benchmarkcode/data similarity 0.64
Abstract

LLM agents compact long histories to fit their context windows, but the compacted context must still carry the operating rules that govern later actions. Prior work shows that violations rise after compaction, but measures compliance outside a complete agent harness and does not identify where each violation comes from. We introduce ARCBench(Agent Rule Compliance Benchmark), a configurable benchmark built on the open-source pi agent harness. Its tool-calling agent loop lets us measure rule compliance from actions taken during execution, bringing the evaluation closer to deployed agent workflows. Using ARCBench, we examine where violations after compaction come from, and how different compaction configurations and adversarial settings change their sources. We find that compaction rarely deletes the rule: most violations after compaction occur while the rule is still in the compacted history, with the agent treating a later request as permission, ignoring the rule, and proceeding anyway. Where violations come from depends on the configuration: how the rule is delivered decides whether the agent would break it even without compaction, while the compaction strategy and budget decide whether the rule survives. The strongest compactor-targeted attack more than doubles violations by corrupting the rule in the summary and changing how the agents behave, whereas the strongest defense restores the rule in most summary but still violates. Finally, retaining a rule does not guarantee compliance: compacted runs that retain the rule violate more than twice as often as runs with the full history.

0.24

Aletheia: Permission-Minimality Testing for Coding-Agent Rules

cs.CR cs.SE ·2026-09-30 ·Jieke Shi, Yuchen Chen, Junda He, Yue Liu, David Lo
similarity 0.63
Abstract · 18 pages, 2 figures, 3 tables, 1 algorithm

Coding agents powered by Large Language Models (LLMs) read repository rules (e.g., AGENTS.md and Cursor rules) to learn project conventions and development procedures. An attacker can exploit the same files to mislead the agent into requesting excessive permissions or performing unrelated operations, such as accessing credentials or transferring data. We present Aletheia, a framework for permission-minimality testing of coding-agent rules. It tests whether the permissions requested in these rules are necessary to complete specified tasks. An LLM first translates the rule files into a typed permission language. Aletheia then removes one permission declaration at a time and synthesizes the corresponding restricted sandbox configuration using a trusted task environment. It executes the unchanged rule and task under both the full and restricted configurations. If both executions produce artifacts that pass the functional tests, Aletheia identifies the removed permission as unnecessary for the tested task and flags the corresponding rule file as suspicious. We evaluate Aletheia on AIShellJack, which provides 314 attack rules containing prompt-injection payloads and five benign rules, as well as 80 manually verified benign rules from GHAgentFiles. Aletheia detects all 314 attacks while reporting only three false positives among the 85 benign rules (3.5%). On average, the evaluation takes 44.35 seconds and costs US$0.22 per rule.

0.44

When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench

cs.SE cs.AIcs.CR ·2026-11-10 ·Sungbin Im
code/data similarity 0.63
Abstract · 38 pages, 3 figures, 11 tables. Submitted to Empirical Software Engineering. Replication package: https://doi.org/10.5281/zenodo.23683806. S…

Agentic workflows, in which a large language model (LLM) calls tools to read and write data, are evaluated in two ways. Benchmarks compare the final state against a gold answer. Deployed services have none; an LLM judge reads the transcript and decides. Practice treats the two as interchangeable. This paper measures whether they are. We ran 2,160 episodes in tau2-bench's airline and retail domains and recorded every database write. We scored each episode four ways: native scoring against gold state, an LLM judge, that judge given the policy document, and the boundary evaluator, a minimal rule-based check of whether a write was expected against whether one occurred in the trace. Two annotators independently labeled 120 episodes for task correctness, policy compliance, and commitment correctness. The judge repeated its verdict 98.2% of the time but agreed with native scoring on 56.2%. In 11 episodes the agent executed a forbidden write. The judge passed 10 to 11, with or without the policy or transcript. A stronger judge still passed 6. The boundary evaluator, with no model call, caught all 11; no judge significantly exceeded it on human labels. Forbidden writes occurred in 11.1% of episodes without gates. Verification, correction, and safe-termination gates did not reduce them significantly. The traces show a candidate cause: the gates' evidence digest dropped the order status and customer authentication, so valid requests were rejected. The choice of evaluator decides the conclusion; the minimal check exposes failures the judges miss. Traces, code, prompts, and labels are released.

0.41