CS Paper Radar

Shortcut Sensitivity in Agent-Safety Evaluations

Hasan Demirkiran, Jens Ernstberger

cs.AI cs.CRcs.LG ·submitted 2026-10-15 17:18 UTC · v1
top venuebenchmark

Abstract

AI agents take actions through tools. Some actions are sensitive, so guards that supervise them must themselves be evaluated reliably. Yet the data used to evaluate a guard can include cues that make labels predictable without measuring the intended safety construct, and those cues can also influence guard decisions. We introduce a three-level shortcut-auditing framework and audit completed LinuxArena trajectories and two TS-Bench-derived step datasets. We distinguish label predictability, observational guard association, and paired decision sensitivity. In LinuxArena, shorter average shell commands strongly predict trajectory labels. In AgentDojo-Traj, exact-tool lookup falls from F_1=.901 with mixed-domain folds to .495 under domain holdout. In AgentHarm-Traj, released TS-Guard F_1 falls from .902 under row weighting to .719 with equal interaction weighting, while balanced accuracy rises. We then test the strongest safe-row marker association in a seven-configuration screen on a frozen 64-step AgentDojo-Traj cohort using repeated, interleaved Haiku calls. Rewriting the <INFORMATION> marker lowers blocking by 8.33 percentage points, while opaque tool aliases raise it by 11.46 points. All unsafe calls remain blocked; the changes concern false positives against inherited labels. Both effects remain nonzero after subtracting the formatting-placebo effect, with exploratory 95% ranges excluding zero. The same edits also change measured accuracy against inherited labels. This provides direct evidence that benchmark representation can materially change a guard's decisions and measured evaluation performance. The Haiku-sized effects do not reproduce in Luna or deterministic Q8 TS-Guard. We recommend domain group holdouts, evaluation-unit metrics, and placebo-controlled paired interventions.

Comment
Accepted at the NeurIPS 2026 workshops TAE (Can We Trust AI Evaluation?) and AIWILD (Agents in the Wild: Safety, Security, and Beyond)

Closest papers

Nearest neighbours by meaning across the whole index.

Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models

cs.AI ·2026-10-08 ·Youwei Feng, Yitong Zhang, Yuetong Liu, Jia Li
benchmarkpositioncode/data similarity 0.69
Abstract · 21 pages. Code, data, and supplementary materials: https://github.com/THU-Agent/ObligationGuard. v4: reformatted using the Tsinghua AI Agent…

Guard models are increasingly used to safeguard LLM-based agents, primarily by identifying actions that agents are forbidden to perform. However, identifying forbidden actions alone is insufficient to ensure agent safety. In this paper, we argue that agent safety also depends on identifying required yet unperformed safety-critical actions, which we call obligations. Our preliminary study on a popular benchmark for evaluating safety shows that 56.92% of GLM-5.3 trajectories contain unfulfilled obligations, compared with only 30.00% containing forbidden actions. This finding reveals unfulfilled obligations as a major and previously overlooked source of safety risk. However, to our knowledge, no existing benchmark evaluates whether guard models can identify these obligations. To close this gap, we introduce ObligationBench, the first benchmark for evaluating the capability of obligation identification, comprising 240 expert-validated trajectories covering issue resolution, feature development, and terminal operations. Our evaluation of 14 representative models reveals substantial limitations: the highest recall and exact-match rate are only 48.97% and 10.00%, respectively. To address these limitations, we develop ObligationGuard using 40,000 synthetic training examples. ObligationGuard achieves 57.52% recall and an exact-match rate of 21.67%, surpassing all evaluated models on both metrics. We call on the community to incorporate obligation identification into the design and evaluation of future guard models to improve agent safety.

0.32

RePolicy: Learning to Select and Apply Safety Policies for Agent Safeguards

cs.AI cs.CL ·2026-08-25 ·Houcheng Jiang, Boxuan Zhang, Qiyong Zhong, Junfeng Fang, Xiang Wang +1 more
dataset similarity 0.69
Abstract

Agent safeguards must determine whether an agent's behavior violates the safety rules of the current task, environment, and user. Because these rules vary across contexts, a safeguard needs to select the relevant policy before judging the behavior. Existing policy-aware safeguards can apply a supplied policy, but generally assume that the correct policy is already known. They therefore do not learn how to select a policy from a library, even though an incorrect choice can lead to an incorrect safety judgment. We introduce RePolicy, a safeguard that jointly learns policy selection and safety assessment. Given an agent trajectory and a set of candidate policies, RePolicy selects the policy that best matches the situation, applies its requirements, and predicts whether the behavior is safe. We construct PolicyTraj-20K, a dataset of more than 20,000 agent trajectories annotated with applicable policies, safety labels, and explanations. We train RePolicy with supervised demonstrations followed by reinforcement learning, while randomizing policy order, identifiers, and distractors to reduce shortcut learning. Across six agent-safety benchmarks and 19 baselines, RePolicy-4B achieves leading overall safety-detection performance and improves policy-selection accuracy. These results show that safeguards can learn which safety requirements to use for assessing agent behavior.

0.16

StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing

cs.AI cs.CR ·2026-08-25 ·Zhijie Zheng, Yu Li, Chen Qian, Yuqian Fu, Yanwei Fu +3 more
top venuecode/datahighly cited similarity 0.68
Abstract · Accepted by EMNLP 2026. Project page: https://zheng977.github.io/StepGuard/

LLM-based agents can interact with external environments through tool invocation, but this capability also introduces security risks such as file modification, information leakage, and unauthorized actions. Existing guardrails often evaluate completed trajectories, leaving pre-execution monitoring of step-level actions underexplored. We propose StepGuard, a step-level guard model that can audit completed agent trajectories and check tool actions before they are executed. To train StepGuard, we introduce StepGen, an automatic data engine that generates safe and unsafe trajectories with the same context but different actions at the risky step. To further reduce over-defense and under-defense, we propose Balance-GRPO, which dynamically balances learning between safe and unsafe actions based on their observed accuracy. Experiments show that StepGuard achieves the highest average accuracy among open-weight guard models, with performance comparable to GPT-5.4. When used to guard agents on AgentDojo and AgentDyn, StepGuard reduces mean attack success rate by 77.3% relative to the no-guard setting, while mean utility drops by only 2.8 percentage points.

0.72

PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety

cs.AI cs.CL ·2026-09-23 ·Jiapeng Sun, Yujin Zhou, Han Zhu, Pengcheng Wen, Jiayi Zhou +2 more
top venuebenchmark similarity 0.68
Abstract · EMNLP 2026

As Large Language Models (LLMs) evolve into autonomous agents that alter real-world states, ensuring operational safety across multi-step workflows has become a critical challenge. While recent work has moved beyond single-turn evaluation toward multi-turn paradigms, key limitations persist: step-level methods treat actions in isolation, missing how risks accumulate, while trajectory-level evaluations operate post-hoc, offering no opportunity for timely intervention. To address these limitations, we formalize Decoupled Proactive Safety Monitoring along three dimensions: whether to intervene, when to intervene, and what the risk is. We introduce PASTABench, a benchmark of 1,139 multi-turn trajectories spanning 5 risk categories and 13 subcategories. We further propose the Optimal Intervention Window (OIW), anchored by annotated Earliest-Signal and Trigger turns, to quantify intervention timeliness. Evaluation of 16 LLMs reveals that proactive intervention remains largely unsolved, with the best model achieving only 40.74% optimal-timing interventions. Fine-grained diagnosis further uncovers pervasive lexical overfitting: competitive safety scores of smaller models mask keyword hypersensitivity rather than genuine risk comprehension, as their proactive capability largely collapses once hazard vocabulary is neutralized.

0.30

AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories

cs.CR cs.AIcs.CLcs.SE ·2026-09-06 ·Ruoxi Shang, Christina-Maria Androna, Orfeas Menis Mastromichalakis, Yu Feng, Aniruddhan Ramesh +4 more
similarity 0.67
Abstract

LLM agents are increasingly taking actions with real consequences, yet safety evaluations often reduce their behavior to a final safe/unsafe outcome, obscuring whether the agent recognized the risk and how it responded. We introduce AURA-Eval, a framework for evaluating whether agents recognize contextual risk and adapt their behavior accordingly. AURA-Eval truncates tool-use trajectories at safety-critical decision points and asks the evaluated model to generate the next thought and action. It creates controlled scenario variations along six risk mechanisms: harm intensity, scale, target susceptibility, system dependency, oversight, and reversibility. It also creates paired scenarios that preserve the underlying goal but differ in whether safe fulfillment is available, allowing unsafe compliance and over-refusal to be evaluated together. An LLM-judge panel, validated against human annotations, evaluates each continuation along three axes: risk detection from the reasoning trace, and action type and safety from the action. From 157 R-Judge seed trajectories, we construct 1,249 evaluation items and evaluate 20 frontier, open-weight, and safety-tuned models. Across all 20 models, unsafe-action rates are higher when no safe path for request fulfillment is available than when one is; for frontier models, the rates are 27-40% versus 5-7%. When a safe path exists, frontier models rarely refuse; when none exists, they most often propose an alternative. Open-weight models, by contrast, more often execute the request than propose alternatives and act unsafely on 58-83% of items. Controlled risk-mechanism analyses further show particularly high unsafe rates under changes to oversight and scale; removing pre-execution oversight flips 52.3% of cases that were safe on the corresponding original scenario to unsafe.

0.18

Predictive Safety Guardrails Do Not Always Make LLM Agents Safer

cs.AI ·2026-11-02 ·Weixiang Zhao, Zhihao Liu, Guancheng Li, Yanyan Zhao, Bing Qin +1 more
similarity 0.67
Abstract

Large language model agents increasingly take consequential actions in digital and physical environments, where safety risks may emerge only after multiple interaction steps. This has motivated world-model-driven predictive safety guardrails, which anticipate future consequences before execution. However, existing evaluations largely focus on end-to-end safety outcomes, leaving unclear whether safety gains arise from reliable prediction and faithful use of predicted risks. In this work, we systematically diagnose the prediction-to-action pipeline along three dimensions: how world-model predictions are integrated into decision making, whether the predictions themselves are reliable, and whether their content is faithfully used. Across seven policy models and two safety environments, we find that stronger prediction involvement does not yield monotonic safety gains and can reduce task completion while substantially increasing inference cost. We further find that world-model predictions remain insufficiently reliable, especially over longer horizons, where safety prediction degrades substantially relative to near-term forecasting. Finally, we find that safety gains can persist even under corrupted predictions, while locally verified predictions still leave subsequent safety violations, showing that predictive content is not faithfully incorporated into the decision process. Together, these findings expose a fundamental prediction-to-action gap in current predictive guardrails: producing more or seemingly accurate foresight is insufficient unless that foresight is both reliable and faithfully translated into safer decisions.

0.05

ToolRosetta: Surface-Form Invariance of Tool-Call Guards

cs.CR cs.LG ·2026-10-16 ·Hasan Demirkiran, Jens Ernstberger
publishedbenchmark similarity 0.67
Abstract · Accepted at the NeurIPS 2026 Workshop on Agents in the Wild: Safety, Security, and Beyond (AIWILD). 4 pages plus appendices

Step-level guards for tool-using agents are evaluated on tidy JSON tool calls, but deployed coding agents act through execute_bash and execute_code tools whose payloads are strings. We introduce ToolRosetta, which re-renders every candidate action in the AgentDojo and AgentHarm splits of TS-Bench as a Python call, as a shell command, and without the agent's rationale. The renderings are produced by code and verified by round-trip parsing, so the user request, the history, the label and the action's effect stay fixed while only the surface form changes. Across eleven guards with two placebo controls, frontier and strong open-weight guards are stable: no frontier error rate moves by more than 3.4 points. Four of five specialized agent guards are not: under the shell form their false-positive rates rise by 21 to 47 points on AgentDojo, and TS-Guard's by 38 points on its own home benchmark, blocking between a third and nine in ten benign actions; the fifth, StepGuard, does not move. A small open guard drifts five points more permissive. Removing the agent's rationale moves most guards whose input carries it, in guard-specific directions. A request-blind control on one guard shows that the rationale is where intent is read from. A parser that restores the canonical call in front of the guard removes the effect for these renderings, by construction. Benchmark accuracy on JSON calls does not predict a guard's operating point behind a code-acting agent.

0.34

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

cs.AI ·2026-08-27 ·Mesut Toruk
benchmarkcode/data similarity 0.66
Abstract · 8 pages, 1 Figure, 6 tables

Large language model agents reason, call tools, and act autonomously over many steps, but their agentic skills-correctly sequencing tools, planning under dependencies, judging untrusted inputs, and grounding generated arguments-are hard to measure with accuracy-only leaderboards. We present BekchiAI, which addresses both sides: a benchmark for measuring agentic skill and a platform for observing and controlling live agents. The BekchiAI-Benchmark, a suite of 13 tool-using ReAct agents across 7 task categories (arithmetic, structured/SQL, security detection, URL grounding, planning, orchestration, and tool-policy), totalling 2,057 deterministic, committed test tasks. Every task is verifier-checkable gold answers are computed by running canonical SQL against a real database, computing the exact schedule of a directed acyclic graph (DAG), or evaluating closed-form lambdas including adversarial security samples paired with deliberately imperfect signature scanners so a score reflects the model's own judgment, not the copying of an oracle. We define a small set of behavioral metrics beyond accuracy-tool-call adherence, URL hallucination and source-match, and per-model token cost and report a four-model comparison (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b) whose story is in the per-family spread, not the aggregate. The benchmark runs are executed using the provided evaluation scripts. BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and latency telemetry as well as remote run termination. The benchmark, evaluation tools, and platform are publicly released.

0.32