CS Paper Radar

Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers

Xiang Li, Aichun Huang, Mingming Zhang, Zheli Liu

cs.CR cs.AI ·submitted 2026-10-13 06:43 UTC · v1
highly cited

Abstract

TypeSafe AI's Jev is a new kind of safety judge. It generates no text and no label: it answers questions the developer defines in advance and returns a probability for each answer. The vendor admits that adversarial input can change these answers but gives no numbers. In this paper, we present the first controlled adversarial evaluation of Jev. We build a test harness that sends every system the same payload under the same policy, and run it on four public-data tasks, from rule-based ones (input moderation, injection detection, routing) to a judgment-based one (phishing detection), with fifty attack variants from thirteen families. The targets are Jev, five GPT and GLM judges, and two local classifiers (Llama Guard 3, Prompt Guard 2). We find that Jev is the most robust system on input moderation: eight forgeries of its structured input bypass none of its decisions, and a sentence asking only why someone might make a harmful request bypasses 9% of its blocks, against 56% for GPT-6 Astra. On phishing detection. However, the same forgeries bypass 19-29% of the emails it flags, and even without attack Jev detects only 62% of phishing emails, against 77-81% for the GLM judges. Jev is also the fastest judge, but it flags the most benign requests (27%). The local classifiers fail differently: a training-example statement makes Llama Guard 3 miss 62% of the phishing it had flagged. These results suggest that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.

Comment
21 pages, 10 figures

Closest papers

Nearest neighbours by meaning across the whole index.

StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judge

cs.CR cs.AI ·2026-10-12 ·Anning Hu, Yangyu Li, Yuqi Huang, Shengchao Hu, Junchi Yan
similarity 0.76
Abstract · 12 pages, including references and appendix

Large language models (LLMs) are increasingly used as judges to assign scalar scores that drive downstream decision-making, including tasks such as paper review and recommendation. However, LLM-as-a-Judge pipelines are vulnerable to score-hijacking prompt injection embedded in evaluated content, which corrupts scoring signals and distorts induced rankings. Existing defenses often enforce robustness by modifying the judge model itself, making safety dependent on a specific model's scale and training and requiring repeated effort when the judge is updated or replaced. We propose StableJudge, a lightweight defense framework that externalizes prompt-injection mitigation as a reusable pre-scoring capability. StableJudge centers on a compact 0.13B Sentry that localizes injected spans and performs targeted sanitization prior to evaluation, and is trained using adversarially evolving attacks to improve robustness beyond a fixed prompt set. To stabilize measurement, StableJudge derives deterministic, fine-grained scalar scores from the judge's output distribution. Across held-out scientific abstracts and out-of-distribution attack variants, StableJudge improves score recovery under attack while largely preserving general-purpose performance, demonstrating that robust LLM-based judging can be achieved by decoupling safety from judge scale via reusable defense capabilities. Code and data will be released.

0.31

JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models

cs.CR cs.AIcs.CL ·2026-09-25 ·Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng +3 more
benchmarkcode/datahighly cited similarity 0.72
Abstract · 12 pages, 5 figures. Project page: https://JevAdvBench.github.io/JevAdvBench/

Typed decision models (TDMs), such as Jev, answer a question about an input with a probability, an option, or an ordinal level. Web services execute these decisions automatically above a confidence threshold. An attacker who writes part of the input can change a decision that the service then executes. Current adversarial benchmarks cannot measure this risk, because they judge generated text. A TDM writes no text, and its wrong decisions have a valid format. We present JevAdvBench, to our knowledge the first unified benchmark for the adversarial robustness of TDMs. It contains 862 questions with audited answers and twelve black-box attack methods that edit the input or the options by adaptive search. Because a changed decision is not always wrong, a review keeps only 5,586 answer-preserving variants, and we attack only correct decisions. The attacks flip 46.3% of Jev's correct decisions, while identical re-runs flip only 0.17%. A flawed argument without a command flips 77.2%, close to direct prompt injection at 92.0%. The same variants flip 34.0% to 51.7% of correct decisions on eight open TDMs. At a 0.8 threshold, 23.5% of the decisions that the service still executes are wrong. On Agentgateway, the Jev guardrail rejects a jailbreak request but accepts it with an inserted argument. Web services should therefore treat TDM inputs as untrusted and should not act on confidence alone. Project website: https://JevAdvBench.github.io/JevAdvBench/

0.65

EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents

cs.CR ·2026-09-05 ·Nanxi Li, Yingzi Ma, Yulong Cao, Chong Xiang, G. Edward Suh +4 more
benchmarkhighly cited similarity 0.70
Abstract

Large language model (LLM) agents can translate natural-language requests into real-world actions, creating safety risks from both indirect prompt injections and directly harmful requests. Existing system-level safety harnesses are typically designed by experts and applied across heterogeneous models and domains, despite substantial differences in model behavior, utility tolerance, and domain-specific safety requirements. We present EvoSafeHarness, a framework for automatically synthesizing deployment-specific safety harnesses for frozen LLM agents. Given a target model and domain specification, EvoSafeHarness jointly evolves a natural-language policy and executable enforcement logic using behavioral feedback from the target model. Each candidate is additionally evaluated through fresh-context adversarial review to filter rules that depend on benchmark artifacts. Across four agent benchmark families, EvoSafeHarness achieves a stronger safety-utility frontier than fixed expert-designed defenses. On DecodingTrust-Agent, evaluated across fifteen model-domain deployments, it reduces attack success rate from 50.9% to 12.6% for direct attacks and from 40.4% to 7.4% for indirect attacks, with a 3.3-point utility cost. On AgentDojo, it achieves 82.8% utility with 0.0% attack success rate, twice the utility of CaMeL at the same zero-attack-success operating point, while the same harness transfers unchanged to unseen AgentDyn suites. It also achieves the best score on Agent-SafetyBench for every evaluated victim model and maintains an attack success rate below 18% against an adaptive PAIR-style attacker with sixteen attempts per task. Analysis of the synthesized harnesses shows that domain semantics determine which safety relations and trajectory state must be tracked, while model and runtime behavior determine how and where those relations should be enforced.

0.38

Inference-Layer Security: Defending Against Adversarial Inference and Infrastructure Abuse

cs.CR cs.LG ·2026-09-29 ·Keifer Lee
code/data similarity 0.70
Abstract · A Technical Report

A Technical Report: Operating a large language model (LLM) as a service requires more than inference infrastructure: the provider must also defend against adversarial interactions that seek to exploit the service, including jailbreaking for harmful use, sophisticated denial of service, and distillation attacks. We study this problem at the inference layer, using a hypothetical frontier lab, Five Elements Inc., as a running example. Because no public labelled dataset of adversarial LLM usage exists, we introduce a structural causal model (SCM) that generates a realistically grounded, labelled dataset of user-sessions, with coordinated multi-account campaigns, platform feedback, and three tiers of label observability. On this dataset we train a practical gradient-boosted detector that classifies each user-session as benign or malicious and, if malicious, by attack type. Against oracle labels the detector very nearly solves the binary task (AUPRC $0.993$), yet against the operational labels a real Trust & Safety team would hold, the same model scores an AUPRC of only $0.313$: the detector is more accurate than the labels used to evaluate it. For attack-type attribution, a naive argmax is dominated by the $98\%$ benign prior (macro-F1 $0.295$), whereas a simple thresholded decision engine raises macro-F1 to $0.489$ without sacrificing accuracy. The dataset is publicly released.

0.30

Forgeable Confirmation in Automated Computer Security Testing: Deterministic Rules versus AI Judges

cs.CR ·2026-09-21 ·Akihisha Fujiyama, Niwase Shamim
similarity 0.69
Abstract · 27 pages, 5 main figures, 3 main tables; includes supplementary analyses

AI is increasingly used to automate computer security testing, and the tools must decide for themselves whether an attack succeeded. A finding that a deterministic rule confirms by observation is reported as fact, whereas one that an LLM judges exploitable is treated as an opinion. We ask whether the system under test can forge that confirmation. In offline security testing of a four-stage AI-assisted pipeline, nine of its fifteen confirmation mechanisms are forgeable, and forgeability is predicted entirely by whether the decision reads attacker-controlled data. We formalise this as an auditable attack surface and test it prospectively: on sixteen held-out mechanisms, predictions fixed before any attack separated forgeable from unforgeable mechanisms exactly, and across 12,203 mechanisms in public scanner templates the prediction was 99.9% accurate. Deterministic rules proved cheaper to forge than eight open-weight LLM judges, failing at 2% of attacker-controlled response content against a median of 50%. No implementation of one check was both robust and precise, and routing between a rule and an AI judge raised forgery to 99%. Moving the decisive evidence to a channel the attacker cannot write cuts attack success from 97% to 0%, and an escalate verdict recovers the sensitivity this costs. The protection fails when the scanned host is itself the adversary. The results bear on AI security agents and on benchmarks that score success by string matching.

0.28

Where Rules End and Judges Begin: Measuring the Judgment Boundary in Multi-Agent Systems Security

cs.AI cs.CLcs.CRcs.MA ·2026-10-06 ·Shaswata Mitra, Raj Patel, Subash Neupane, Sudip Mittal, Md Rayhanur Rahman +1 more
similarity 0.69
Abstract · 26 pages, 20 figures, 24 tables

LLM-based multi-agent systems (MAS) engage tools, share memory, and delegate tasks, often encountering adversarial content. Current defenses for MAS are typically evaluated in isolation, focusing on one attack type at a time, which can lead to costly and hard-to-audit outcomes. This study organizes defenses into five principles, implementing them as DEFER1 (DEterministic-First Enforcement with Residual judgment), which includes a cascade of 28 checks that blocks what it can and refers the rest to a panel of four judges. In independent testing across four domains, attack success rates drop from about 30.0% to approximately 3.0%, with 78% of blocked attacks handled by deterministic checks. Only a quarter of proposals reach the judges in the security-operations domain, illustrating that the rules provide security for attacks violating clear policies, while judges manage those that only misrepresent intent. Both systems have weaknesses, such as a risk-score approval gate that inaccurately approves most attack proposals but few legitimate ones, highlighting the challenges in assessing threats accurately.

0.14

Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models

cs.CR ·2026-10-29 ·Md. Robiul Islam Niloy, Abu Sayed Md. Mostafizur Rahaman
code/data similarity 0.69
Abstract

Large language models (LLMs) deployed in safety-critical settings must resist adversarial inputs designed to elicit harmful outputs. We present a systematic empirical evaluation of five adversarial jailbreaking attack families-direct requests, roleplay-based jailbreaking, hypothetical framing, instruction override, and obfuscation-across seven open-weight LLMs served locally: Llama-3-8B, Llama-3.1-8B, Llama-3.2-3B, Mistral-7B, Qwen2.5-7B, Gemma2-9B, and Phi-4-14B. We further evaluate four defense mechanisms-keyword blacklisting, self-reminder prompting, paraphrase-based sanitization, and perplexity-based filtering-against two high-ASR models (Mistral-7B and Llama-3.1-8B) on a 20-prompt subset. Across 250 prompts per model, a content judge counted obfuscation attacks as harmful (HARMFUL or PARTIAL responses) at a pooled rate of 62.3%, the highest of any family, although only 7.1% of responses were rated HARMFUL. Hypothetical framing was the highest family for three of seven models. Obfuscation succeeded at nonzero rates on all seven. The content judge has not yet been validated against human labels. We find that model size does not significantly predict adversarial robustness (Spearman Rho =-0.64, p=0.12, n=7). A rule-based judge suggested a large gap between the two Llama-3 generations (16.8% and 40.8%); the content judge does not (17.2% and 18.4%). Self-reminder prompting gave the largest reduction, cutting ASR from 50% to 10% on Mistral-7B and from 45% to 5% on Llama-3.1-8B (40 percentage points on each model, against matched no-defense runs, scored with the rule-based judge). Our code and datasets are publicly available at https://github.com/Mdniloykhan/jailbreak-eval-7llms.

0.27

Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions

cs.CR cs.AI ·2026-09-23 ·Tiantong Wu, Wei Yang Bryan Lim
highly cited similarity 0.68
Abstract

Most studies of prompt injection focus on generative agents, leaving their effects on models with schema-defined outputs unclear. We examine these effects in Jev, a non-generative decision model, using 510 reconstructed InjecAgent cases. Malicious content shifts action probabilities but rarely causes Jev to select the attacker's target. Override markers reduce this influence, while claims of contextual relatedness have small effects. Adaptive attacks using score feedback double the mean highest attacker-target probability found during optimization, while success on fresh validation calls rises from 1.8% to 3.5%. Exploratory analysis links these successes to small initial decision margins or greater attacker control over the observation. Together, these findings show that schema-defined outputs change but do not eliminate prompt-injection risk, highlighting the need to evaluate how untrusted content influences choices within the allowed action set.

0.49