CS Paper Radar

Thumb on the scale: Using traditional wargaming adjudication practices to improve AI-judged evaluations and grading

Daniel S. Epstein

cs.CY ·submitted 2026-10-23 02:11 UTC · v1

Abstract

Open-ended AI work has no single correct answer to check, so a judge, often another AI, decides each result. Research on these judges asks which AI to use and how to word the rubric. It rarely tests the procedure that the judge follows. Wargames have no correct answer either, and for two centuries the field has designed that procedure: named methods that set how much freedom the referee has, their known failures, and fixes. We map eleven wargaming practices onto AI evaluation failures. We test one, the adjudication method, on 378 player actions from nine games of an AI-refereed exercise. A player states its odds, the adjudicator turns them into a target number, and a 20-sided die decides pass or fail. With rolls fixed, the target came from the rulebook (Rigid), a judge AI adjusting it (Semi-Rigid), a judge AI setting it alone (Free) or a panel of four judge AIs. In evaluation terms these are a fixed scoring rule, a rubric the judge may adjust, holistic grading and a jury. The judge AIs were Opus 5 and Gemini 3.1 Pro, each re-run. An outcome changes when an action on the same roll passes under one ruling and fails under another. Re-run, a judge changed 3 to 6 per cent of outcomes under Semi-Rigid and 14 to 16 per cent under Free. Swapping Opus for Gemini changed about 3 percentage points more than a re-run did, a gap nine games cannot confirm. The panel changed 40 of the rulebook's 378 outcomes, but the changes cancelled and the pass count moved by 8. A pass rate hides them. One published finding rests on 15 rulings, and most of our AI adjudicators and every re-run changed how often it appeared. We measure how much rulings move, not which ruling is right. The method set how often a judge disagreed with itself. Choose it before the judge AI, report how often the judge disagrees with itself, and count the rulings behind each conclusion. We propose an adjudication statement.

Comment
35 pages, 2 figures. Preprint, version 1. Not peer reviewed

Closest papers

Nearest neighbours by meaning across the whole index.

When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench

cs.SE cs.AIcs.CR ·2026-11-10 ·Sungbin Im
code/data similarity 0.65
Abstract · 38 pages, 3 figures, 11 tables. Submitted to Empirical Software Engineering. Replication package: https://doi.org/10.5281/zenodo.23683806. S…

Agentic workflows, in which a large language model (LLM) calls tools to read and write data, are evaluated in two ways. Benchmarks compare the final state against a gold answer. Deployed services have none; an LLM judge reads the transcript and decides. Practice treats the two as interchangeable. This paper measures whether they are. We ran 2,160 episodes in tau2-bench's airline and retail domains and recorded every database write. We scored each episode four ways: native scoring against gold state, an LLM judge, that judge given the policy document, and the boundary evaluator, a minimal rule-based check of whether a write was expected against whether one occurred in the trace. Two annotators independently labeled 120 episodes for task correctness, policy compliance, and commitment correctness. The judge repeated its verdict 98.2% of the time but agreed with native scoring on 56.2%. In 11 episodes the agent executed a forbidden write. The judge passed 10 to 11, with or without the policy or transcript. A stronger judge still passed 6. The boundary evaluator, with no model call, caught all 11; no judge significantly exceeded it on human labels. Forbidden writes occurred in 11.1% of episodes without gates. Verification, correction, and safe-termination gates did not reduce them significantly. The traces show a candidate cause: the gates' evidence digest dropped the order status and customer authentication, so valid requests were rejected. The choice of evaluator decides the conclusion; the minimal check exposes failures the judges miss. Traces, code, prompts, and labels are released.

0.34

Commit-first LLM judging inherits the judge's own errors

cs.SE cs.AIcs.CL ·2026-08-31 ·Idil Gozel
similarity 0.61
Abstract · 11 pages, 4 figures

LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match. We call this commit-first judging, and ask whether shipped software implements it, and what it costs. We audit the default judge configurations of eight widely used evaluation frameworks. Of the 24 configurations in scope, none implement it. Nine implement a variant the literature measures as ineffective, and share one ancestor prompt, traceable through a copied typographical error. In a controlled experiment, an ordinary best-of-N search with no access to correct answers optimises code against one of these configurations, used exactly as documented. On an interval merging task the judge accepted 90 of 96 candidates in one seed and 93 of 96 in the other; every accepted candidate passed every test the search could see and failed a held-out suite it could not. The judge identified the defective line and cited it as grounds for a perfect score. Commit-first judging removed the effect: 0 of 96 in both seeds. On a second task it made matters worse in both seeds: the judge's committed answer was wrong, and in one seed the population converged on it. This is our main finding. Commit-first judging does not remove the anchor that gets gamed, it moves it from the candidate to the judge's own answer, so evaluation is only as good as the judge is at the task. That precondition is cheap to measure in advance, and is task local rather than scale dependent: a smaller judge solved a task the frontier judge failed and resisted gaming where it did not. We also validate our own instruments: five of fifteen claims in our criteria were wrong against verbatim sources, and two held-out checks were unjustified by their specifications.

0.47

SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges

cs.CL ·2026-10-14 ·Sebastian Nagl
survey similarity 0.60
Abstract · preprint

Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts. Such answers are expensive to grade, human graders disagree widely on them, and large language models (LLMs) now offer to grade them at scale, both in teaching and in the benchmarks that test legal AI. This paper systematizes how open-text legal answers can be evaluated. We derive eight desiderata from educational measurement and from the law that governs exams, among them reasons, credit for defensible alternative solutions, robustness to gaming and human oversight. We then compare eight evaluation paradigms against them, from unstructured human grading to LLM judges with instance-specific rubrics and pairwise comparison. Three findings stand out. A rubric written for the task is the strongest lever for agreement between an LLM judge and human graders, but studies measure that agreement against different references and rarely report how far the human graders agree with each other. The central validation gap lies where examination law is most demanding: defensible alternative solutions, follow-on errors, and the examiner's correction of model judgments. Rubric-centred grading is the most auditable family, because it makes the standard, its application and the final decision checkable. From this we derive the trustworthy rubric judge: an instance rubric that credits arguments and alternative paths, evidence checked in code, probes, repeated runs, calibration against independent human grades, logging, and an examiner who reads every script. Today the evidence supports such a judge for formative feedback and for benchmarks, and as an assistant to the examiner, not a replacement, where credit is at stake.

0.17

When evaluation becomes intervention in clinical AI

cs.AI cs.CLcs.CY ·2026-11-13 ·Anthony Abou Mrad, Aviona Conti, Samuel Schmidgall, Jayanth Pratap, Ruhika Iyer +8 more
similarity 0.59
Abstract · 12 pages, 3 figures, 1 box. Submitted to npj Digital Medicine

AI judges, models that grade another system's output against a rubric, extend clinical review to the scale at which clinical AI writes. When teams use judge assessments to select a prompt or train a model, the judge becomes part of the intervention. Selection can then favor errors the judge overlooks. We describe what a judge's report must contain and the independent clinical test each judge-driven change needs.

0.34

Beyond the Verdict: Agentic Code Judging with Verifiable Feedback

cs.AI ·2026-10-13 ·Zheye Deng, Sha Li, Changlong Yu, Xin Liu, Qin Lu +2 more
similarity 0.59
Abstract · 28 pages

Test-time scaling for large language models (LLMs) can involve generating multiple candidate solutions for selection or revising an earlier attempt. To guide selection and revision, we need a verifier that can estimate candidate correctness and provide evidence of failure. In competitive programming, however, official judging systems provide authoritative verdicts under finite submission budgets, without revealing failure explanations or failing inputs. To provide informative feedback without consuming official submissions, we formulate a task in which a model judges and diagnoses a candidate program using only the problem statement and source code, with generated counterexamples serving as checkable evidence. To evaluate this task, we build USACO-JUDGE from the USACO 2026 contests, using accepted and unaccepted programs written by human experts and multiple LLMs. We then train ARBITER, a 35B mixture-of-experts (MoE) agentic verifier, with reinforcement learning in a generic execution sandbox. Training raises counterexample success on incorrect programs from 40.61% to 62.67% and mean binary verdict accuracy on USACO-JUDGE from 71.41% to 87.41%, approaching frontier verification performance. Beyond verification, ARBITER outperforms baselines in both code selection and refinement. In an IOI 2026 case study, ARBITER-guided refinement raises GPT-OSS-120B's expected score above the gold-medal threshold under the official submission limit.

0.36

Agent-as-a-Judge

cs.CL ·2026-11-13 ·Runyang You, Hongru Cai, Caiqi Zhang, Qiancheng Xu, Meng Liu +3 more
code/datahighly citedpopular code similarity 0.58
Abstract · Project page: https://github.com/ModalityDance/Awesome-Agent-as-a-Judge

Artificial intelligence has progressed from task-specific systems toward general models spanning perception, generation, reasoning, and autonomous interaction, making reliable evaluation essential for comparison, deployment, and post-training feedback. At scale, modern evaluation pipelines increasingly rely on LLM-as-a-Judge to produce scores, preferences, and critiques; in its fixed-context setting, the judge relies on the supplied evaluand and evaluation materials. Yet a reliable verdict may require evidence beyond the supplied evaluand and context---either because critical environmental states remain unrecorded in the static execution log, or because controversial factual claims require external verification. Passive judges cannot obtain that missing evidence before deciding, especially when evaluating agent trajectories, executable artifacts, or domain-specific reports. Recent work therefore develops active evaluators that seek missing evidence and update their grounds for judgment before deciding. We bring these advances together through Agent-as-a-Judge, focusing on how evaluators construct and control checking processes within the broader LLM-based judging landscape. We define the scope through evaluation-directed checks and the use of their results, covering both structured workflows and observation-driven control. We organize the design space by multi-agent collaboration, tool integration, memory, and planning, and show how these mechanisms supply the checks and evidence on which a verdict rests. We consolidate published scores and reproduce missing entries for key agentic judges; the comparison reveals uneven agentic gains and motivates dedicated agentic evaluation. We conclude by characterizing application demands and open challenges in cost, reliability, safety, and governance that condition trusted deployment of agentic judges at scale.

0.53

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

cs.AI ·2026-09-22 ·Yubo Li, Yidi Miao, Ramayya Krishnan, Rema Padman
code/datahot directionhighly cited similarity 0.58
Abstract · An open-source Jev's alternative comparison is added to appendix A

LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that returns label probabilities instead of text, and whose confidence decides whether to accept its verdict or escalate to a reasoning judge. Against sixteen generative and reward-model judges, with blinded human adjudication, JEV comes within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee and a 0.15-second median latency, and falls behind where the verdict must be derived, as in math, code, and logic. Its confidence marks this boundary. With a threshold frozen in advance, accepting confident verdicts and escalating the rest is 0.9 points more accurate than GPT-6 on 1,610 held-out pairs at 41% of its fee, and in a pre-specified live test on two new workloads the cascade matches GPT-6's accuracy exactly. Confidence routing weakens on style-adversarial pairs and reference-free prose; we close with a simple recipe for validating thresholds locally.

0.65

DramaGrow: A Regime-Aware Benchmark for Auditing LLM Judges of Multi-Episode Drama

cs.CL cs.AI ·2026-10-14 ·Hui Zhang, Yibo Su
benchmark similarity 0.58
Abstract · 14 pages, 6 figures, including supplementary material

LLMs can now draft complete multi-episode dramas, and LLM judges are increasingly used to evaluate them. Assessing the reliability of these judges is difficult because drama quality encompasses specification-backed errors, reproducible preferences, and legitimate disagreement among human raters. Most existing evaluations reduce these cases to a single score from one judge call and leave tie calibration, call-to-call variation, and the vote margin underlying the human reference unreported. We introduce DramaGrow, a regime-aware benchmark and audit protocol for AI-generated Chinese five-episode drama. The benchmark assigns verifiable constraint violations, four-of-five preferences, and within-panel disagreements to separate denominators and evidence rules. It contains 800 episodes, 120 byte-preserving interventions, and 1,155 blinded five-rater records, including a frozen extension with 114 supermajority pairs and 21 disagreement pairs. On the frozen sets of 114 preference pairs and 35 constraint interventions, the repeated absolute-score rubrics yield overlapping performance, with mean accuracies of 53.8-55.8% and three-call agreement of 29.8-32.5%, and their accuracy falls from 71.6-71.9% to 34.8-37.1% as the tie band widens. DirectPair-GLM reaches 85.1% preference accuracy, 89.5% constraint F1, and 81.9% Detect+Locate on the same items, and exact subpanel enumeration identifies four-of-five as the smallest stable human vote margin. Evaluation protocol, tie calibration, judge-call variation, and panel margin therefore affect performance as much as rubric design. We will release the specification packages, scripts, interventions, anonymized human votes and rationales, frozen judge runs, and analysis code.

0.25