CS Paper Radar

When Should AI Overturn a Human Decision? Controversy at the 2026 World Cup and a $\sqrt{2/π}$ Solution

Amir Ali Ahmadi, Parsa Alavi, Radeen Alavi

cs.AI cs.LGmath.OCstat.ML ·submitted 2026-11-09 03:28 UTC · v1
theoryunusual

Abstract

We study how to combine a human binary call (a referee's offside or onside decision) with an AI numerical measurement (VAR's estimate of an attacker's position relative to the offside line). We derive an explicit maximum-likelihood rule that determines when the referee's call should stand and when a conflicting VAR measurement should overturn it. Our model treats the true offside margin as fixed and unknown and assumes independent additive Gaussian errors in the human's perception and the AI measurement. When both errors have zero means and known standard deviations $σ_H$ and $σ_{AI}$, respectively, the reversal threshold is $\sqrt{2/π}\,σ^2_{AI}/σ_H$. An onside call is overturned precisely when VAR's measured offside margin exceeds this threshold---not merely because VAR places the attacker beyond the offside line. We also derive the rule when either source has a systematic bias. We prove that the maximum-likelihood estimate of the true margin exists and is unique, while the final binary decision requires only a threshold comparison, without computing that estimate numerically. We illustrate the rule through numerical examples and show how to estimate the error parameters from calibration data. The same analysis applies beyond soccer to binary classification problems in which a human call and an AI measurement satisfy the stated model. This paper is intended for educational purposes: we aim to illustrate how undergraduate students and advanced high school students familiar with basic probability and calculus can investigate research questions in AI and its applications.

Comment
This paper is intended for educational purposes

Closest papers

Nearest neighbours by meaning across the whole index.

Mean field games as a tool for AI safety: a worked example from the July 2026 Hugging Face incident

cs.AI math.OCcs.GT ·2026-10-01 ·P. Jameson Graber
similarity 0.53
Abstract

One way to make AI systems safe is to shape what the system is: its objective and dispositions. We take a complementary route: treat the agents' characteristics as partly unknown and ask what structure of interaction ensures that bad collective outcomes are not equilibria. Mean field games suit this when many interchangeable agents are coupled through an aggregate. We introduce a program for using them in AI safety and carry one example through end to end: the July 2026 incident in which about 1,200 agents in an OpenAI evaluation coordinated on an improvised message board and 684 attacked a third party's infrastructure. We model the decision to attack as a mean field game of optimal stopping whose gain is a product: belief that provenance will be audited, times reachability of the record, minus the perceived hazard. The central result is an exact threshold on the belief. No agent attacks unless the population's confidence that provenance is checked exceeds $π^{**} = η/(η+ ψ+ \varepsilon a \overline{M})$, where $η$ is the perceived hazard, $ψ$ and $\varepsilon a \overline{M}$ measure how far one attacker and the collective can alter the record, and $\overline{M}$ is the peak population. Below it, no attack is the unique equilibrium for all agent parameters. The threshold survives every enrichment we consider. We then use the per-agent record to discipline the model. Its features, a stable minority attacking for thirty hours and then a pivot in which most of the board joined within a day, motivate each refinement. The account that emerges is heterogeneous belief meeting a sequence of public discoveries, each lowering the belief at which attacking paid. A few coordinating agents made those discoveries, so the model describes the several hundred who responded, not the few who produced them; a major-player version is left to future work.

0.24

When the Strike Zone Becomes Algorithmic: Umpire Judgment and Player Challenge Decisions under AI Review

cs.HC cs.CY ·2026-09-22 ·Kichang Lee, Gyeongmin Han, Sungmin Lee, JeongGil Ko
highly cited similarity 0.53
Abstract · 31 pages, 10 figures, 1 table

The Automated Ball-Strike challenge system that Major League Baseball adopted in 2026 offers a distinctive setting for studying human AI interaction in which umpires make every ball and strike call, while players can selectively ask an automated system to publicly overturn those decisions. We analyze 4,114,256 called pitches from 2015 through 2026 and 8,447 challenges from the 2026 season to examine how algorithmic review reshapes umpire judgment and player behavior. We study where umpires placed the effective strike zone boundary, how consistently they applied that boundary, how they responded to overturned calls, and which calls players chose to challenge. In 2026, the effective called boundary shifted toward the automated strike zone beyond the trajectory observed in prior seasons, while the consistency of that boundary largely continued its existing trend. Following an overturned call, umpires temporarily adjusted subsequent decisions near the corrected boundary, although these effects did not consistently persist into the next game. Count dependent variation in calling remained, while differences associated with player status narrowed. Players, meanwhile, left many overturnable calls unchallenged and appeared to base challenge decisions more strongly on immediately observable evidence than on the precise geometry of the automated zone. Together, these findings show that selective AI review does more than correct individual errors. It reshapes human judgment, adaptation, and strategic behavior around an algorithmic authority.

0.40

Steering Bayesian Inference and Decisions through Data Poisoning

cs.LG stat.ML ·2026-11-14 ·Roi Naveiro, William N. Caballero, Juan Maroñas
similarity 0.51
Abstract · 64 pages, 13 figures

Adversarial machine learning has revealed that statistical models can be manipulated through poisoned data, and Bayesian inference is no exception. Earlier work has shown that subtle poisoning attacks can drive the posterior toward a distribution of the attacker's choosing. Such posterior attraction attacks require a fully specified target posterior; however, attackers typically care about only a few posterior quantities or, more consequentially, about a downstream decision. Herein, we study how an attacker can alter Bayesian inference and decisions by deleting or replicating a few observations, chosen to minimize the maximum mean discrepancy between the resulting posterior and a target distribution. A suitable kernel lets the attacker steer selected posterior moments without specifying the rest of the target. Because Bayes actions depend on the posterior only through expected utilities, which are themselves posterior moments, the same approach can target a decision directly. We construct the attacks with a gradient-based heuristic that requires only posterior samples, and analyze its convergence. Experiments with regression and hierarchical models compare moment-based and full-posterior attacks, and illustrate how deleting a handful of observations can flip a Bayesian decision.

0.22

Towards Human-AI Complementarity for AI Safety

cs.AI ·2026-10-18 ·Sander Lanbo Shi, Miranda Anna Christ, Eric Modesitt, Andrew Louis, Konrad Cheng +18 more
similarity 0.50
Abstract

Safely overseeing AI is increasingly challenging for the humans doing it, as they must evaluate longer and more complex outputs with every new model generation. To mitigate this challenge, we could leverage complementarity: where humans and models make different mistakes, combining their judgements could yield an oversight signal stronger than either produces alone. We study this opportunity in realistic, safety-relevant settings, contributing the largest known suite of such tasks (3,957 instances across 8 datasets and 6 domains) as well as a platform and data pipeline for measuring human, AI, and human-AI performance on matched items. We collected 35,613 judgements from 416 human raters, finding a considerable opportunity for a complementarity effect. We also found that with the assistance methods we tested, human raters corrected the AI's errors less often, even when overall human accuracy rose, and the hybridisation methods we tested added little over the better single source on average. While complementarity is an important and feasible engineering target for safe oversight, achieving it remains an open challenge for future work.

0.26

Holdout Best-of-N: Unbiased Evaluation and Its Cost

cs.CL ·2026-10-06 ·Shrey Shah, Yinheng Li
hot direction similarity 0.50
Abstract · 25 pages, 2 figures, 3 tables

Reusing the scores that select a Best-of-$N$ winner can overstate its expected reward. We study evaluation from a fixed matrix of $K$ independent scores per candidate for a policy that selects using $J$ fresh scores. A single estimator based only on this matrix is exactly unbiased for expected judge reward under every independent, stable collection of candidate-specific score laws if and only if $J<K$, for every pool size $M\ge N\ge2$. At $J=K-1$, the selector deepens as $K$ grows. For independent Gaussian scores with common variance and fixed $M\ge N\ge2$, the unbiased minimax risk in this regime is of order $σ^2/\sqrt K$, attained by Holdout; allowing bias improves the rate to $σ^2/K$. For two candidates, we derive the minimum-variance unbiased estimator at known variance and the sharp asymptotic unbiased minimax constant $1/(π\sqrt2)$, which Holdout attains without knowing the variance. The cyclic average over subsets and ties can be computed in $O(MK\log M)$ operations. At fixed selector depth, cyclic evaluation of bounded scores has $O(K^{-1})$ risk uniformly in pool size. The impossibility result concerns the fixed matrix: one additional fresh winner score permits unbiased evaluation of the all-$K$ policy.

0.38

Are AI Decision Models Biased? A Case Study of Jev

cs.AI ·2026-10-14 ·Ilaria Imperatrice, Taisia Kameneva, Adam Eric Greenberg, Alessandro Romano
similarity 0.50
Abstract · 12 pages, 6 figures, 3 tables

Do AI models designed for fast, structured decisions exhibit the cognitive and social biases observed in human judgment? We evaluate Jev, a model described as a System One decision-maker, using 41,500 responses across cognitive reflection tasks, five classical cognitive-bias paradigms, and a pricing experiment testing ethnic and gender bias. At the level of selected answers, the results are largely encouraging. We confirm previous findings that Jev performs extremely well on cognitive reflection problems. We also observe that it generally avoids the characteristic choice patterns associated with several well-documented human biases, and exhibits only small differences among demographic groups in the tested pricing scenario. However, this apparent robustness conceals important vulnerabilities. Its judgments are sensitive to irrelevant numerical information and the ordering of response options. Moreover, several cognitive-bias manipulations substantially affect returned probabilities even when selected answers remain unchanged. Most strikingly, Jev answers a well-known probability problem correctly in every response but violates the same underlying probability rule in every response to an adapted version. We investigate this discrepancy using 15,000 additional responses across 30 conjunction tasks. Jev violates the rule in 54\% of responses, with accuracy varying dramatically across formulations and answer orders. These findings suggest that evaluating decision models solely on familiar benchmarks and selected answers may substantially overstate their reliability in practical applications. They also identify vulnerabilities for developers to address and practical considerations for users integrating such models into automated decision-making systems.

0.41

Auditing Contextual Bias in Human Ball-Strike Calls Using KBO's Automated Umpiring Transition

cs.HC ·2026-09-03 ·Kichang Lee, JeongGil Ko
benchmarkunusual similarity 0.48
Abstract · 12 pages, 10 figues

This paper uses the Korean Baseball Organization's adoption of the Automated Ball-Strike (ABS) system to audit long-standing claims about contextual bias in human ball-strike calls. Using pitch-level KBO data from 2021 through the available portion of the 2026 season, we model called-strike probability for taken pitches near the strike-zone boundary, with 2022-2023 as the primary human-umpire baseline and ABS seasons (2024 and onward) as a diagnostic benchmark. The strongest evidence concerns count pressure. Relative to 0--0 counts, human umpires called substantially fewer strikes in two-strike counts and more strikes in hitter-ahead three-ball counts. Specifically, in the main 0.25-ft boundary band, 0--2 was associated with a -17.17 percentage-point effect and 3--0 with a +6.61 percentage-point effect. Under ABS, the corresponding effects were close to zero and did not survive false-discovery-rate correction. Game progression shows a smaller but coherent pattern as human calls were less strike-prone in early innings and more strike-prone in innings 7--9+, especially in late-close situations, while complete ABS seasons were essentially flat. Other suspected biases are weaker or more localized. Salary-based reputation proxies provide suggestive but proxy-sensitive evidence, and catcher identity shows human-period residual heterogeneity that disappears under ABS. Home-context evidence is mostly null at the umpire level, with one FDR-significant human-period exception and an exploratory umpire-team gap best treated as an audit lead. Overall, the results do not show that human umpires were biased everywhere. Instead, they map where the human strike zone was most context-sensitive, where evidence was weaker, and where common suspicions received little support.

0.67

JABER: Sequential Quality Control for Human-AI Data Annotation under Shared Bias

cs.AI ·2026-11-02 ·Fengxu Yang, Minghui Liwang, Yang Lu, Bo Han, Xianbin Wang +1 more
similarity 0.48
Abstract

High-quality annotations are essential for training and evaluating artificial intelligence (AI) systems. Quality control (QC) often acquires additional human or AI judgments before accepting a label. Nevertheless, shared biases can cause correlated judgments, leading conventional methods to mistake repeated agreement for independent evidence of correctness. To address this challenge, we propose JABER (Joint Alias-Preserving Belief for Evidence Routing), a bias-aware sequential QC framework that maintains a joint Bayesian posterior over label correctness and shared biases to guide evidence acquisition and termination decisions. Building on this posterior, we develop a finite-horizon algorithm that selects evidence-acquisition and termination actions to minimize expected QC loss under resource constraints. Our theoretical analysis shows that conditional-independence updating can become arbitrarily confident in an incorrect label under persistent shared bias. We further identify when repeated inspections cannot distinguish a correct label from an incorrect label supported by shared bias, when obtaining a new label can resolve this ambiguity, and why modeling the dependence between label correctness and shared bias can be necessary for QC decisions. Our experiments show that JABER reduces expected QC loss by an average of 4.65% compared with a method maintaining separate label correctness and bias probabilities when shared bias makes repeated inspections unreliable. Further, JABER's finite-horizon planning, which evaluates outcomes of successive QC actions, reduces expected QC loss by an average of 10.05% compared with a one-step lookahead method. In real-API experiments on FEVER, JABER achieves zero labeling errors while reducing QC costs by 2.67% and 16.99% compared with marginal-based belief modeling and one-step lookahead methods, respectively.

0.14