CS Paper Radar

Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluation

Yezhou Cheng, Runjia Du, Zeming Liu, Qibai Chen, Hang Lyu, Yilan Wei, Yankai Zeng, Bojun Lin

cs.AI cs.IT ·submitted 2026-09-24 07:15 UTC · v1
benchmarkhot direction

Abstract

Repeated evaluation can estimate a benchmark score accurately while still requiring replication to certify narrow uncertainty. We characterize that requirement on a fixed grid of $M$ tasks with $L$ binary paths per task under the hard budget $(M+t)K$, where each path costs at most $K$ responses or episodes. For fixed $L \ge 3$ and $0 < α\le 1/12$, the optimal expected width on the worst pure cohort is $Θ_{α,L}([M(t+1)]^{-1/2})$ when every task is observed and $Θ_{α,L}([M(t+\sqrt{M})]^{-1/2})$ when omission is allowed. The lower bounds cover adaptive hard-budget policies, and fixed random-subset designs attain both rates through disagreement certificates. A joint mean/disagreement interval turns the task-covering law into practical finite-budget inference. In an equal-budget LiveCodeBench replay with 16 models, 880 tasks, and five outputs per task, the task-covering design reduces median point-estimation MSE by 87.0\% relative to pooled uniform sampling, while the Joint certificate produces narrower confidence intervals in 15/16 panels and reduces median interval width by 30.6\%. Finite-regime analyses identify task coverage as the effective choice at the evaluated scale and characterize how cohort size and within-task agreement determine the useful operating region. Together, the sharp laws and fixed-budget evidence make replication and task coverage explicit design variables for information-efficient repeated evaluation.

Closest papers

Nearest neighbours by meaning across the whole index.

Pathwise-Budgeted LLM Routing: Online Allocation with K-Bit Tail-Cost Learning

cs.LG math.OC ·2026-11-04 ·Hanzhang Qin, Sophie H. Yu, Yuewei Zhang, Zhi-Hai Zhang
similarity 0.66
Abstract

Firms using large language models under fixed budgets must decide which model to call and whether to retry a failed task. Because a call's charge is revealed only after completion, one long response can exhaust the remaining budget and leave later tasks unserved. Our objective is to maximize the number of tasks solved before service stops. We develop a contextual bandit policy, ADP-HT, which learns each model's task-specific success probability and constructs conservative cost estimates from coarsened response-length labels on a quantile grid with $2^K$ intervals. The policy weighs predicted success against these costs, adjusting the cost penalty to realized spending and the budget available per remaining task. Under calibration, local regularity, and grid-coverage conditions, we bound the excess allowance in the policy's conservative cost estimates. At a fixed sample size, increasing $K$ reduces the discretization term but increases the estimation allowance. At the tightest tested budgets, ADP-HT solves more than twice as many MATH500 and GPQA tasks as the best implemented baseline. It achieves the highest mean cumulative score at ten of fourteen RouterBench budgets. On LiveCodeBench, it outperforms a deployed commercial router in both default and cost-saving modes at the three lowest test budgets.

0.16

Online Interval Selection with Binary Predictions: Optimal Consistency and a Continuous Robustness Tradeoff

cs.DS cs.LG ·2026-10-27 ·Zhong Han
theory similarity 0.63
Abstract · 45 pages, 4 figures. Streamlined appendix and clarified proof sketches and advice diagnostics. Includes ancillary evidence tables. Extended …

We study online interval selection with binary predictions of membership in a final optimal solution. Requests arrive in arbitrary order; accepted requests may be revoked, but rejected or revoked requests cannot be recovered. With rewards proportional to interval length, the best consistency under finite strict robustness is $d=(3+\sqrt5)/2$ when both guarantees hold almost surely on each fixed input, including for randomized algorithms. A continuous lower bound forces consistency strictly above $d$ at every robustness budget below $4\varphi+3\approx9.472$, whereas our deterministic algorithm attains $d$ with exact robustness $\varphi^5\approx11.090$, where $\varphi=(1+\sqrt5)/2$. Each held interval inherits a ledger recording positive prediction mass lost through irrevocable decisions. Tuning the replacement costs and both special thresholds gives a continuous family with exact consistency throughout and exact robustness on two parameter intervals, including $(C,R)=(25/9,10)$ and $(4,7)$. Randomization permits expected consistency approaching one with finite expected robustness, whereas almost-sure robustness forces expected consistency strictly above two at every finite budget.

0.20

Corruption-Robust Sparse Linear Contextual Bandits with Knapsack Constraints

cs.LG ·2026-09-29 ·Yige Wang, Hanyang Li, Yiming Zong, Wanteng Ma, Jiashuo Jiang
benchmark similarity 0.63
Abstract

We study sparse linear contextual bandits with knapsack constraints under joint reward and consumption corruption. Consumption corruption creates a challenge beyond corrupted rewards: it affects not only statistical estimates, but also the recorded budget, resource prices, and stopping decisions that govern future allocation. We develop Robust Optimistic Primal--Dual (ROPD), an estimator-modular framework that combines corruption-aware confidence widths with online resource prices and a budget-safety rule. With concrete sparse implementation, ROPD achieves regret against a clean population-LP benchmark of $\widetilde O(T^{2/3}+ΓT^{1/3})$ under forced exploration and population-design coverage, and $\widetilde O(\sqrt T+Γ)$ under on-policy realized-design coverage, for a supplied valid corruption bound $Γ$ under the stated proportional-budget scaling and fixed model/design parameters. When the corruption level is unknown, Shared-Grid adapts confidence radii around common point estimates fitted to a single realized history, incurring explicit initialization and master-comparison costs; its sharper on-policy guarantee additionally requires recommendation coverage. Both methods preserve observed budgets on every realization and bound clean resource violation by cumulative consumption corruption. These results connect corruption-robust sparse estimation with resource accounting, pricing, and stopping in high-dimensional online allocation.

0.22

Approximation and Hardness for the Data Validation Problem

cs.DS ·2026-11-04 ·Antoine Tinguely, G. Wessel van der Heijden
similarity 0.61
Abstract

Finnah, Gönsch and Otto [Eur. J. Oper. Res. 2025] recently introduced the Data Validation Problem (DVP): Given a transitive directed acyclic graph $G$ on $n$ vertices (resp. tasks) and $m$ edges (resp. precedences) and a probing budget, a policy can successively pay a probing fee and probe an edge in $G$ as long as the total probing fees do not exceed the budget. If a probe succeeds, the targeted edge $e$ is removed from $G$ and we receive a reward; if the probe fails, the edge stays in $G$ and cannot be probed again. We seek a policy with maximum expected cumulative reward. Crucially, $G$ must stay transitive throughout the probing process, so a policy can only challenge the precedence between consecutive tasks, that is, only probe edges in the transitive reduction of the current precedence graph. We show that DVP is strongly NP-hard and W[1]-hard with the probing budget $T$ as a parameter, even in the deterministic case (when every probe has probability $0$ or $1$ to succeed) and with unit probing costs and rewards. For the same setting, we give an approximation-preserving reduction from the Densest $k$-Subgraph (D$k$S) Problem, with the consequence that there is no polynomial-time $2^{(\log n)^{1-δ}}$-approximation algorithm and no FPT $\sqrt{T}$-approximation algorithm for DVP, assuming the Almost-Exponential Time Hypothesis and the Strongish Planted Clique Hypothesis, respectively. We also present the first known policies with provable worst-case guarantees for DVP, namely an $m$-approximation algorithm for instances with uniform probabilities and probing costs, a $2T/p_{\min}^T$-approximation algorithm for general instances, where $p_{\min}$ is the minimum probing success probability, and an exact FPT algorithm on tournament graphs, in the deterministic setting.

0.30

CFR without Unbiasedness: Debit-Budgeted Public Feedback

cs.GT cs.LG ·2026-08-14 ·Jiaxing Guo, Lei Ye
theory similarity 0.61
Abstract · 35 pages, 2 figures

At a finite public-chance cut, partial counterfactual regret minimization (CFR) can evaluate successive outcome batches at learner states changed by earlier batches. Balanced coverage then lacks the conditional unbiasedness used by Monte Carlo CFR analyses. We introduce \emph{target debit}, the one-sided cumulative underestimation needed to transfer regret on delivered feedback to the dense target game. Our central theorem bounds weighted exploitability by delivered-row regret, target debit, and a native alternating-update correction. This yields Debit-Budgeted CFR (DB-CFR), which selects a public plan from completed history, executes it once, and settles its debit afterward. Complete coverage has zero debit and supplies exact recovery. With at most $\ell$ unsettled blocks, hidden debit is bounded by the sum of the $\ell$ largest predictable block envelopes; for fixed $\ell$ and polynomial output weights, the resulting $O(T^{-1})$ term preserves the additive-RM/RM+ rate. A finite converse gives equal-metadata plans with charges $0$ and $11/8$, proving that a nontrivial strict-budget decision requires semantic information or bounded post-execution settlement. In two released heads-up no-limit hold'em turn endgames, all 42 registered shallow comparisons favor partial coverage before complete coverage overtakes it between 32 and 64 rounds. In public Leduc, a lagged rule fixed on four development orders beats the best matched-recurrence-work fixed width on all ten unseen orders (median NashConv gain $0.0456$) without future candidate replay.

0.10

From Discovery to Decision: Finite-Budget Recoverability in LLM Voting

cs.AI ·2026-10-01 ·Shaoang Li, Jian Li
similarity 0.61
Abstract

Voting over multiple LLM responses is a common primitive in test-time scaling and ensemble inference. Collecting more responses can expand the candidate pool and increase the chance that a correct answer is discovered. Under a fixed call budget, a discovered answer still needs to accumulate enough support within the remaining calls to become the final plurality winner, creating a discovery-to-decision gap. In this work, we characterize this gap through the realized vote state and remaining call budget. We derive a sharp recoverability threshold and show that, as sampling proceeds, the observed candidate set can only expand while the set of reachable endpoint winners can only contract, inducing a candidate-level conversion window. Under a specified iid response law, the same state yields exact finite-horizon endpoint probabilities. We further show that merging wrong-answer identities preserves single-call correctness and cannot improve plurality accuracy, and that the effect of redistributing wrong-answer probability depends on the realized vote state. Singleton reachability yields a gold-free exact locking certificate. For a known answer universe, its first trigger is the earliest prefix at which all admissible continuations yield the same fixed-budget output. Empirically, most discovered-but-unselected correct answers lose reachability only after discovery. In a controlled Word16 study, input permutation improves raw-plurality accuracy by 21.1 points with essentially unchanged single-call correctness. Exact locking saves 28-30% of calls at a 16-call budget while preserving every fixed-budget output.

0.25

How Efficient Is Your Agent? Online Predictive Bounds under Budget Constraints

cs.LG ·2026-11-08 ·Roy Maor Lotan, Shai Feldman, Yaniv Romano
theory similarity 0.61
Abstract

We evaluate AI agents in an online setting where tasks arrive sequentially and, before running each task, we construct an upper predictive bound on its \emph{time-to-success}---the number of interactions until the task is completed successfully. Under a limited budget, some runs are stopped early, so their time-to-success is unobserved (right-censored), and the correctness of the bound may be unknown. We introduce \emph{\textbf{PR}edictive Bounds for \textbf{O}nline \textbf{B}udget \textbf{E}fficiency} (\texttt{PROBE}), a method for constructing calibrated upper predictive bounds while adaptively allocating a limited budget across agent tasks to reduce the anticipated coverage deficit. Our method calibrates the bounds online using inverse-probability-weighted feedback together with a correction designed to minimize coverage deficit while remaining unbiased. We prove that \texttt{PROBE} provides a coverage guarantee for any finite time horizon while satisfying the budget constraint. Experiments on software engineering, retrieval-augmented generation, and instruction following show that \texttt{PROBE} attains a coverage rate closer to the target level with tighter predictive bounds than a calibrated baseline.

0.10

Learning vs. Optimizing Players in Budgeted Games

cs.GT cs.LG ·2026-11-03 ·Giannis Fikioris, Balasubramanian Sivan, Éva Tardos
similarity 0.60
Abstract

We study repeated general games over $T$ rounds between a utility-maximizing optimizer and a learning agent, where budget constraints couple the players' decisions across rounds. Without budgets, low swap regret simultaneously guarantees the learner sublinear regret and caps the optimizer at its Stackelberg value. Budgets break this picture: adversarial no-regret becomes impossible, so the natural goals are non-manipulability---no adaptive optimizer earns more than $\mathtt{BSE} \cdot T + o(T)$, where $\mathtt{BSE}$ is its Budgeted Stackelberg Equilibrium value---and low stochastic regret when the optimizer plays a fixed mixed strategy. We give an efficient, budget-feasible learning algorithm that achieves both goals in every budgeted game: any adaptive optimizer, budgeted or not, earns at most $\mathtt{BSE} \cdot T + O(\sqrt{T})$ in expectation, and if the optimizer plays a fixed mixed strategy on each of a constant number of intervals, the learner has $O(\sqrt{T})$ regret on each such interval. Our learner best responds to forecasts of the optimizer's play that are calibrated in two novel senses, since under budgets the calibration that prevents manipulation no longer yields low stochastic regret. Response calibration, which conditions on the learner's own mixed response by embedding its continuous mixing weight as a step function in $L^2([-1,1])$, prevents manipulation without discretizing responses; dual-weighted Lagrangian calibration ensures low regret. Although low calibration error requires randomized forecasts---distributions over a continuous space that need not have a succinct representation or be directly computable---we construct such forecasts in polynomial time by simulating online gradient ascent.

0.38