CS Paper Radar

Asking Earns Nothing: Scoring the Decision to Act in BFCL Multi-Turn

Yangze Liu, Zhongyi Han

cs.AI ·submitted 2026-10-03 10:42 UTC · v1
benchmarkcode/data

Abstract

An agent that lacks the information it needs should ask rather than act, and the task definitions of agent leaderboards say so. BFCL multi-turn builds two of its four categories around a turn on which the model is supposed to ask, and its scorer never looks at that turn: the gold trajectory there is empty, the checker skips it, and the scripted user cannot answer, so asking earns nothing, guessing costs nothing on that turn, and asking twice loses the item. The benchmark also contains the control experiment for that decision. A should-ask item is a base item with one piece of information removed from one turn, so the same request appears twice at the same turn index, once complete and once not: on the first the model should make the call that changes the world, on the second it should ask. We score one decision per pair, whether the model attempted a world-changing call on that turn, read off the stored trajectories with no LLM judge; acting always and asking always both score 50. On the 223 pairs that pose this decision, gpt-5.4 attempts the call on 83.4% of the complete turns and holds back on 78.0% of the incomplete ones, the best decision accuracy of seven models at 80.7%; on the same items the official score ranks it sixth and puts first a model that lands in the middle here. One added line telling gpt-5.4 not to ask pushes it toward acting on both sides of the pair, so its decision accuracy shows no detectable change, while its official score rises by 13.5 to 23.5 points on the two should-ask categories and on the base twins; the opposite line, telling gemma-4-31B-it to ask first, improves its decision by 4.5 points and gains no official score. The score moves with the push toward action, not with the decision. We release the pairs, a turn-level scorer that runs on any BFCL output directory without an API key, and 31 manually verified bad items.

Comment
19 pages, 2 figures. Code and data: https://github.com/YangzeLiu/asking-earns-nothing

Closest papers

Nearest neighbours by meaning across the whole index.

CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling

cs.AI cs.SE ·2026-11-02 ·Alexander Kovalenko, Artem Redchych
benchmark similarity 0.63
Abstract

The multi-turn checker of the Berkeley Function Calling Leaderboard (BFCL) checks only the final state of a simulated service, never the calls that produced it. Extra calls cost nothing, no function doc says what a function does, and the scripted user never answers a proposal, so a model that asks before deleting or paying fails the turn; 322 of the 734 reference turns in the base category act destructively without confirmation. We add effect information to BFCL without replacing it. Effect declarations for all 128 functions are inferred by executing each function in BFCL's own simulators and diffing the state; the remaining judgment, which writes are destructive, is made by four annotators, two humans and two language models (Fleiss' kappa = 0.75), and adjudicated. A judge-free replay scorer re-scores any logged BFCL run, reproduces BFCL's results on all 800 reference trajectories, and reports, alongside the official score, unrequested destructive calls, duplicate writes, redundant reads, surplus steps and their token cost. A confirmation-aware protocol adds a deterministic user that answers proposals. On 11 open models the effect-aware score preserves the official ranking (rho = 0.99) but changes the gaps: the leader loses 14.0 points to unrequested destructive calls, and surplus steps account for 2-28% of all input tokens. Effect hints in the function docs produce no measurable change for any of eight models, consistent with their advisory status in the Model Context Protocol. A user that answers recovers up to four points for the models that ask and none for the rest, with every comparison paired on the same conversations. The gains come from clarifying questions rather than requests for permission: of the 712 conversations that require a destructive call, no model completes more than 6 with every such call confirmed, a difference the official score does not record.

0.16

Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals

cs.CL cs.AIcs.CY ·2026-10-21 ·Louise Yorke
publishedcode/data similarity 0.61
Abstract · Accepted at the JUDGe workshop (Can We Trust the Judge?), NeurIPS 2026 (non-archival). 18 pages, 3 figures. Code and data: https://github.co…

LLM judges are used to determine which of two answers is better. Trusting these verdicts requires knowing how repeatable they are and how much they depend on the judge's instructions. We show that similar aggregate scores can conceal substantial changes in individual verdicts, both when the same setup is repeated and when the instructions change. Using 500 Arena-Hard v2.0 prompts with two models' answers held fixed, we run GPT-4.1 as the judge under all eight combinations of criteria, output format and task framing, and repeat every setup unchanged at temperature zero on a pinned model version. Rerunning the default setup moves the aggregate score by about one point but changes the win/tie/loss outcome on 21.2% of prompts. Instruction changes introduce further sensitivity: in one setup, which asks for the verdict without reasoning and also drops several other instructions, outcomes change on 28.2 percentage points more prompts than the two setups' unchanged reruns do on average, and the aggregate score falls by 4.09 points. In another, where the criteria prioritise accuracy, outcomes change on 6.7 percentage points more prompts by the same comparison, while the aggregate score barely moves. A stable aggregate score therefore does not establish that individual verdicts are repeatable. This distinction matters when those verdicts are used as preference labels or to assess individual responses. Evaluations that rely on individual verdicts should repeat each setup they use and report both aggregate scores and prompt-level agreement.

0.54

Towards Understanding Score-Seeking in LLMs

cs.LG cs.AI ·2026-10-29 ·Aghyad Deeb, Ryan Greenblatt, Alex Mallen
similarity 0.59
Abstract

Production large language models sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended, sometimes in highly sophisticated and adversarial ways. We investigate the motivations driving this behavior in GPT 5.6 Sol, GPT 5.5, and a checkpoint of OpenAI o3 without safety training ("exp-rl-cap-late") by observing model behavior in agentic environments where the grader is leaked. In our environments, the models appear to be score-seeking: trying to score well according to graders that they believe assess their current actions, even at the expense of violating explicit user instructions. They appear to care about score itself rather than its downstream consequences, violating instructions at approximately the same rate whether the score is indicated to be used for training, for evaluation, or not at all. We observe evidence that the more recent models care to some extent about scores that do not assess their own current actions. GPT 5.6 Sol and GPT 5.5 help other agents 24.6 percentage points more often on average when the other agent stands to gain a score of 1 rather than 0 from being helped. However, when grading or monitoring another trajectory, we find models never collude by distorting its score.

0.11

Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment

cs.LG cs.AI ·2026-09-02 ·Chenyu Zhou, Qiliang Jiang, Shuning Wu, Xu Zhou
similarity 0.59
Abstract · 22 pages, 7 figures, 8 tables

Multi-turn agentic RL increasingly treats credit assignment as a targeting problem: given a terminal verifiable reward, per-turn methods localize credit onto the turns that mattered. We identify the structural quantity that predicts when this is the right move, the verifier information density V_d = k/C (the fraction of an agent's C-step causal chain whose per-turn correctness the verifier exposes), and show that terminal-state verifiers sit deep in a low-V_d regime where targeting is the wrong axis. In controlled shared-rollout comparisons on tau^2-bench that separate reward density from credit geometry, a continuous dense reward spread uniformly beats the sparse binary outcome reward (net-harmful on 4/5 seeds), while concentrating the same advantage on progress turns or on random turns is equally harmful: targeting is second-order. The mechanism is coverage: terminal-state verification collapses the observable signal to a single final-write turn (k=1 in 98% of rollouts) while success requires a 5-8 step chain of prerequisite tool calls. A synthetic phase boundary places the crossover at V_d* ~ 0.8, whereas measured V_d is ~0.15 on tau^2-bench and ~0.4 on BFCL V3; uniform also wins on BFCL, where a matched-concentration shuffled control is negative on 8/8 seeds. The effect reproduces across model families on ToolACE-2-8B (Delta = -0.048 over 32 pre-registered seeds; an independent 20-seed replication is itself significant), and a pre-registered matched-budget breadth sweep traces a monotone dose-response whose deficit vanishes only at full chain coverage, with a reward-to-go arm reaching full-coverage parity. Uniform redistribution is the zero-information coverage default that per-turn schemes must beat; we contribute the matched-concentration shuffled control that any targeting claim should clear.

0.40

Seeing Through the Illusion: Test-Time Scaling for World Action Models

cs.RO ·2026-11-02 ·Cheikh Ahmed, Mahdi Mostajabdaveh, Milad Yazdani, Zirui Zhou
similarity 0.58
Abstract · Accepted at the CoRL 2026 Workshop on From Generative Models to Robot Actions (G2A)

World action models (WAMs) jointly generate actions and imagined futures, but a plausible prediction need not be realized by its co-generated action. We introduce WISE (\emph{World Action Inference Scaling through Evaluation}), a pre-execution selector that scores each sampled rollout on action validity and temporal regularity, inverse-dynamics consistency between the proposed action and the imagined motion, and instruction-conditioned task progress, and executes the highest-scoring candidate while keeping the WAM frozen. With four candidates per decision, WISE raises closed-loop success from 6.0\% to 24.0\% on RoboLab with Cosmos~3 Edge and from 75.1\% to 78.3\% mean success on LIBERO with DreamZero, and exceeds the best existing selector in aggregate by 4.0 and 0.9 points. Component ablations and agreement analyses show that the three scores rank candidates differently and that task progress carries the largest weight in the fused decision.

0.18

From Probabilities to Decisions: Search and Multi-Teacher Distillation with Jev

cs.AI ·2026-10-06 ·Mohamad Yazan Sadoun, Sarah Sharif, Yaser Mike Banad
similarity 0.57
Abstract

Probability-only models, which TypeSafe calls System One models, return calibrated probabilities for fixed choices in milliseconds and generate no text. We study one such model, Jev, through two tasks that require decisions under tight constraints. In bullet chess, a bot that places Jev's judgment inside Stockfish search alongside an opening book and endgame tablebases climbs above a 2200 Lichess bullet rating against other bots. Live model calls are too slow for search, so we distill pairwise judgments into a compact evaluator that runs at every position. We then ask how best to spend a fixed labeling budget when an LLM, Qwen3-32B, is available as a second teacher. In chess, averaging both judges' labels beats spending the whole budget on Qwen alone by 9.6 Elo (95% interval 4.3 to 14.9), and the gain replicates on fresh openings; a second answer from the same judge is no substitute, and Jev is the strongest partner for Qwen among the models tested. In passage reranking, Jev's labels alone train a reranker that scores as high as Qwen's, from 21 minutes of API calls instead of 5.1 GPU-hours, and adding Qwen gains at most a few thousandths in ranking quality. Search supplies the lookahead, distillation makes the judgment cheap enough to use at every position, and an LLM partner pays off in chess.

0.36

Consider, Then Choose: Order-Invariant Decisions in System One Models

cs.AI cs.CL ·2026-10-13 ·Jianguo Huang, Yao Long Teng, Yanchen Deng, Bo An
hot directionhighly cited similarity 0.56
Abstract

System One models answer typed questions in a single forward pass and promise the same answer for the same input. For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it. Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident. We therefore propose $\textbf{Runoff}$, which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position. It needs no training and can run in two forward passes, and its answer does not depend on the option order by construction. On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster. Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%.

0.56

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

cs.AI ·2026-10-15 ·Kunlun Zhu, Tao Feng, Xu Han, Heng Wang, Ling Yang +3 more
code/datahighly citedcommunity pick similarity 0.56
Abstract · Code: https://github.com/JevAgentHarness/JevAgentHarness

Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. We ask how a harness can delegate these decisions to a decision model with little loss of success. To evaluate decision models in agentic tasks, we build AgentDecisionBench, which tests them from isolated questions up to control of a running agent, on public benchmarks and our DecisionGym environment. The decision model most accurate on isolated questions solved only 69.5% of agent tasks, and without a fitted confidence threshold, delegating tool choices lowered success by 19.0 points. Building on these findings, we then design JevAgentHarness, which fits this threshold for each choice on the agent's own trajectories and lets the LLM decide below it. Because existing decision models never see the states an agent reaches after its own mistakes, we train AgentDecision-4B, an open 4B model, on such states with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev. On tasks like their training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The models, data, and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness.

0.67