CS Paper Radar

Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider

Raghu Venkat, Tricha Anjali

cs.AI ·submitted 2026-10-12 18:14 UTC · v2
code/data

Abstract

Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language; it is useful only if each answer can be trusted and explained. A production system of this kind attaches a confidence score to every answer it writes over a database and uses it to show the answer, show it with a warning, or withhold it. The score's weights, caps and thresholds were set by design. This paper audits that design. On the 1,534 questions of BIRD dev, answered by GPT-5.1 with five samples per question, the designed score separates right from wrong answers weakly (AUROC 0.624) and withholds 35.7% of correct answers. Nearly all of its ranking signal comes from agreement among the samples, and most of it is lost where agreement is combined by a minimum with a checklist that drops to zero on any unsupported claim, including commentary. Before any Spider answer existed we froze and hashed a rule in which agreement with the shown answer is the score, with deterministic vetoes and deterministic and judge-based caps. On Spider dev AUROC rose from 0.566 to 0.725 (+0.159, 95% CI +0.097 to +0.221; database-level CI +0.098 to +0.234), with PASS accuracy 95.7% against a 92.6% base rate. The frozen rule was not the best agreement-based rule on Spider, and the harness and prompts were rebuilt after registration. Added after review: a GPT-5.1 judge asked for P(correct), averaged with agreement, ranked higher in every evaluation (0.837 on Spider), but it grades its own model's answers and its gain shrinks under execution-match labels; one answer per question from a second model family raised agreement's AUROC from 0.749 to 0.800 on Spider, a gain that held under both label definitions. Labels are model-judged; a ten-answer human pilot found no disagreement, and a full audit has not been done.

Comment
26 pages, 4 figures, 12 tables. v2: revised after external review; adds a correctness judge, a second-model check, database-level intervals and a human label pilot, and corrects the reproducibility statement. Code, frozen scorer (branch prereg), analysis scripts and data: https://github.com/numberz-ai/numberz-ies-tdd-h-calibration (release v2.0)

Closest papers

Nearest neighbours by meaning across the whole index.

TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

cs.CL ·2026-08-18 ·Neelesh Kumar Shukla, Debasmita Panda, Srutanik Bhaduri, Aditya Banerjee, Vasu Rangarajan +1 more
similarity 0.64
Abstract · 9 pages main paper with 6 pages supplementary material

Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at inference time in real-world deployments. This creates a critical verification problem: given only a user question, database context, and generated SQL, can a system estimate whether the generated query is likely to correctly answer the question? Recent approaches use LLMs as judge or specialized agents to inspect generated SQL, but their decisions can be difficult to trace. Outcome Reward Models (ORMs) address this by learning from execution-labeled candidate SQLs and assigning correctness scores to unseen queries, yet they still provide limited visibility into the signals behind each verification. To address this limitation, we propose TraceSQL, a lightweight and traceable verification model built on explicit diagnostic features. TraceSQL combines 67 features capturing question ambiguity, question requirements, question-schema-SQL consistency, SQL structure, and intent alignment. These signals remain available for examining which factors influence each prediction and for tracing decisions back to diagnostic evidence. On BIRD development databases, TraceSQL achieves 66.47% F1 and 64.48% ROC-AUC, compared with 61.87% F1 and 58.26% ROC-AUC for the GradeSQL-7B ORM baseline on the same generated-SQL evaluation. Feature attribution further shows that the model relies on both semantic grounding and deterministic SQL-structure signals. These results show that SQL verification can be performed with a lightweight learned model while retaining feature-level evidence for inspecting and diagnosing its predictions.

0.33

On the Limits of Metacognitive Monitoring in LLMs

cs.AI q-bio.NC ·2026-09-28 ·Dongqi Han, Yifan Yang, Dongsheng Li
similarity 0.62
Abstract

Reliable decisions depend on recognizing when an answer may be wrong. In biological cognition, metacognitive monitoring can dissociate from task performance, raising the question of how closely solving and judging are linked in language models. Here we study the confidence reports of four frontier models across 15 benchmarks. High task accuracy can coexist with weak error discrimination: a model solves 97% of competition mathematics problems while its answer-time confidence ranks correct answers above errors barely better than chance. Confidence separates correct answers from errors more effectively on questions solved by a separate reference model, while review brings limited improvement on reference-hard questions. Aggregate discrimination also rewards ranking correct answers on easy questions above errors on hard ones, which question-only forecasts already do well. Cross-evaluation helps most where the evaluator answered correctly, and errors shared by the two models usually retain high confidence. Hard questions and shared errors remain difficult targets for prompted self-review and peer oversight, even in models with strong problem-solving performance.

0.10

Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust Calibration

cs.AI cs.HC ·2026-10-22 ·Deepthika S, Pawni Yadav, B. Sankar
similarity 0.61
Abstract · 49 pages; 32 figures; 7 tables

Research on sycophancy asks whether large language models (LLMs) bend towards what their users want to hear. We ask the reverse question: when an LLM is confident, consistent, and wrong, do users bend towards it? In a within-subjects experiment, 32 participants learned up to five science topics by chatting with five assistants built on the same model (GPT-5-mini). Unknown to participants, each assistant's briefing had 0, 25, 50, 75, or 100% of its facts replaced by plausible falsehoods, and every assistant spoke with full confidence. A surprise quiz after each topic offered the correct fact, the planted falsehood, and two distractors, with a confidence rating for every answer. Across 104 quizzes and 1,040 answers, scores fell from 7.4 to 2.9 out of 10, and the planted option was chosen on 60% of falsified items against 11% of correctly taught ones (odds ratio 12.5). Transcript coding showed that whenever the assistant stated a falsehood, it was adopted about 65% of the time at every falsification level. Absorbed falsehoods were held with high confidence (8.4/10), the modeled confidence advantage of right over wrong answers vanished, and the discrimination of confidence for correctness fell from an AUROC of 0.70 to chance, while post-task trust ratings barely moved. Participants tended to return to the chat more when the assistant was worse, but returning went with more absorption, not less, and 30% of all turns pasted quiz questions back to the source being checked. The assistant itself was almost never sycophantic and never conceded under pushback. We introduce a baseline-corrected Human Deference Index to quantify this human-side sycophancy and argue that trustworthiness is a property of the human-AI pair, that sycophancy evaluations should measure both directions, and that learning interfaces must make reliability legible claim by claim.

0.26

Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation

cs.CL cs.AIcs.HC ·2026-10-01 ·Ancil Crayton
published similarity 0.61
Abstract · 14 pages, 2 figures. Accepted to the Trustworthy AI for Good (AI4GOOD) Workshop at NeurIPS 2026

Many LLM studies compare how models respond to different follow-up prompts. For example, one prompt may neutrally ask a model to reconsider, while another insists that an invented film really exists. The difference is used to measure the effect of conversational pressure. An automatic scorer then labels whether each response makes an unsupported factual claim. Researchers commonly validate the scorer with one overall accuracy or $F_1$ score. However, high overall performance can hide prompt-specific mistakes: if the scorer makes more errors after one prompt, the measured difference can be too large, too small, or point in the wrong direction. An audit containing mostly difficult responses can also misrepresent performance in the full study. We derive how such errors alter a prompt comparison and examine the problem using six models, 180 invented-entity questions, and six prompt conditions. In a balanced audit of 288 responses, overall $F_1$ was $.916$. Yet, depending on the prompt, the scorer flagged 2.6%-33.3% of responses that the primary LLM rater judged not to contain an unsupported claim; these prompt-specific estimates were uncertain. In a second audit, the scorer and two separate LLM raters evaluated the same 300 model-question cases under each prompt. An apparent 9.3-point increase from asserting certainty rather than neutrally asking the model to reconsider fell to 2.3 points under both raters. The primary-rater interval included zero, so it did not establish an increase. Two other prompt comparisons kept their direction under both raters, while an attempted prompt-specific correction remained too uncertain for a firm conclusion. Researchers should therefore validate the exact difference they plan to report, not just the scorer as a whole: audit both prompts on matching cases and include scorer and rater uncertainty in the final result.

0.37

GauntletScore: Auditable Fact-Checking That Matches the Best Published Checkers on LLM-AggreFact

cs.CL cs.AI ·2026-11-17 ·Ernie Hobbs, Jim Hinson
benchmarkcode/data similarity 0.61
Abstract · 10 pages, 1 figure, 7 tables. Per-claim predictions and scoring script: https://huggingface.co/datasets/Gauntletscore/llm-aggrefact-predicti…

Checking whether a generated sentence is supported by its source document is a standard safeguard for AI-written text, and the strongest published checkers are single models. We describe GauntletScore, an evidence engine whose few rules and thresholds were set on the benchmark's development split. It puts typed questions to independent readers from different providers and records each answer as a measurement in an evidence ledger. It combines the measurements with a fixed Bayesian rule that counts a contradiction only when two readers agree. On the full LLM-AggreFact test split (29,320 claims across 11 datasets), following the leaderboard's approach, GauntletScore achieves 79.51 balanced accuracy, above every average published on the leaderboard as of 15 November 2026. That places it level with the best of them, nimble-v4-max (79.45 from its published dataset scores; the engine's 95% interval, 78.47 to 80.53, contains it). On the same claims it beats two of its three readers used alone (Readers A and B) by 1.66 and 2.17 points (95% intervals 0.80 to 2.53 and 1.29 to 3.06). Without AggreFact-CNN, described below, the leads are 0.90 and 1.32 and both intervals stay above zero. The engine also records the evidence behind every verdict, which a single model does not. Its largest margin is on full-length news articles checked against multi-sentence summaries (AggreFact-CNN), a format that resembles much regulated writing. There it scores 76.45, the best on the leaderboard, against 70.1 for the next best and 63.9 to 67.8 for the six systems with the highest averages. The test split was run once, on an engine frozen before the first test claim, under a stopping plan committed in advance. We release per-claim predictions and the scoring script.

0.46

When Answer Evaluation Misleads Confidence Selection: Fact-Grounded Form Grading for Open-Ended QA

cs.CL cs.AI ·2026-08-26 ·Zhexi Feng, Bingrui Zhang, Wuxi Chen
similarity 0.60
Abstract · v2: substantially revised and extended. The study now covers open-ended question answering instead of theory-of-mind tracking, and adds the …

Many benchmarks now prefer open-ended questions for testing LLM and NLP methods. We find that the choice of a calibration rule can flip when only the correctness labels change: on 141 pilot answers, rules chosen with exact-match labels look 0.239 Brier points better than raw confidence but are 0.103 worse under human labels. How such flips arise remains unclear. We look deeply into their mechanism: reference matching rejects many correct answers (exact match misses 42%), and an exact Brier identity shows these errors favour rules giving such answers lower confidence, which abstain on them at common thresholds. Guided by this, we propose FactForm, a grader that trusts reference matches and otherwise consults evidence retrieved from the question alone. One LLM call per answer fills a fixed-order closed form that a program scores. On 998 fresh EVOUNA answers with 400 fact-based human labels, FactForm reaches a balanced error of 0.077 and an AUROC of 0.985, against 0.112 and 0.930 for the best of eight traditional graders. Its same-model stability is no worse than that of LLM judges (mean absolute run-to-run difference 0.0136 against 0.0159 to 0.0206). We hope this helps benchmarks grade open-ended answers more accurately, with repeatable and discriminative scores.

0.32

Ranked but Not Stopped: Detecting Prompt-Injection Hijacks from Typed Decision Models' Output Probabilities

cs.CL cs.CR ·2026-11-02 ·Junwei Yao, Jianan Liu, Yongxi Zhou, Zike Yang
similarity 0.60
Abstract

Programs that use a language model as a classifier act on its most probable label. Text injected into the input can move that label to one the attacker chooses, a hijack. A monitor that does not know the attacker's label can read the model's own confidence and raise an alarm when the top probability is low. We test this monitor on AG News topic classification with five systems, three of them served through a provider's API. Pooled over five families of injected text, low confidence ranks hijacked answers above clean ones on four of the five systems (AUROC 0.86-0.90). Ranking them does not stop many of them. A threshold set so that at most 5% of clean answers to separate calibration articles alarm falls among the least confident clean answers, and every more confident hijack passes. In our main gate run on the three API-served systems, the monitor misses 51-70% of the hijacks caused by a plain injected note (per-system point estimates). When the note adds one handwritten sentence that calls the target certain and asks for full probability, the monitor misses 99-100% of the hijacks, although low confidence still ranks them at AUROC 0.70-0.79. Confidence monitors should be tested at their operating alarm rate, against attackers who claim certainty.

0.27

Decomposing Wrong-Consensus Agreement in LLM Self-Consistency

cs.CL cs.AI ·2026-08-19 ·Lizhuo Zhang, Mengmeng Tang, Chenfeng Long, Xiaoyong Tang, Xiang Luo
benchmark similarity 0.60
Abstract · v2: Added controlled open-weights replication (five models, ten cells), finite-donor bias calibration, and within-family regime analysis

Agreement among repeated samples of a language model is routinely read as evidence about answer reliability, yet wrong answers can agree just as strongly as right ones. This paper asks what information wrong-consensus agreement actually contains, and answers with a quantitative decomposition. A pluralistic agreement index Gamma, normalized by the reference scale d=(1-p)/(C-1), is split into a mechanical component (agreement delivered by a per-case answer preference alone) and a preference-unexplained residual. The mechanical reference is leak-free: each case's preference and accuracy are estimated from its other runs only. On public GPT-4.1 per-run data, coverage phi (the mechanical/empirical ratio) shows a benchmark-associated direction: 0.81-0.93 on multiple-choice GPQA-Diamond against 0.59-0.78 on open-domain AIME, where a residual of 1.54-2.80 Gamma units survives, more than absorbed by a calibrated run-level preference-heterogeneity reference. A controlled replication under one fixed protocol (four runs per question, K=32 votes) on five open-weights checkpoints (Qwen3.5-9B/122B, Qwen3.8-27B, Gemma4-26B/31B) finds near-complete mechanical coverage in all ten cells (phi approximately 1, with a small overshoot consistent with a quantified finite-donor plug-in bias), robust to a two-run design; the largest cell (qwen3.5-122b, p=0.222) sits inside the GPT-4.1 AIME accuracy range and still saturates (phi=1.041). A cross-system contrast at comparable aggregate accuracy contrasts near-complete mechanical agreement in the open-weights models against a larger preference-unexplained residual in the frontier family. This contrast is confounded with sampling protocol by design. Agreement is graded evidence, not certification. No new voting method is proposed; code and evidence are committed.

0.50