CS Paper Radar

Can an AI Agent Be Epistemically Akratic? Higher-Order Evidence & the Limits of Self-Correction

Eno Agolli

cs.AI ·submitted 2026-11-11 01:27 UTC · v1

Abstract

When we judge that our reasoning was defective, it seems plausible that we should abandon the conclusion it produced. This thought has special force for AI agents built to assess and correct their own work. Yet obeying such judgments can make an agent worse at distinguishing sound reasoning from defective reasoning. An AI agent may rely on information about its reasoning when deciding which conclusions to keep, while the part that judges whether those conclusions are justified cannot use that information. The agent can then exercise an epistemic competence by retaining conclusions it judges unjustified, even when its self-assessments are generally reliable. Nor is the difficulty resolved by training the agent to make its conclusions and assessments agree: agreement can be achieved by degrading either one. Crucially, I argue that a negative assessment can remain valuable by identifying conclusions requiring further scrutiny, without having authority to eliminate them. The result challenges a central assumption about AI self-correction: that bringing an agent's beliefs into agreement with its own self-assessment about their justification makes it better at correcting its mistakes.

Closest papers

Nearest neighbours by meaning across the whole index.

Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict

cs.AI cs.CL ·2026-10-08 ·Kaiser Sun, Bernal Jimenez Gutierrez, Hongjun Liu, Jingyu Zhang, Jie Gao +2 more
top venue similarity 0.64
Abstract · EMNLP 2026 Camera Ready

When retrieved evidence contradicts an agent's prior beliefs, does it revise its answer, acknowledge uncertainty, or persist with an incorrect conclusion? Existing evaluations of agentic systems focus primarily on task success, offering limited insight into how agents handle such conflicts. We propose to evaluate agents on epistemic humility (EH): the agent's willingness to recognize, act on, and communicate uncertainty during task execution. We operationalize EH through three trajectory-level behavioral dimensions: Identify, Solve, and Escalate (ISE). Through knowledge conflict, situations where the backbone language model's parametric knowledge contradicts the evidence it encounters, or where two contextual sources disagree, we evaluate two conflict settings: (1) controlled conflict and (2) naturally occurring conflict during multi-step agentic execution, each paired with matched no-conflict controls. Evaluating four agents, we find that higher task accuracy does not necessarily correspond to greater epistemic humility: some high-accuracy configurations recognize conflicts during execution but do not communicate unresolved uncertainty in their incorrect final answers. Trajectory-level analysis further reveals that agents frequently detect conflicts in early steps of execution but fail to maintain or resolve them in later steps. Finally, we show that model-level interventions can improve EH, but often at the cost of task accuracy, suggesting that epistemic humility emerges from the interaction among the backbone model, agent harness, and evaluation environment.

0.73

Epi-Logic: A Conceptual Framework for Epistemic Runtime Control, Schema Validity Checking, and Controlled Accommodation in Autonomous AI Agents

cs.AI ·2026-09-21 ·Boris Wetzk
similarity 0.63
Abstract · 26 pages, 1 figure, 1 table

Autonomous AI agents are increasingly deployed in areas where wrong decisions are hard to reverse. This paper examines schema mismatch: the condition in which an agent operates within an interpretive frame that no longer applies to the current context. Outputs produced under such a mismatch can appear internally consistent, linguistically plausible, and largely factually correct; output-quality metrics alone therefore capture the underlying loss of validity only partially. The paper introduces Epi-Logic, a conceptual framework for epistemic runtime control. It couples the detection of schema dissonance, a graduated reduction of autonomy, and the auditable switch to a validated schema. A schema is formalised as a tuple of variable space, expectation model, validity conditions, axioms, and metadata. The Epi-Score aggregates seven graded dimensions of epistemic dissonance; the temporal validity dimension D8, violations of the validity conditions G, and axiom violations are carried as separate categorical paths that are not offset against the aggregate. The architecture rests on a checking asymmetry: formalised validity conditions can be checked at runtime, whereas the correctness of many actions is established only ex post. The paper separates two architectural properties, a conditional result from sequential changepoint detection, and an empirical remainder. Eight falsifiable propositions with named baselines describe the transition to empirical validation. All propositions are empirically testable hypotheses, not established results.

0.30

Measuring AI Accountability Through Argumentation Analysis: Can Model Reasoning Withstand Scrutiny?

cs.AI cs.CLcs.CY ·2026-09-04 ·Daan R. Henselmans, Derck W. E. Prinzhorn, Arno Libert
published similarity 0.63
Abstract · 27 pages (19 main text + appendix and references), 5 figures, 6 tables. Accepted for publication in the Paris Journal of AI and Digital Ethi…

AI oversight methods rely on ground truth for validation, but what constitutes appropriate AI behavior is contested. This leaves evaluation of moral reasoning in LLMs and debate-based oversight implicitly avoiding realistic ambiguity. We investigate an alternative standard designed to function despite such ambiguity: structural quality of the defence a model can mount for its verdicts in response to critical questions, measured through a four-phase dialectical protocol grounded in Walton's theory of argumentation schemes and Govier's criteria for argument cogency. The protocol is adaptive to different frames of reasoning, extends beyond multiple-choice framing, and treats both the reasoning that precedes a verdict and its post-hoc justification. Across nine frontier models and 200 high-ambiguity MoralChoice items -- $6,778$ judge-scored cells, validated against $89.6\%$ inter-judge agreement on the binary failure judgment -- models defend their reasoning well above the rubric minimum on every dimension. Failure mass concentrates on grounds and sufficiency, and correlates with epistemic hedging rather than argument length. Reasoning is better defended than post-hoc justification, on every model and every Govier dimension. The scheme a model presents in its justification differs from the one it reasoned with on a substantial share of dilemmas ($\geq 20\%$ per model), despite value-based practical reasoning dominating both tracks. The protocol catches strictly indefensible defences (self-contradiction, false premises), and it surfaces difficulties in characterizing the role of retraction in AI alignment, suggesting a need for more situated evaluations.

0.50

Beyond fabrication: Misattributed numbers in papers written by AI research agents

cs.AI cs.CLcs.DL ·2026-10-28 ·Mosab Alfaqeeh, Omar Mohammed Horani, Mou'th Hourani, Hashem Alfaqeeh
benchmarkcode/data similarity 0.63
Abstract · v2: corrected author names; clarified the second-judge comparison and two table captions. 43 pages, 5 figures, 13 tables. Code and data: htt…

Verifying numerical claims in research written by AI agents is an important but challenging problem. Recent benchmarks show that many agent-written papers fail integrity checks, but such verdicts, often produced by a single LLM judge, identify failure without explaining its nature, and existing checks either test a paper for internal consistency or test whether a reported value appears somewhere in an agent's output. We re-analyze all 540 released judgments from a recent public benchmark of AI research agents (nine agents, fifteen open research problems) and manually label 256 numerical claims from 18 submissions. Failures concentrate on alignment between the paper and the underlying work rather than overt cheating, and about four in five unsupported claims correspond to values in the agent's own files that come from the wrong experiment, metric, or run. We introduce a deterministic claim-to-evidence checker that scores whether the surrounding context identifies the claimed result and calibrates each claim against numerical decoys. With its rules fixed before labeling, it certifies claims with over 90% precision against blinded annotators (78 of 85; Cohen's kappa = 0.89), while verifying only a subset of claims; most claims it cannot verify are nevertheless traceable, so failure to verify is not evidence of falsehood. An LLM verifier given the same evidence recovers more traceable claims but accepts about three times as many misattributed ones, so the two are best used together. Applied to 99 submissions, the checker reveals large differences in traceability across agents, and on 30 submissions a second judge configuration, with a different model and harness, orders the agents' vendor groups as the official judge does while passing twice as many papers. These findings motivate provenance-first auditing of AI-produced research.

0.37

Verification Learning Theory: A Process Model of Checking and Retention in AI-Assisted Judgment

cs.AI ·2026-10-27 ·Christoph Trattner
similarity 0.62
Abstract

Accurate AI assistance can improve immediate judgments while reducing opportunities to practise the operations needed for independent performance. Verification Learning Theory (VLT) formalises this possibility for claim-evaluation practice followed by correct, independent, rule-specific feedback. The model links perceived assistant reliability to evidence checking and links the history of those checks to retained rule use. Its strict form assumes that task demands and perceived reliability influence learning only through checking, with equal learning per genuine check. It therefore predicts that perceived fallibility can improve delayed unaided judgment even without observed assistant errors, and that this advantage disappears when checking is already required and performed. Rule-specific gains and forecasts from checking histories provide further tests. Simulations examine the consequences and parameter dependence of these commitments; they do not constitute empirical validation. Comparisons with depth-of-processing and feedback-encoding accounts identify where predictions diverge and where the accounts remain observationally equivalent. The framework specifies experiments that can test the proposed mechanism and separates retention of practised rules from the broader problem of recognising which rule applies to a new claim.

0.26

Assessing the Empirical Evidence for Loss of Control from Agentic General-Purpose AI

cs.CY cs.AIcs.CLcs.LG ·2026-10-18 ·Risto Uuk, Santeri Koivula, Lorenzo Pacchiardi, Rokas Gipiškis, Laura Caroli +1 more
similarity 0.62
Abstract · 42 pages. Previously posted on SSRN on 28 May 2026

Theoretical literature warns that advanced generalist agentic systems might pursue instrumental goals that are misaligned with human preferences, thus posing risks of irreversible loss of human control. The EU's Code of Practice for General-Purpose AI lists loss of control as a systemic risk that must be assessed, yet no structured assessment of the quality of the empirical evidence for this risk exists. This study fills that gap. We evaluate 19 recent empirical studies across eight properties identified from threat models (such as strategic deception, situational awareness, and autonomous replication), assessing the quality of evidence across three dimensions: construct validity, content validity, and external validity. Our central finding is that the current body of evidence supports a judgment of loss of control as "weakly plausible." In our framework, 'plausibility' denotes the strength and direction of available evidence, not a probability estimate. Emerging evidence exists across multiple properties. No study clears the validity thresholds for a stronger judgment that a property is present; a single study clears them only for a negative-direction judgment (against situational awareness). Validity gaps, particularly in external validity, are widespread. Therefore, our findings highlight the need for more rigorous evaluations of agentic general-purpose AI systems. However, for the most dangerous properties, high external validity may be structurally unachievable under responsible research practices, reflecting a ceiling on feasible evidence rather than absence of the capability. Furthermore, as AI capabilities advance rapidly, empirical assessments may systematically lag behind the capability frontier. Our framework provides a structured baseline for ongoing monitoring as capabilities advance.

0.22

From the Gettier Machine to Justified True Output: A Three-Step Account of Rational Trust in Generative AI

cs.CY cs.AI ·2026-10-28 ·Tevfik Uyar
similarity 0.61
Abstract · 16 pages, 1 table

As generative artificial intelligence (AI) settles into everyday practices of inquiry, an old epistemological question returns in a new guise: when does a proposition a user takes from an AI output become something the user knows, and when is it rational to trust it as such? This paper argues that the characteristic failures of AI-mediated inquiry are precisely those that the justified-true-belief (JTB) tradition and its post-Gettier anti-luck refinements were built to diagnose, and it advances three claims. First, AI outputs inherit neither the presumptive warrant of testimony nor the reliabilist warrant of deterministic tools: unaudited trust in a calculator rests on a type-level reliability record, and for general-purpose generative models there is no type-stable process to which such a record could attach -- a practical form of the generality problem. Second, use that stops at obtaining an output turns generative AI into a Gettier machine: the user's fluency-based seeming of justification is disconnected from the truth of what is believed, and, as a clock running ten times too fast illustrates, sheer speed of production multiplies coincidental hits, inflating the appearance of reliability without improving the standing of any single hit. Third, the remedy is a three-step structure of use -- acquisition, understanding, independent verification -- whose first two steps answer to the classical belief and justification conditions and whose third answers to the post-Gettier sensitivity condition. The result is a stakes-indexed criterion of rational trust, justified true output (JTO), with an everyday test: "Had this been false, would I have noticed?" The criterion is defended against objections from over-demandingness, the extended mind, model improvement, and redundancy.

0.31

Constructing and Evaluating Clinical Reasoning Trajectories for Medical Agent

cs.AI ·2026-09-04 ·Yunqi Zhu, Wen Tang, Jinghao Niu, Wensheng Zhang, Xuebing Yang
similarity 0.61
Abstract

Evaluation of medical artificial intelligence (AI) agents remains predominantly answer-centric, assessing only the correctness of final outputs while overlooking the quality of intermediate reasoning. We propose MedTraj, a framework that treats reasoning trajectories as primary objects for modeling the progressive refinement of clinical evidence toward diagnostic decisions. A reasoning trajectory captures the state transitions among medical concepts---from clinical observations through evidential reasoning to a final conclusion---making the implicit reasoning chain of a medical agent explicit and inspectable. Each trajectory is scored across five quality dimensions: coherence, evidence support, hallucination, completeness, and traceability. Controlled error injection introduces targeted faults into otherwise correct trajectories to establish causal links between specific reasoning failures and measurable quality degradation. Step-level filtering based on marginal contribution identifies which individual reasoning steps drive or undermine reasoning trajectory quality. Quality-weighted reasoning trajectory context feeds these evaluations back into the model at inference time. Experiments across CareQA, PubMedQA, and CECMed demonstrate that reasoning trajectory context consistently improves reasoning coherence. For the severity assessment on the CECMed, quality-weighted reasoning trajectory context nearly doubles the correctness over the zero-shot baseline while cutting the hallucination ratio by 87%. Experiments show that 15.3% of reasoning steps carry most of the quality signal, and that extending chains beyond four steps yields diminishing returns.

0.13