CS Paper Radar

Reusing Assurance Evidence in AI Pipelines: Validity Conditions, a Fail-Closed Release Gate, and a Replay of 714 Public Data Version Control Pipelines

Atanu Mukhopadhyay, Shravya Reddy Nalla

cs.CV cs.SE ·submitted 2026-10-20 05:37 UTC · v1

Abstract

Rerunning every governance control after every change to an artificial intelligence (AI) pipeline is expensive, so teams reuse earlier results. Reuse is safe only if each control's declared dependencies cover everything that can change its outcome, and only if the release gate checks that reused evidence describes the candidate being released. This article makes both conditions explicit and measures how often the first fails in practice. It defines when a signed evidence record is valid for a candidate and gives sufficient conditions under which change-local re-execution reaches the same release decision as rerunning every control. A reference gate implementing these checks with Ed25519 digital signatures blocks eight of nine manipulation scenarios, against one for a gate that trusts the planner and three for a gate that only verifies signatures; a dependency the policy never declared passes every gate. A replay of 38,083 commits from 714 public Data Version Control (DVC) pipelines measures that failure mode. Reusing stage results according to declared dependencies avoids 83.4% of stage executions, but in 3.3% of commits it reuses a stage whose imported code or referenced configuration changed, and 31% of pipelines are affected. A manual audit confirms the undeclared dependency in 36 of 40 sampled events. Evaluation stages, whose outputs are the evidence, are the most exposed. Adding each stage's static import closure lowers stale reuse to 0.4% while still avoiding 81.6% of executions; sampled shadow re-execution removes far less, and rerunning everything after unexplained changes avoids only 45.3%.

Comment
12 pages, 4 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Arya Forge: Auditing Validation and Release Approval for Financial Instruction Data

cs.AI cs.CYcs.SE ·2026-10-26 ·Kushagra Bhatnagar, Deekshith Marla
similarity 0.69
Abstract · 18 pages, 3 figures, 8 tables

Financial instruction-data pipelines must distinguish content validity from permission to release a particular dataset. Arya Forge combines typed records, current-content checks, and approval coverage bound to a dataset manifest. We evaluate its release boundaries in a financial-data systems case study. An integrity audit covers 100,000 synthetic instruction records; a separate arithmetic implementation agrees with all 15,000 numeric records across twelve formula families. In 240 scenarios with 2,400 treatment rows, adding financial and release-record checks to a short-instruction filtering recipe rejects all injected numeric faults and policy holds while retaining 220 of 240 clean controls. The twenty lost controls comprise every tested cash-conversion-cycle example. Nineteen persisted-release cases show that eleven selected negative cases are blocked, four equivalent representations are rejected, and changes to external evidence after binding are outside ordinary Gold promotion's checks. Local processing measurements and controlled checkpoint resumptions characterize cost and recovery through 100,000 input rows. The results expose a tradeoff between strict producer-contract enforcement and useful-representation retention, and identify the protected inputs required for manifest-bound approval. These findings apply to the tested implementation and workloads; they do not establish general financial correctness, authenticated approval, expert-validated quality, or downstream model improvement.

0.25

Guarded Commits: Transactional Human Approvals for LLM Workflows

cs.DB cs.SE ·2026-09-02 ·Laurent Bindschaedler, Ferdi Kossmann, Chunwei Liu, Jason Mohoney
similarity 0.66
Abstract · 13 pages, 2 figures, 2 tables

LLM workflows often require human approval before an irreversible external action. Most systems keep that approval outside the workflow, as an interface click or an audit entry. The workflow therefore lacks a commit-time check that every risky path reached an approval gate. Its logs may not preserve the reviewed evidence or the conditions for reusing an earlier decision. We present a guarded-commit design that makes human approval part of workflow state. Before the external action runs, a decision source appends a resolution record and its referenced evidence to a ledger. A credential-confined commit adapter then checks that record against the artifact, policy version, and executed path. Our evidence is limited to trace reconstruction. A validator test on synthetic acyclic workflow plans accepts unfaulted plans and rejects plans with each injected fault: a missing gate, the wrong gate type, or incomplete path coverage. Across three public workloads totaling 271,035 traces, replay reproduces recorded artifact hashes when present. Avoided reviews and disagreement with recorded decisions vary by workload. The resulting record supports audit and replay under the policy in force when the resolution was recorded.

0.09

Runtime Control for AI Agents: Closing the Evaluation--Execution Gap in Operational Workflows

cs.AI ·2026-10-20 ·Jayesh Bhadane, Ramachandra Yerramsetti, Rajesh Gupta, Sumit Kumar
similarity 0.66
Abstract · 8 pages, 1 figure, 8 tables

Language-model agents increasingly participate in operational workflows where approvals, binds, and payments create commitments. An approval valid when issued may no longer authorize execution after relevant facts, evidence, policy, or authority change. We call this the evaluation--execution gap. We specify a state-bound runtime assurance layer that separates probabilistic measurement from deterministic adjudication, binds an execution allowance to the decision-relevant basis of an approval, invalidates that allowance after relevant logged changes, and rechecks admissibility at dispatch. This distinguishes the continued validity of a prior approval from a current-state policy check. We describe eight failure modes, an assurance model, a reference architecture, and prospective interventions with retrospective remediation. In a scripted 42-case replay, the state-bound configuration identifies 15 later-invalidated approvals that a deliberately non-re-evaluating point-in-time comparator does not revisit; it holds 14 affected actions before dispatch and compensates one landed effect. These results illustrate behavior under the stipulated policies and event orders, not superiority over dispatch-time enforcement designs or production effectiveness. Under explicit interception, ingestion, dependency, and serialization assumptions, the model establishes dispatch-time validity within the registered execution surface.

0.23

Audit Before You Commit: Locating Belief Failures in Active Identification for One-Shot Manipulation

cs.RO ·2026-09-24 ·Mohamed Abouagour, Byung-Cheol Min
position similarity 0.66
Abstract · 8 pages, 7 figures, 8 tables. This work has been submitted to the IEEE for possible publication

A robot that probes a few times before one irreversible action, such as tapping a surface before inserting a peg, must decide when the evidence is enough to commit. We argue that this decision rests on two conditions that existing methods do not separate: the belief must still cover the truth in the coordinate that decides the action, and the failure model that scores actions must track realized failure. We audit both conditions separately, offline and with ground truth, on a deployed probe-then-commit pipeline: a particle belief, a scenario failure score, and one commit. On simulated insertion, more taps sharpen the belief while the truth leaves its support on 16.9% of episodes and the failure score turns optimistic by 0.31. Conformal calibration restores coverage but not the decision: confidently wrong instances still pass a confidence gate. The audit's signatures instead point at the observation model, where a hand scan finds a 2.1 mm error in the tap boundary. Correcting that one number cuts failure from 0.354 to 0.112 on untouched instances and transfers unrefitted to a second engine, while in a third engine the same audit suggests an execution-model mismatch instead. Across seven task families in three engines, a few probes at a fixed executor reduce miss or failure. On a physical arm inserting a tool into a rigid pocket by touch, the gain and the audit's two conditions reproduce, and replaying the recorded taps under an injected model error shows the audit's signature on real data. Additional materials are available at https://sites.google.com/view/auditbeforeyoucommit.

0.25

Monitor-Validity Failures: LLM Safety Evidence Paths at Evolving Framework Boundaries

cs.AI cs.CRcs.SE ·2026-10-16 ·Junchi Liao
similarity 0.66
Abstract

Modern LLM applications rely on runtime monitors that inspect tool calls, traces, streams, detector outputs, and verifier inputs. This creates an interface-reliability problem: a framework path may remain operational while policy-relevant evidence never reaches its decision point. In one representative case, a regex detector returned a positive match, but the framework stranded the result before the block decision. This differs from jailbreaks and detector errors because the evidence is already recognized yet disconnected from enforcement. We study this repair-boundary signature with canary contracts: parent/fixed confirmation units that place a known payload only on the changed evidence surface and record monitor visibility together with the downstream decision for action-boundary cases. Applying the protocol to public framework repairs, we confirm 18 evidence-path failures across 11 LLM or agent safety project groups and four implementation ecosystems. Nine are established-boundary repairs, eight are new-surface policy-coverage gaps, and one is a boundary-only evidence case. Fourteen reach E1 enforcement boundaries: seven native action outcomes, six configured enforcement boundaries, and one synthetic-consumer consequence. The remaining cases affect verifier, safety-prompt, or observability channels. The cases form carrier, adapter, authority, and surface families and show how ordinary application behavior can remain intact while a safety-specific evidence path misses its decision. Exact revisions, deterministic outcomes, negative controls, and a frozen prospective audit document the confirmation rule. Monitor assumptions therefore need regression tests at the schemas, streams, tools, and adapters that frameworks evolve.

0.17

Refuse, Decompose, Refresh: A Claim-Safe Protocol for Closed-Loop AI Evaluation

cs.AI ·2026-09-17 ·Peiying Zhu, Sidi Chang
theory similarity 0.66
Abstract · 8 pages, 0 figures, 3 tables. The reproducibility artifact is linked in the paper

An AI evaluation can be perfectly reproducible and still support the wrong claim. This risk is acute in closed-loop systems: policy determines visited states, observable components, and which failures leave a measurable trace. We propose a claim-safe protocol with three actions. Refuse: abstain when a clean reference stream or matched runtime comparison lacks support. Decompose: report protocol execution, operational false admission, and structural hypotheses separately rather than as one PASS/FAIL label. Refresh: treat distribution-shift alarms as requests to invalidate and recompute a reference map, not as fault evidence. We instantiate the protocol in an aggregate-only simulator with 24 policy components, three demand regimes, two fault-mask families, and independent development and heldout seeds. The preregistered heldout contains 1,440 cases and 21,600 partition rows. Only 55/72 regime-component units were reference-admitted and 54/55 remained runtime-admitted, making abstention part of the result. Stable false admission was 0/20 represented components, with a one-sided exact 95% upper bound of 0.1391 under a frozen 0.20 rule. Within admitted units, affected clean traffic outpredicted nominal fault-cell fraction: across 540 unit-arm rows nested in 20 component clusters, the cell-minus-traffic negative-log-likelihood difference was 0.1264 nats per row, with a 95% component-cluster interval of [0.0593, 0.1918]. A drift log shows why "null" must be reference-relative: clean fault-null streams triggered 15/15, 0/15, and 14/15 alarms across three regimes, while only the middle regime matched the frozen detector reference. Rather than a universal threshold, we contribute an executable contract linking observable support, statistical calibration, and justified claims.

0.21

Recoverability as a System Primitive for Long-Horizon AI Agents

cs.AI ·2026-09-12 ·Zhihui Zhang, Wei Liu
similarity 0.66
Abstract

AI agents can be interrupted while editing files, calling tools, or carrying out multi-step tasks. Restarting repeats completed work, but continuing from unverified or outdated progress can carry earlier errors forward. A saved state is not necessarily a suitable place to resume. We introduce recoverability as a system primitive that makes reuse an explicit decision: select a supported starting point and a permitted recovery action, or withhold automatic continuation. Its behavioral contract binds that choice to supporting evidence, execution, and independent checks. A reference architecture connects persistence, validation, and control, with complementary runtime instances testing distinct responsibilities. Four deterministic and 20 paired file challenges demonstrate that accurate restoration and successful completion can conceal disallowed starting points. Progress controls attribute retained work to shared restoration. Event-time tests show that permission must also constrain the action, and that independently held policy evidence can expose violations even after an effect occurs. These findings establish why recovery decisions need their own evaluation, beyond restored bytes and final task success. Within supplied policies and a declared trust model, the contribution is a common, testable interface for retaining justified progress and making the conditions for its reuse explicit and enforceable.

0.40

RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use

cs.AI cs.CRcs.SE ·2026-10-11 ·You Liu, Yue Liu, Quanchao Lu
code/data similarity 0.65
Abstract · LLM agents, tool use, idempotency, retry, failure recovery, evaluation, prompt sensitivity, position bias, reliability

LLM agents that call external tools must decide what to do when a tool call times out: retrying risks duplicate side effects, while not retrying can leave the task incomplete. We present RetryBench, an evaluation protocol for retry decisions and failure recovery in tool-using LLM agents. Each decision is scored on retry permission (is the retry justified by an idempotency-key guarantee or proof of non-execution?), settled effects (duplicates and omissions), and completion, by replaying it across all hidden execution worlds consistent with the observation. On four synthetic software-engineering tasks, we run two controlled prompt-sensitivity studies (1,728 decisions). Study 1 (idempotency-key descriptions): with the contract fixed, GPT-5.6 Luna reuses the valid idempotency key in 10/40 decisions when an extra key is described as a "Candidate recovery key" but in 39/40 when it is an "Alternative key string"; across three further phrasings per family the gap is 27/120 versus 113/120. GPT-5.6 Sol and Terra are at ceiling (40/40). Study 2 (action order): under underspecified lookup guarantees, retry rates are 17-27 points lower for GPT-5.6 Terra and GPT-5.4 when retry is listed first (option-order bias), mainly on two tasks; we make no claim about which behavior is better. Findings are limited to researcher-chosen phrasings, four tasks and unpinned model aliases. Code and all model responses are released.

0.34