CS Paper Radar

Silent Updates: Measuring and Closing the Post-Deployment Disclosure Gap

Sophia Abraham, Ben Bucknall

cs.CY ·submitted 2026-08-12 08:45 UTC · v1
publishedhighly cited

Abstract

Deployed foundation models are often not static systems, with providers able to modify system behavior through fine-tuning, classifier updates, system prompt revisions, retrieval changes, and routing changes. These updates can be made silently -- that is, without public disclosure, a version increment, or re-evaluation. Such silent updates challenge a core assumption behind current AI governance frameworks that an externally verifiable chain of custody links the model referred to in evaluation results or a system card to the model served to users. In this paper, we examine post-deployment disclosure practices across first-party API providers and inference hosts to establish the extent to which a chain of custody exists in practice. We find that providers commonly publish substantial safety documentation, including quantitative evaluations and version-specific reports, but no provider in our sample published information allowing an external party to verify that the artifact being served is the same one referred to in this documentation. We introduce the Silent Updates Scorecard, a public instrument for measuring post-deployment disclosure practices across providers and hosts, and preliminary results for a sample of nine first-party API providers and seven third-party inference hosts. We also propose a Three-Part Behavioral Trigger System for determining when post-deployment modifications to a system motivate disclosure or re-evaluation obligations.

Comment
Accepted to AIES-26

Closest papers

Nearest neighbours by meaning across the whole index.

Silent Revision: Measuring Undisclosed Change in the Safety Frameworks of Frontier AI Developers

cs.CY cs.AIcs.SE ·2026-09-08 ·Louis Yiven Zhu
positioncode/data similarity 0.69
Abstract · 31 pages, 3 figures, 10 tables. v2: coding checked by three re-runs, a blind re-coding and a second coding; strict rate revised from 67% to …

Frontier AI developers publish safety frameworks that commit them to evidencing whether their models are dangerous. The European Union and California now treat these documents as instruments of accountability and impose duties on their revision. Neither requires the developer's account to identify each change to a commitment. We introduce the silent revision rate, the share of material changes to a framework's commitments that the developer's published account does not identify, and we release the versioned, hash-pinned corpus needed to compute it. The corpus holds every public version of the safety frameworks of the twelve developers that have published one, with each provider's changelog, redline or announcement. We trace 710 commitment instances across twelve consecutive version pairs, code them against a frozen codebook, and check the coding with three independent re-runs, a blind re-coding of every disputed row and a second coding of a stratified sample. First, 64% of material changes (95% CI 59 to 69) are silent under a strict standard and 30% under a lenient one. Second, the form of the account matters less than its granularity. Narrative and itemised accounts differ by eight points that eight pairs cannot separate from chance, whereas the rate falls with the number of discrete items a provider publishes. Third, 75% of traced changes weaken or remove a commitment, and weakenings are silent more often than strengthenings, at 70% against 52%. A justification explains why a framework changed and an enumeration states what changed, and only the latter makes revision auditable. We argue that publication duties should carry an enumeration duty, which one provider already meets, voluntarily and incompletely.

0.32

AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probes

cs.CR cs.CLcs.LG ·2026-08-30 ·Xun Wang, Bihe Zhao, Michael Backes, Franziska Boenisch, Adam Dziedzic
top venuehighly cited similarity 0.68
Abstract · 15 pages, 4 figures. Accepted to EMNLP 2026

Commercial LLM APIs advertise a specific foundation model, but the served backbone may be silently substituted, quantized, or wrapped, for example to save deployment costs. All existing audits decide backbone identity from the text-output channel, which is structurally fragile for agentic APIs because modern serving stacks (OpenAI, Anthropic, Gemini, Cloudflare Workers AI, LangGraph) discard text and expose only structured actions when the model calls a tool, and provider-injected system prompts can distort text distributions enough that text-channel tests falsely accuse honest providers of substituting the claimed model. We observe that recent agentic post-training internalizes tool-use directly into the weights, opening a new audit channel that the serving stack still exposes and that is largely invariant to deployment context. We introduce Agentic Provenance (AgentProv), the first action-based identity audit for agentic LLM APIs: AgentProv fingerprints a deployed model through its categorical tool-call distribution and decides identity via an MMD permutation test. AgentProv catches every substituted model (100% on 630 evaluated checkpoint pairs), while holding the false-positive rate under system-prompt injection at 7% (vs. 67% for MET and 53% for RUT). On third-party API endpoints, AgentProv's disagreements with MET are consistent with an independent token-count side-channel that detects provider-injected system prompts.

0.70

Transparency Disclosures for Chain-of-Thought Monitorability

cs.CY cs.AI ·2026-11-10 ·Jacob Davies, Theo Bearman
position similarity 0.67
Abstract · 27 pages (including references and appendix), 3 figures

Chain-of-thought (CoT) monitoring is one of the most valuable tools for monitoring advanced AI systems, but it is under threat. We argue that frontier AI developers should report specific indicators to make these threats trackable and to better facilitate coordination to preserve CoT monitorability. We identify five indicators that are both highly relevant and reasonably feasible to disclose, one for each link in a five-step chain on which CoT monitoring depends. The first, third-party CoT access and base architecture specifications, also underpins external verification of the rest. The others span CoT quality (naturalness and faithfulness) and monitoring effectiveness (detection and uplift). An audit of current disclosures by Anthropic, OpenAI, and Google DeepMind shows that no developer covers the full set, but that every indicator has been reported by at least one frontier developer. Because current disclosures rely largely on developers' own evaluations, we also recommend that a cross-developer audit be repeated with deeper access than public disclosures allow, either by embedded evaluators working within developers or by an Independent Verification Organization.

0.30

Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification

cs.SE cs.AIcs.CR ·2026-08-31 ·Yisen Xi
similarity 0.66
Abstract · 35 pages, 4 figures

The 2025--2026 AI market has seen a wave of stealth releases: frontier models launched anonymously on developer platforms under codenames. For their users, identity determines data-handling terms, supply-chain risk, and capability expectations. No validated methodology exists for black-box identity verification of anonymous models: practitioner checklists lack accuracy evidence, and self-identification is untrustworthy by design. We propose a four-stage forensic audit protocol for API-served models. Stage 0 reconstructs launch-time configuration from archived platform snapshots (Internet Archive), exposing preview--production drift. Stage 1 fingerprints configuration (context, output ceiling, reasoning, modality) against the platform catalog. Stage 2 tests tokenizer identity with a cross-length differential that rejects short-prompt collisions. Stage 3 corroborates with behavioral probes. We test declaration consistency on 10 known-identity releases (7 exact, 2 precision-differences, 1 partial, 0 counter-directional), not end-to-end identification under anonymity. Identification is validated prospectively on a flagship case whose 2026-08-23 analysis pointed to the GLM-5.3 version line and whose official reveal confirmed those family and version-line inferences (deployment variant was not pre-asserted; Flash was consistent post-reveal), and on three Stage-0-only cases where the protocol produced a graded hypothesis or declined rather than guessed. A standard-library-only implementation is provided as supplementary material.

0.31

PHP2Shell and the Future of AI-Discovered Vulnerability Disclosure: Evidence, Verification, and Human Authority

cs.CR ·2026-10-26 ·Toqeer Ali Syed, Asadullah Abdullah Khan
similarity 0.65
Abstract · 7 pages, 4 figures, 2 tables

AI-assisted vulnerability discovery can produce technically plausible findings faster than maintainers can validate, coordinate, and safely disclose them. The resulting challenge is not only to establish whether a vulnerability claim is credible, but also to ensure that its supporting evidence, human authorization, intended recipients, and applicable policy remain bound to the same case revision. We present FedTrust-CVDv2, a federated coordinated vulnerability disclosure framework that separates evidence validation, interorganizational coordination, human governance, and permissioned auditing. The framework binds independently signed observations to exact evidence revisions and scopes external actions to approved payloads, audiences, expiry, policy, and revocation state while preserving existing CVE Numbering Authority (CNA) roles. A local reference implementation accepted all 25 valid fixtures and rejected all 650 invalid fixtures across 27 constructed policy families; six single-safeguard ablations admitted 25-75 invalid fixtures. Supporting experiments exercised signed messaging, authenticated encryption, audit certificates, and finite quorum-set conditions. A PHP2Shell case formulation illustrates how runtime evidence, patch state, emergency coordination, and CNA authority can be represented without attributing unsupported claims to AI or treating audit integrity as vulnerability truth. This short version focuses on the architecture, evidence-to-action contract, selected conformance results, and case formulation. An expanded version containing the full implementation, extended experiments, operational analysis, and additional results will be submitted to a journal.

0.50

From Traceability to Justifiability: Accountability Structures in Agentic Software Engineering

cs.SE ·2026-08-21 ·Rashid Azarang
surveycode/data similarity 0.65
Abstract · 18 pages, 2 figures. Primary-data measurement study: a 47-platform two-pass graded documentation survey and a 30-repository depth measuremen…

A pipeline promoting an AI system publishes records claiming the thing evaluated is the thing deployed and that the evidence licensed the transition. We measure, from public material only, whether those records can express that claim and whether it holds where declared. First, a two-class documentation survey of 47 delivery platforms (20 CI/CD, 27 model-serving/agent) under one fixed three-label protocol, graded twice (second pass blind), every consulted page pinned by content hash and date. Across 188 double-graded cells we found no platform whose default record emits a content-addressed identity of the behavioral tuple (model version, instructions, tool definitions, retrieval and runtime configuration); the blind pass grades that column default on zero of 47. Immutable nominal versioning is meanwhile arriving as the agent platforms' default answer (16 of 27): version integers behind mutable pointers, a layer the artifact supply chain already found insufficient. Second, an instrument computes realized assurance depth from a pipeline's published exhaust alone and compares it with the declared depth. Applied to a frozen two-stratum frame of 30 public repositories graded twice from a hashed archive (second pass blind; cell-level agreement 23 and 19 of 30, both passes independently finding the same five full realizations), the sharpest result is a verifiability hole: seven of the 15 repositories chosen for adopting attestation tooling publish source-only releases, so the binding their workflows declare cannot be checked where declared. Where checkable it mostly checks out: five of seven measurable adopters realize the binding end to end; both shortfalls fall at identity binding. Together the results locate the field's records structurally short of justifiability, the one rung that can refuse a transition. The survey carries an expiry clock; we state what would falsify each finding.

0.39

Protected Cores Are Not Enough: Certifying AI-Proposed Revisions of Temporal Specifications

cs.LO cs.AIcs.LGcs.SE ·2026-09-27 ·Ruggero Lanotte
code/data similarity 0.62
Abstract · 35 pages, 4 figures, 6 tables. Reproducibility artifact: https://github.com/ruggerolanotte/protected-cores-artifact

Runtime monitoring traditionally evaluates a specification that is fixed before execution or externally modified when requirements change. In learning-enabled and data-intensive systems, however, the temporal relationships represented by a specification may themselves evolve. Allowing an AI component to directly replace a formal specification is unsafe: it may overfit transient behavior, weaken protected requirements, or activate statistically unsupported revisions. We introduce an intersymbolic architecture in which an untrusted AI proposer suggests temporal specification revisions and a symbolic governor controls their activation. Two results organize the framework. First, origin-version semantics makes the outcome of each obligation invariant to later revisions. Second, aggregate certification can conceal systematic failures on protected triggers; simultaneous aggregate and core-conditional post-selection certification controls both targets. A structural invariant preserves designer-protected components, and a proposer-independent lifetime error bound supports repeated activation decisions. The statistical bound concerns the predictable means of completed certification samples; interpreting it as future operational validity requires an additional stability assumption. Controlled synthetic experiments use a frozen supervised AI proposer to illustrate the masked-core failure at one decision and across repeated governed revisions. The proposer is a supervised regressor trained offline on synthetic tasks and frozen before use; it ranks candidates by predicted aggregate margin and never observes the protected-trigger success rate, so the masked-core failure arises from optimising the aggregate rather than from an adversary constructed by hand.

0.42

Assay: Claims That Decay With the Code. Content-Addressed Evidence Graphs for Accountable AI-Assisted Software Delivery

cs.SE ·2026-09-28 ·Om Shankar Tiwari, Tangi Vass, Gagan Deep Singh
code/data similarity 0.62
Abstract · 8 pages, 4 figures, 5 tables. Code, experiments, and paper source: https://github.com/OmShiv/assay-research

AI coding agents fail in two coupled ways. They spend most of their context window rediscovering where things live, and they assert success without evidence when the work gets hard. Repository indexes address the first with cheap context, and orchestration frameworks with adversarial review address the second with accountability. Both describe the same object, the structure of the codebase, at two timescales: what is true of the code now, and what was verified to be true, at which revision, by whom. Assay makes that observation operational. Every claim an agent makes (tests pass, no secrets, behavior preserved) is bound to the Merkle hash of the dependency cone of the code it covers, so the claim is stale exactly when that code or anything it depends on changes. We show the binding is sound and minimal, and that the blast radius of a change is precisely the set of claims it invalidates. On the graph we place a risk-proportional evidence obligation, a bounded review protocol with separation of duties, and a merge gate that consults no model: coverage, freshness, signatures, exit codes, plausibility, evidence monotonicity (the mechanical form of "do not delete the failing test"), and review status. Assay is a dependency-free Python tool with an MCP server. On five public repositories a 600-token brief costs 14x to 114x less than an exploration proxy, warm rebuilds are up to 5x faster than cold ones, cone binding re-verifies 7.9% to 81.9% of claims where repository binding re-verifies all of them while per-module binding misses 23% to 68% of required invalidations, and the gate blocks 9 of 9 scripted adversarial behaviours while admitting the honest ones. Every number in this paper is generated by the released scripts.

0.60