CS Paper Radar

VRMaaS: Verifiable Reward-Model-as-a-Service for Trustworthy LLM Alignment

Guofu Liao, Hailang Xi, Taotao Wang, Shengli Zhang, Rongcheng Tu, Hao Liao

cs.CR ·submitted 2026-11-11 03:41 UTC · v1

Abstract

Reward models (RMs) are increasingly deployed as service components in LLM alignment pipelines, where payment, audit, moderation, and governance workflows act on the scores they return. An inference proof alone does not show whether a score belongs to the intended request, model checkpoint, numerical configuration, and verifier: in a controlled comparison that isolates the accepted statement from prover scale, a generic inference proof wrapped for on-chain checking remains valid after six classes of such service-field substitution. We formulate verifiable reward-model-as-a-service (VRMaaS) as a service-output integrity problem and present a framework in which each returned reward becomes a proof-carrying artifact. Its off-chain acceptance rule binds request identity, committed weights, verifier and configuration identifiers, proof-task order, and the digest of the exported package, so auditors can check a reward without rerunning the private RM or exposing inputs, checkpoints, or hidden activations; an optional on-chain zero-knowledge wrapper gives contracts a partial attestation of selected consistency checks and supplied service fields. The framework decomposes Transformer RM serving into arithmetic and lookup-backed proof tasks, aggregates them into a composite proof, and links hidden operands to the committed weights. A GPU prototype on 1B, 3B, and 8B reward models over 128-1024 tokens preserves 93.5-99% pairwise agreement with the baseline RM and rejects every tested service-field substitution in complete artifacts. Using exact integer witnesses and request-level aggregation of weight-binding openings, it generates complete artifacts in 49.8-206.1 seconds across the evaluated settings; auditors verify retained artifacts in 1.94-5.11 seconds, and partial public attestation costs about 217K gas. These costs suit delayed audit, settlement, and governance workflows.

Comment
24 pages, 6 figures, including appendices A-O

Closest papers

Nearest neighbours by meaning across the whole index.

Verifier-Grounded Agentic RL for Provably Correct Code

cs.LG ·2026-10-13 ·George Ma, Yuhao Zhang, Bing He, Zeren Shui, Ferhat Erata +3 more
similarity 0.69
Abstract

Reinforcement learning for code usually rewards passing tests, which a model can do while remaining wrong on the inputs the tests do not cover. A program verifier instead decides whether a program satisfies a formal specification for all inputs. Language models are almost always paired with verifiers at inference time only, and turning a verifier into a reward raises two difficulties. It reports only acceptance or rejection, which gives no gradient when every attempt fails, and a policy can cheat it, most simply by telling it to assume what the proof was supposed to establish. We build an agentic environment and reward that address both. The reward reads the verifier's diagnostics rather than its verdict, scoring a candidate by the weighted fraction of proof obligations it discharges, extended with hidden probes from a reference proof's annotations. An integrity audit outranks the verifier, rejecting any candidate that alters the specification or assumes what it was meant to prove. The model works as an agent against a live verifier over many turns, with a separately trained proof specialist as a frozen collaborator. Across Qwen3 models from 1.7B to 14B and a 35B Qwen3.6 mixture-of-experts model, held-out verification on code generation rises from 66.1% to 93.5% on DafnyBench and from 56.3% to 100% on SWE-Proof. This reward leads every other training signal we compare, including a binary pass-or-fail reward trained under otherwise identical settings, with the widest margin on code generation. When the agent must also deliver the repository patch, graded by each project's withheld tests, the fraction of held-out issues it resolves rises from 59.6% to 72.5%. Evaluated unchanged on three verification suites withheld from training, the checkpoints improve on all of them, including one whose language and verifier are absent from training.

0.24

Negative-Only Policy Optimization for One-Sided Verifiable Rewards

cs.LG cs.CL ·2026-11-12 ·Jiacheng Xu, Shuo He, Fuxiang Zhang, Chaojie Wang, Bo An
top venuebenchmark similarity 0.63
Abstract · Accepted at NeurIPS 2026 (Spotlight)

Reinforcement learning with verifiable rewards (RLVR) typically relies on a verifier that can reliably judge whether sampled solutions are correct. However, in many realistic settings, complete verification is generally unavailable, while cheap checks can only falsify failures. We formalize this regime as one-sided verifiability (OSV): verified negatives are reliable counterexamples, but verified positives are merely non-falsified candidates and may contain many false positives. This asymmetry makes standard RLVR unreliable, because reinforcing verified positives can directly reward shortcuts that pass partial checks. To address this issue, we propose Negative-Only Policy Optimization (NOPO), which treats verified positives as unlabeled and updates the policy only by suppressing trusted verified negatives. Specifically, NOPO combines a reference-relative negative suppression loss with a group-level adapter that scales each update by the proportion of non-falsified responses. Across unit-test, constraint-check, and LLM self-verifiers, NOPO improves pass@k over baselines on code, logic, and math benchmarks when oracle rewards are unavailable. These gains persist under inference-time scaling and cross-benchmark transfer.

0.30

Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

cs.LG cs.CL ·2026-10-14 ·Sangyeon Yoon, Wooseok Seo, Hamish Ivison, Hyesoo Hong, Donghoon Kim +3 more
similarity 0.63
Abstract · 15 pages

Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. We show that frontier models can receive full credit for responses with little substantive answer content. Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards. To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality. CALIF evaluates response likelihood under the request alone using the existing frozen reference policy, requiring no additional judge or learned reward model. Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels. It also outperforms corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks, demonstrating gains beyond training verifiers.

0.37

ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance

cs.AI ·2026-09-08 ·Praphul Singh, Shanu Kumar, Akshat Agarwal, Ganesh Kumar
similarity 0.63
Abstract

As LLM agents move from answering questions to carrying out procedures, failures can be unwarranted rather than visibly wrong: the final response looks acceptable even though the system skipped the check, branch, dependency, or invariant that made the answer justified. Output-only evaluation sees the answer, and trace-aware judging sees activity, but neither identifies which obligations were active for the query. We introduce CONTRACTEVAL, a diagnostic framework for making those active obligations explicit. It represents procedural instructions as query-active obligations and matches them against response or trace evidence, turning omissions, wrong branches, ordering errors, extra actions, invariant breaches, and output-contract violations into distinct conformance failures. On a controlled suite of audited procedural contracts, output-only and trace-aware LLM judges miss many injected structural failures; under gold expected and observed graphs, ContractEval detects and localizes all of them. LLM-backed extraction preserves much of this signal but remains calibration-sensitive. ContractEval is therefore not a compliance guarantee; it makes procedural conformance auditable rather than implicit in final-answer quality.

0.09

Optimistic Verifiers Get Hacked: Reward Hacking Under an Exact Process Oracle

cs.LG cs.AIcs.CL ·2026-11-16 ·Arunkumar V, Manoranjan Gandhudi, Gangadharan G. R., Arun Prakash, S. Senthilkumar
published similarity 0.63
Abstract · 30 pages. A shorter version, "The Verifier Gap", appears at the NeurIPS 2026 Workshop on Meta-Agents (non-archival)

Reinforcement learning from verifiable rewards inherits whatever its verifier gets wrong. Outside mathematics and code the verifier is a proxy, a rubric, a learned judge or a hand-written checker, and its distance from the true objective is never measured, so a rising reward cannot be told apart from a hack. We build a setting where the true reward is exactly computable at the process level: executable authorization policies over 124,416 contexts admit an oracle that lists every minimal sufficient reason for every decision. We compile these policies into exact process rewards, degrade the verifier along seven realistic axes (outcome-only, sufficiency, recall-only, extraction noise, coarse taxonomy, stale policy, LLM judge), and train against each while the exact verifier scores every rollout. Four phenotypes appear. Optimistic verifiers are hacked by padding: the sufficiency proxy ends at 0.940 while true faithfulness falls from 0.731 to 0.667 on 96 held-out episodes. Outcome-only verifiers saturate and stop teaching, noise dilutes, and ranking-preserving gaps do no harm. A model-free audit of the verifier, its selection error and gap sign, together with a short saturation pilot, anticipates which phenotype a run will produce. Across nine configurations spanning two frameworks, two model vendors, two adapter sizes, three precisions, two initialisations and two seeds, the phenotypes recur, sufficiency damage is positive in all 10 replication cells and beyond two standard errors in 7, and optimistic gaps remain the steeper branch. The damage magnitude does not replicate, and five pre-registered attempts to explain it are reported as they fell. We release the testbed, the training instrumentation, and 14 real public policies (Cedar, OPA, Gatekeeper) ported to executable oracles.

0.24

TrustBound: Mechanized Completeness for Payment Epochs Initiated by AI Agents

cs.CR cs.LO ·2026-11-14 ·Sandip Nandi, Aishwarya Bose, Apurba Bose, Soovadeep Bakshi
theorycode/data similarity 0.63
Abstract · 16 pages (LNCS), 2 figures, 8 tables. Code and Lean models: https://github.com/tokencraftdata/trustbound

Payment operations requested by an AI agent may remain executable, or their outcomes unresolved, after the agent has stopped. We study when a trusted payment service can account for every payment in an epoch and exclude further payments under earlier authority. We give a Lean model in which operations are recorded before they receive execution authority and both actions are serialized with an admission fence. We prove that fencing fixes a journal that covers every modeled effect and outstanding right; that discharging those rights prevents further effects; and that truthful outcomes for the recorded identities then give exact accounting. A provider-contract model, mapped to the admission model, makes created payment objects inert until registered. Premise-removal mutations confirm that each key premise is needed. We connect the models to an executable service: six service histories are checked against the model in Lean, and bounded schedules, scripted clients and concurrent workers exercise its reconciliation. The concurrent experiments uncovered a conflict caused by reordered provider reads, which serializing retrieval with journal updates removes in all tested cases. The result separates two obligations that are easily conflated in practice - preventing future payments and accounting for past ones - and states precisely which service guarantees establish each.

0.56

One Node, Two Roles: Simultaneous Contests for Validation and Attention in Rollups

cs.CR ·2026-10-08 ·Pranay Anchuri, Ben Berger, Matteo Campanelli, Akaki Mamageishvili
similarity 0.62
Abstract

An optimistic rollup is safe as long as at least one honest validator executes its state transition function (STF) and disputes any assertion that is inconsistent with the result of the execution. Allowing anyone to participate, however, does not give incentive to do so: as long as every assertion is correct, verifying assertion correctness goes unpaid. Attention mechanisms address this by paying nodes to execute the rollup's STF, regardless of whether they participate in the validation process. Since a single node operator can back many registered identities with a single execution, paying per identity does not buy execution diversity, that is, the number of operators that independently execute. In this work we provide a new modeling framework for this problem. We formalize the underlying cryptographic primitive as a new notion, arguments with registered provers, whose properties include a form of non-amortizability that, to our knowledge, has not been studied for modern succinct cryptographic proofs before. We design validation and attention reward mechanisms and analyze them as two coupled Tullock-like contests, in which an operator pays once for the required execution and then chooses how many identities to deploy in each role. We quantify the attainable range of execution diversities in equilibrium as a function of the marginal cost of deploying an extra attention identity. We use our framework as a lens through which we compare two attention mechanisms, TRACE (MARBLE 2026) and Proof of Diligence (AFT 2024). In particular, we show that as opposed to Proof of Diligence, in TRACE the marginal cost mentioned above can be adjusted to support a higher execution diversity. As a case study, we apply our results to Arbitrum One, a widely deployed optimistic rollup.

0.16

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

cs.AI cs.MA ·2026-10-01 ·Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier +2 more
similarity 0.62
Abstract

As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.

0.42