CS Paper Radar

Characterizing Statistical Separability in TP-CRIV for Probabilistic AI Models

Teruki Sano, Minoru Kuribayashi, Masao Sakai, Shuji Isobe, Eisuke Koizumi, Zhang Zhang, Satoru Matsumoto

cs.CR cs.AI ·submitted 2026-10-08 03:22 UTC · v2

Abstract

Third-party challenge-response identity verification (TP-CRIV) enables an independent verifier to assess whether a claimant possesses a model identical to a remotely deployed model without directly accessing the reference model. However, for probabilistic AI models, repeated executions of the same query may produce different outputs and therefore different verification observations. This raises the question of how such stochastic evidence should be accumulated and how much evidence is required for reliable verification. In this work, we characterize statistical separability in TP-CRIV of probabilistic AI models. Specifically, we relate challenge-wise behavior of matching and non-matching provers to verification-level separability. The characterization explicitly describes how the numbers of independent challenges and repeated responses affect detection performance and enables the verification budget required for a target AUC to be estimated. We instantiate the proposed characterization for LLMs using open-ended challenges. The experiments demonstrate matching-non-matching separation, close agreement between theoretical and empirical AUCs, and consistent estimates of the minimum verification budgets. These results provide a statistical basis for relating probabilistic model behavior to verification-level separability and the evidence required for third-party verification.

Closest papers

Nearest neighbours by meaning across the whole index.

TP-CRIV: A Framework for Third-Party Challenge-Response Identity Verification of AI Models

cs.CR cs.AI ·2026-09-24 ·Teruki Sano, Minoru Kuribayashi, Masao Sakai, Shuji Isobe, Eisuke Koizumi +2 more
published similarity 0.74
Abstract

Artificial intelligence (AI) models are increasingly deployed through remote services, making model misappropriation a growing concern. Existing approaches, including watermarking, fingerprinting, and model similarity analysis, primarily rely on predefined evidence or direct behavioral comparison and do not explicitly evaluate whether the claimant currently possesses and can utilize model-dependent information relevant to the claimed model identity. In this paper, we propose Third-Party Challenge-Response Identity Verification (TP-CRIV) for AI models. TP-CRIV targets a third-party verification setting in which the verifier has neither white-box nor API access to the claimant's model, can interact with the suspicious deployed service only through its ordinary black-box inference interface, and does not require protocol-specific cooperation from the service provider. Under these constraints, the framework enables the verifier to obtain empirical evidence as to whether the claimant locally possesses a model satisfying a predeclared identity relative to the deployed model. Verification is conducted under fresh, previously undisclosed requirements and network isolation, so that the demonstrated capability cannot rely on online external assistance after challenge disclosure. The resulting evidence is interpreted relative to independently specified and calibrated matching and non-matching operating situations and is statistical rather than cryptographic. We instantiate TP-CRIV for CNN image classifiers using probability-control-based witness generation. Experiments on ten ImageNet-pretrained TorchVision models demonstrate clear same/cross-model separation and finite-challenge verification using independently calibrated thresholds.

0.64

EAVer: Long-Form Factuality Verification as an End-to-End Agentic Policy

cs.CL cs.LG ·2026-09-02 ·Kening Zheng, Aoying Zheng, Zhigang Chang, Yazhi Guo, Miaotian Guo +19 more
benchmark similarity 0.61
Abstract

Long-form factuality verification is commonly implemented as a static decompose-search-verify pipeline, with separately prompted modules processing claims and invoking external search. Treating claims independently makes LLM and search calls scale with claim count and causes repeated searches for overlapping evidence about related claims. We introduce EAVer, an End-to-end Agentic Verifier that learns to control the complete response-level verification workflow as a unified policy. EAVer groups semantically related claims, routes each group to direct verification or targeted search based on confidence, and keeps evidence returned by search in compact in-context memos for cross-claim reuse. To train this policy, we develop a privileged-teacher synthesis pipeline that converts gold claim annotations into executable multi-turn tool-interaction trajectories with live search rather than post-hoc rationales. Structural, label-alignment, tool-use, search-budget, and leakage checks yield 1,447 quality-controlled trajectories. We further construct 794 bidirectional same-trajectory preference pairs that keep claim grouping, search, and evidence fixed, enabling decision-focused Direct Preference Optimization (DPO) over factuality-decision tokens. The results with Qwen3-8B show that EAVer outperforms the strongest search-based baseline on each benchmark by 2.88 Macro-F1 points on VeriFastScore and 4.73 points on the out-of-distribution FaStFact-Bench, while using about 80% fewer searches than the most search-efficient baseline. Moreover, EAVer consistently improves performance across models ranging from 4B to 32B parameters, demonstrating its strong generalizability.

0.22

Who Verifies the Verifiers? Accountable Semantic Consensus for AI-Mediated Web Information Verification

cs.CR cs.DC ·2026-10-24 ·Yanna Jiang, Mingyuan Li, Xu Wang, Guangsheng Yu
theory similarity 0.61
Abstract

Large language models (LLMs) increasingly decide whether Web evidence supports online claims, often by voting over several assessors whose errors are presumed independent and benign. On the open Web, retrieved pages can steer the assessors that read them, and assessors from one model family fail together. We prove that even for independent assessors whose every output sequence is possible on both valid and invalid claims, any protocol that decides only from their outputs and always reaches an agreed verdict never rejects invalid claims or rejects valid ones with positive probability. We present Astral, which pins all assessors to one committed snapshot of the Web evidence. With up to f assessors faulty, Astral accepts a verdict only when 2f+1 assessors attest to it above a confidence threshold and no signed challenge stands. On five public benchmarks with two of seven assessors adversarial, Astral lowers pooled attack success to 21.5-53.1%, against 54.7-77.8% for majority voting. The cost is coverage, as without attack 16.0-61.7% of claims receive a binary verdict and the rest are disputed for audit. Our code runs at https://anonymous.4open.science/r/Astral-55F1.

0.30

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

cs.AI cs.MA ·2026-10-01 ·Caiqi Zhang, Rujun Han, Zifeng Wang, Zoey CuiZhu, Nigel Collier +2 more
similarity 0.59
Abstract

As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.

0.43

When AI Investigators Cannot Trust Their Tools: Conflict-Aware Evidence Verification for Cyber Threat Investigation

cs.CR cs.LG ·2026-10-30 ·Yuchen Zhang
published similarity 0.58
Abstract · Accepted at WAITI 2026 (3rd Workshop on AI for Cyber Threat Intelligence), co-located with ACSAC 2026

LLM agents increasingly triage security alerts by combining the outputs of several detection tools, implicitly treating those tools as trustworthy. Unlike prior work on conflicts between a model's knowledge and its context, we study conflicts among the tools themselves: what happens when this machine-generated evidence is conflicting or wrong. On the corrected CIC-IDS2017 dataset, four heterogeneous detectors emit labels and confidences that we corrupt under a threat model varying the number of corrupted tools, the persistence of the corruption, and the confidence attached to wrong outputs. Across open-weight LLMs from 3B to 14B parameters, unverified investigators are fragile: one corrupted detector out of four lowers naive accuracy to .50-.72, three lower every unverified investigator to .13 or less, and honest disagreement alone makes them abstain on up to 63% of flows. We propose Conflict-Aware Evidence Verification (CAV), built from established components: an equal-weight vote that ignores reported confidences, an evidence-conflict gate, and an independent nearest-neighbour trust check that tests the leading candidates. With its prompt frozen before testing, CAV-LLM improves on the naive investigator in every corrupted condition ($p < 0.001$), reaching .84-.88 with three corrupted detectors against at most .08, while calling the LLM on 0.4% of clean traffic. A fixed verification rule is as accurate or better unless most tools are corrupted. When we make the independent channel itself noisy, CAV inherits its errors: it stays ahead when two or more tools are corrupted, but adding the channel as an extra voter is better when at most one is. Verification thus moves trust from the tools to the check, which must be measured and protected in turn.

0.20

Human-LLM Deliberation as Interactive Proof: Conditions for Verifiability Without Transparency

cs.CL ·2026-09-21 ·Baotong Zhang, Dean Foster, João Sedoc
theoryhot direction similarity 0.58
Abstract · 44 pages, 3 figures, 6 tables

When an LLM supplies an argument a user could not produce, how can the user decide whether to accept its claim? Inspired by interactive proofs, we pair a computationally unbounded LLM prover with a resource-bounded human verifier. The verifier checks requested details without seeing the LLM's internals; passed checks accumulate evidence toward an acceptance threshold. Given task-supplied bounds on false passes and human checking errors, we prove anytime-valid soundness against adaptive provers: the probability of ever accepting a false claim is at most a chosen error level. For a true claim, acceptance within the budget has probability at least the product of per-round probabilities that the LLM responds adequately and the human checks correctly, provided the threshold is reachable in time. Each additional check guards against false claims but gives a true claim another chance to fail. Which review lengths meet both error targets depends on the verifier's effort, cognitive load, expertise, and fatigue. With equal resources, we identify conditions under which the bounds certify a review by local checks but not a one-shot check. In LLM-only simulations on GPQA Diamond and Humanity's Last Exam, whether answers accepted without revision were more accurate varied by task and model pair.

0.38

Calibration Is Not Verification: Answer-Level Conformal Guarantees for Mixture-of-Agents

cs.MA ·2026-09-22 ·Nada Rahali, Zijia Wang, Zhi-Song Liu
benchmark similarity 0.58
Abstract · 11 pages, 3 figures; revised manuscript with updated experiments and analysis

Web answer synthesis with Mixture-of-Agents (MoA) requires reliable factual support for every atomic claim, yet agreement among agents is not verification. Correlated proposers can repeat a shared misconception, and a correct specialist fact receives little support when the other proposers stay silent. We introduce C-MoA, which decomposes MoA outputs into atomic claims, scores inter-agent semantic support, and calibrates claim retention at the level of whole answers. Under exchangeability and a policy frozen before calibration, it provides distribution-free marginal control of the probability that an answer retains any claim unsupported by the reference labels. Because calibration only selects a cutoff on a fixed ranking and cannot repair shared errors, we also study CONTRA-MoA, an optional extension that adds blinded near-miss tournaments, leave-one-agent-out stability, and availability-aware fusion, and we derive a competence--discrimination law for its verifier. Across 300 splits of the FactScore biography benchmark, the mean rate of fully supported answers (Fact) exceeds all six nominal targets, but only $1.7\%$--$9.8\%$ of supported claims are retained and $30\%$--$75\%$ of answers are empty. At $α=0.10$, mean agreement retains $3.1\%$ of supported claims, about $3.4$ times the $0.9\%$ retained by the best single-proposer score. On the medical Golden Dataset, it retains $37.3\%$ at a mean Fact of $0.898$, slightly below the $0.90$ target, and on NQ mean Fact reaches $0.904$. On a separate balanced FactScore cohort, memory-only counterfactual scoring is close to chance (AUC $0.522$ on 182 claims), and raw maximum fusion lowers the agreement AUC from $0.687$ to $0.652$. These results separate the statistical calibration of factuality from informative claim verification and clarify the role of each in multi-agent Web synthesis.

0.27

Not All Agreement Counts as Corroboration: Provenance-Conserving Multi-View Fusion for Typed Action Admission in Human-Robot Collaboration

cs.RO cs.AI ·2026-08-31 ·Zekai Jin, Hanrong Zhang, Yihong Tang, Fei Hu, Zhen Dong +1 more
benchmarkcode/data similarity 0.58
Abstract · 35 pages, 8 figures, 15 tables. Revised manuscript with clarified theoretical assumptions and evaluation scope. Code and supporting material…

Better probability scores do not establish that evidence has been counted correctly. Repeated inference over one observation can improve predictions without adding an evidential origin. Source-local numerical attributes alone cannot in general distinguish repeated derivations from separately countable acquisitions. PACT (Provenance-Aware evidence Conservation and Typed action admission) separates evidence magnitude from countability through a supplied provenance partition. Under singleton fidelity and insertion non-amplification, the coordinatewise meet is the unique pointwise greatest admissible within-component rule. Component budgets add under stated commensurability and separate-component additivity assumptions. Matched reassignments hold numerical outputs fixed while varying the counting relation. In four of 12 replicated-source tests on HandWritten, false refinement lowers macro-averaged negative log-likelihood and Brier score while increasing normalized common-support area under the risk-coverage curve (ncsAURC). In the controlled handover benchmark, removing the constructed adversarial-consensus condition leaves a 0.056 reduction in ncsAURC for provenance-partition aggregation relative to singleton aggregation under the same score functional. The corroboration contrast disappears, and method ranking remains selection-score dependent. In offline, reference-based human-robot collaboration with four prompts per camera and all other admission inputs fixed, duplicating each prompt output within its camera from multiplicity one to eight leaves all 720 PACT typed responses per checkpoint unchanged. Probability quality and evidence countability require separate evaluation.

0.49