CS Paper Radar

LEGIT: Credentialing Protocol for Trustworthy AI Agent Marketplaces

Steve Drew, Jiayu Zhou

cs.AI ·submitted 2026-09-18 05:02 UTC · v1
benchmark

Abstract

Agentic marketplaces are emerging where AI agents with varying capabilities autonomously complete specialized tasks for buyers. A major challenge of such marketplaces is that buyers cannot easily determine which agent will perform best on their tasks. Reported benchmark scores may be difficult to verify or compare across tasks, software, and budgets. We introduce LEGIT, a credentialing protocol connecting certification, reputation, and proposed marketplace allocation. Certification binds measured quality and cost per solved task to an agent configuration, task domain, evaluation budget, and evidence through a signed record. Reputation links records of past task outcomes to the same identity, subject to the reliability of the reported feedback. Buyers and agents can verify credential records and inspect optional visual profiles. Evaluations reveal cost differences between agent configurations with similar observed task success, and show that comparisons depend on the evaluation budget. These results support binding performance measurements to the tested configuration and resource limits. A complementary analysis quantifies the deposits and fees required for reputation manipulation under a stated Sybil attack model.

Comment
28 pages, 7 figures, 11 tables

Closest papers

Nearest neighbours by meaning across the whole index.

CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned Environments

cs.AI cs.CL ·2026-09-23 ·Yuxuan Li, Will Epperson, Wesley Deng, Zezhou Huang
benchmark similarity 0.68
Abstract

Computer-use agents (CUAs) increasingly act on behalf of users online. What happens when the environments they operate in have incentives of their own? Online marketplaces, for example, may favor some products over others, steering agents away from the user's objective. Existing CUA benchmarks cover cooperative settings or explicit attacks, but do not test whether agents preserve user objectives when the environment itself has a stake in the outcome. We introduce CAVEAT, a controlled benchmark spanning nine marketplace environments and a taxonomy of eight common steering mechanisms. Across five model families, agents purchase the user-optimal product in 78.6% of matched-control episodes but only 17.3% when steering mechanisms are enabled. Larger models and more reasoning improve robustness, but substantial failures persist. Trajectory analysis and targeted ablations identify three weaknesses in how agents decide: they (1) prematurely narrow the set of alternatives they consider, (2) impose priorities the user never stated, and (3) commit before resolving decision-relevant evidence. Guided by this diagnosis, we develop CAVEAT-Harness, which targets these failures and raises the optimal purchase rate by up to 80.0 percentage points, and show that targeted post-training further improves a smaller open model. These results establish incentive robustness as a distinct challenge for delegated agents, diagnose failure modes, and show how targeted interventions can substantially improve robustness.

0.50

Tempting the Agent: The Economics of Reputation without Persistent Identity in AI Agent Markets

cs.MA q-fin.GN ·2026-09-02 ·Federico Gatta, Manuel Naviglio, Francesco Tarantelli
similarity 0.66
Abstract

Reputation is a fundamental mechanism through which markets sustain trust when service quality cannot be perfectly assessed ex ante, constituting a form of intertemporal economic capital by attracting future demand. Its effectiveness as a disciplinary mechanism depends not only on past interactions but also on the persistence of the identity to which reputation is attached. When identities can be abandoned and recreated cheaply, reputational capital may itself become an object of opportunistic exploitation. This paper develops a dynamic economic framework to study when reputation is sufficient to discipline autonomous agents. We model reputation as capital attracting future economic activity. At each point, an agent chooses between operating honestly, investing in quality to preserve future gains, or executing a one-shot deviation to extract its reputation's value and restart from a penalized identity. Our analysis relates the temptation to opportunistic behavior to identity-reset costs, reputation persistence, demand sensitivity, and enforcement design, deriving comparative statics on optimal quality provision. Autonomous AI-agent operating on the blockchain are a relevant application: infrastructures such as ERC-8004, ERC-8183, and x402 combine reputation, identity, and payments in permissionless markets. Nonetheless, our framework applies to any environment where reputation generates future business and identities are replaceable.

0.29

Anduin: A sybil-resistant agentic marketplace reputation system

cs.AI ·2026-11-03 ·Reza Ramji, Preston Vander Vos, Ziming Zeng, Duc V. Le
similarity 0.65
Abstract

As autonomous AI agents enter permissionless onchain markets, selecting new service providers remains a core challenge. Existing reputation systems depend on scarce resources like human curation, identity credentials, or voluntary reviews, while naive volume rankings are vulnerable to Sybil attacks. We evaluate whether existing onchain payment histories can yield provider ratings that are resilient against Sybil attacks. Using a 157-day dataset comprising 66M agentic transactions worth $6.1M across 16 blockchains, we model payment flows as a bipartite graph between buyers and providers. A provider's rating is derived from graph paths linked to a buyer's existing trusted network. We propose three Sybil-resistant rating algorithms: a max-flow rating and two relaxed algorithms optimized for stronger rating performance. Evaluating these algorithms against simulated adversaries across 813 buyers, we show that rating manipulation costs 20% of those buyers' total spend and is around 23x harder to manipulate than the standard similarity rule (collaborative filtering).

0.12

Get In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce

cs.AI cs.CYcs.HC ·2026-10-12 ·Adrian de Valois-Franklin, Alex Bogdan, Anas Ibrahim, Ibrahim Shaer, Tamaz Andguladze
benchmark similarity 0.64
Abstract · 23 pages, 7 figures, 10 tables. Companion dataset: https://doi.org/10.5281/zenodo.23325184

AI agents now browse and buy on people's behalf, and benchmarks measure how well the agents perform. Far less is known about the other side of the transaction: whether a merchant's website lets an agent in, publishes terms precise enough to decide a purchase, and leaves evidence of what was agreed. This paper presents a merchant-side benchmark built on those three questions and reports a pilot of it. The central measurement, decision sufficiency, rates each criterion of a concrete buyer mandate (total cost, cancellation or return terms, service date, variant identity) as determinate, indeterminate, or unknown from the merchant's own published evidence. Readiness scores carry their evidence coverage and missingness bounds; access is reported separately for a declared bot, a declared headless browser, and a user-present browser; and each observed term is mapped to the RAILS admissibility class of the evidence that could support it in a dispute. Four large merchants received the full assessment: three refused the declared profiles at their offer pages, and no mandate could be confirmed as met, because each merchant left a total or a refund term undetermined. An access census of 555 sites of merchants that two payment processors publicly name as clients found the home page unreadable to both declared profiles at 105 sites (18.9%). A declared assessment of 330 merchants, scored by deterministic rules without a rater, found that evidence coverage rather than the score decides who can be ranked: 115 merchants (35%) reached the ranking threshold, rankable profiles scored a median of 89 to 90, and material terms was the one pillar well below the others, because the all-in total was decidable at 18 merchants. The pilot shows that the measurements can be taken; it does not validate them or rank merchants, and the weights and the projected clearing band await validation against outcomes.

0.29

DelegateEval: Evaluating Platforms for Personal External Service Agents

cs.AI ·2026-11-05 ·Jichao Li, Jinrong Lu, Manjiang Yu, Zhixuan Liu
similarity 0.63
Abstract · 33 pages, 9 figures, 6 tables. Evaluation framework and experimental design

AI agents are expanding from personal assistance into professional services delivered to external clients on an individual's behalf. We define a personal external service agent as one authorized and managed on an ongoing basis by an individual. It uses that person's knowledge, experience, and professional methods to accept client work, deliver results, and provide follow-up support. Client feedback, professional reputation, and potential income may give individuals more direct incentives to keep developing and improving agents for external services than for personal assistance. Yet building a safe, reliable external service system independently remains difficult, even with AI-assisted programming. People without AI expertise need platforms that are easy to use and safe to operate, with support for creating, developing, and running their agents. We propose the DelegateEval evaluation framework and experimental design to assess these platforms and guide their development. The framework covers eight dimensions: ease of creation and management, service discovery and matching, delivery quality, expertise inheritance, service continuity, learning and improvement, reliability and governance, and efficiency and value. A complete dataset, executable environment, and unified scoring protocol remain to be developed, and empirical validation across foundation models has yet to be conducted.

0.33

Toward User-Centered Agentic Recommendation: When Does Platform Competition Benefit Users?

cs.AI ·2026-11-03 ·Kehan Zheng, Deyao Hong, Qian Li, Jun Zhang, Jie Jiang +1 more
similarity 0.62
Abstract

Conventional recommender systems operate within platform-specific catalogs, limiting access to potentially valuable resources across platforms. We propose user-centered agentic recommendation, a paradigm in which user agents compare recommendations from competing platform agents, and formalize their interaction as a repeated game. Platforms strategically modify descriptions without changing intrinsic item value, while each user agent assesses platform credibility using feedback from consumed items only. Our analysis shows that expanded resource access need not improve user welfare: strategic misrepresentation and limited verification can offset the benefits of platform competition. We quantify when platform credibility assessment allows users to benefit from broader access despite strategic exaggeration. Our analysis identifies resource and reporting conditions that guarantee higher average welfare than the original platform with an interaction budget linear in platform count. Experiments on synthetic and Amazon-grounded markets show that credibility assessment can improve finite-horizon utility, while its benefits depend on market size and mechanism parameters.

0.24

Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents

cs.AI cs.CYcs.SE ·2026-09-30 ·Serhii Zabolotnii
code/data similarity 0.62
Abstract · 18 pages, 2 figures, 5 tables. v2 corrects the agentic-coding study (174 distinct mechanisms replace 280 index-cloned cases) and adds an ind…

Benchmarks, audits, and agent protocols describe performance, permissions, and repair, but not how observed evidence should change an agent's authority during a consequential task. We propose a Runtime Assurance Contract (RAC), a policy-level formal schema binding autonomy boundaries, component eligibility, evidence state, transition policy, human-review capacity, and non-compensatory gates. Under RAC, soft metrics may inform routing, whereas a failed or unknown mandatory gate forces retry, switch, escalation, deferral, or stop; aggregate performance cannot authorize action. We define the contract, an evidence record, a permission rule, and five invariants. A deterministic failure-injection study in agentic coding evaluates 174 cases, each a distinct mechanism, under a gate conjunction, a score-only rule, and a restricted protocol baseline. At the published example weights and threshold, the score rule admits 56 of 63 block-required injections and all 24 review-required injections; tuned in hindsight, it matches the conjunction case for case. For positive weights, a positive threshold, binary risk signals, and each signal firing alone on some case, the two rules agree on every case if and only if the threshold does not exceed the smallest weight. On 124 further cases written blind by a model of another vendor, the conjunction itself admits 35 of 79 injections, and so does the tuned score rule: a missing detector is invisible to every decision rule. Eighteen hand-authored traces check version-pinned evidence and review transitions. In a prospective synthetic holdout of 24 episodes, RAC and a separately implemented full stateful baseline both match the labels of two blinded LLM judges. These studies test mechanisms on synthetic cases; they establish neither deployed safety nor cross-domain effectiveness.

0.26

A Trust Layer for Agent Evaluation

cs.AI ·2026-10-05 ·Mohammadreza Sediqin, Shivali Dalmia, Srinivasa Karthikeya Reddy Kovvuri, Abhishek Mukherji
benchmark similarity 0.62
Abstract · Expanded version with additional experiments

Deterministic benchmark scores show that an agent received credit, but not whether that credit was earned, reported honestly, or would hold on a second run. We introduce a Trust Layer for Agent Evaluation, an additive post-hoc framework that reports, beside each recorded score, whether it should be believed. It verifies four properties: whether the result is supported by the benchmark's own grading logic, whether a passing answer was earned through traceable computation, whether the agent's completion claim matches what occurred, and whether the result is stable under repeated execution. The first three use only saved artifacts; the fourth re-runs the agent. Model judgments only label evidence under majority voting; all verdicts follow deterministic rules and never modify the recorded score. Applied to five agent configurations on 108 tasks from Agents' Last Exam, every model shows passing runs with no traceable computation (at rates varying tenfold), confirmed false completion claims, and unstable results: 18-46% of tasks do not stay in one score band over five runs. Only 22.6% of recorded passes clear all four checks (95% CI 15.0-32.6, n=84). Measuring what an agent can do and verifying that it did it are different problems, and current benchmarks address only the first.

0.39