CS Paper Radar

WAR-CRAFT: Refining Web API Recommendations through Requirement Auditing

Zishuo Xu, Dezhong Yao, Zhiqiang Xu

cs.AI ·submitted 2026-10-31 02:42 UTC · v1
code/data

Abstract

Web API recommenders return a set of APIs for a mashup request but do not check it against provider documentation, the main public evidence of an API's capabilities. The set can therefore contain unsupported or redundant APIs and miss requested capabilities. Filling every apparent gap does not fix this, because a gap may already be served, may not need its own API, or may have no documented provider. We study post-recommendation refinement, which revises a frozen recommender's output using only the request, the catalog, and provider documentation. We propose WAR-CRAFT, which treats refinement as three decisions. An LLM removes unsupported or subsumed APIs by judging candidates jointly, then audits the remaining set against a requirement checklist derived from the request. For each uncovered requirement, a fine-tuned catalog-constrained Resolver proposes a catalog API or abstains, and an independent documentation check admits only supported proposals. On ProgrammableWeb with 7 heterogeneous upstream recommenders, WAR-CRAFT raises mean SetF1 from 0.558 to 0.644, above the strongest of 5 refinement baselines (62.5%), and roughly halves functional redundancy (0.127 to 0.065) without changing mean functional coverage. Gains range from +2.2 points on generative recommenders to +18.2 on fixed-cutoff rankers. At matched set sizes, WAR-CRAFT still outperforms truncating the upstream output, so the gain is not explained by set-size calibration alone. Our code is available at https://github.com/ZsXu-enrico/WAR-CRAFT.

Closest papers

Nearest neighbours by meaning across the whole index.

A Code-Level Audit of X

cs.CY cs.SE ·2026-11-12 ·Luka Bekavac, Simon Mayer
similarity 0.65
Abstract · Submitted. Currently under peer-review

The transparency regime of recommender systems on Very Large Online Platforms (VLOPs) is undergoing significant change under the EU Digital Services Act (DSA). The DSA requires platforms to disclose information about their recommendation pipelines and grants vetted researchers access to non-public data, which may extend to recommender code. However, it is not known to what extent such code releases enable substantive and meaningful third-party auditing of a production recommender system. We address this question by proposing a code-level audit methodology for recommender systems and applying it to the X platform, which has voluntarily open-sourced its recommender pipeline. Our methodology is a backwards-walk traversal of the repository, tracing each decision-relevant component from the user-facing timeline back to its upstream producers, checking that it is reachable from the served pipeline, and classifying each under a four-tier inspectability scheme. Applying this procedure, we identify 13 transparency dead-ends on the live serving path: models, services, label lists, and configuration values that the public code depends on but the release does not contain. We find that the components that decide which posts are filtered and how they are ranked fall into the two least inspectable tiers, a pattern that intensifies with the 2025 integration of the proprietary Grok language model into moderation, annotation, and ranking. We further show that the release keeps 2023 code that the 2025 pipeline no longer reaches, with 36 parameter references that would pass for dead-ends without a reachability check. Finally, we discuss which additional artifacts would be required for meaningful auditing.

0.18

AutoTailor: Automatic, User-Aligned Capability Selection and Adaptation for Web Agents

cs.AI cs.SE ·2026-09-11 ·Xinyun Cao, Adriana Szekeres, Fazle Elahi Faisal
similarity 0.63
Abstract

Web agents can utilize reusable tools to reduce the cost and latency of low-level browser interaction, but automatically discovered tool collections can be large, redundant, and poorly aligned with user demand. We present AutoTailor, a meta-agentic framework for constructing and maintaining a compact set of trajectory-derived Model Context Protocol (MCP) APIs. Offline, AutoTailor converts web trajectories into parameterized browser-automation programs, applies a Quality Filter to remove APIs with unsuitable granularity and redundant functionality, and applies a Usage Likelihood Filter to prioritize broadly useful capabilities while preserving semantic coverage. Online, Dynamic Reselection monitors task outcomes and API usage, identifies recurring coverage gaps, adds relevant candidates, and prunes persistently unused capabilities. We evaluate AutoTailor on 106 WebArena Postmill tasks. Offline filtering reduces the initial 1,283 unrefined APIs to 87, and Dynamic Reselection produces a 33-API set. With reasoning and acting (ReAct) fallback, this set achieves 90.6% correctness, compared with 87.5% for ReAct alone, while reducing average total request-token cost by 57.8% and latency by 29.4%. Without ReAct, it achieves 60.1% correctness, marginally matching the performance of unrefined set, while reducing request-token usage by 94.9%. Together, these results show that static filtering produces a compact inventory of APIs expected to support core, high-likelihood tasks, while dynamic reselection further tailors that inventory to observed user needs. This combination improves accuracy and latency while sharply reducing token usage and end-to-end cost, demonstrating the value of user-aligned capability management for efficient web agents.

0.10

Tool Recommendation for Multi-Step Agents: Finding Every Tool a Task Needs

cs.LG cs.IR ·2026-10-16 ·Sam Partee, River Stevens, Thierry Damiba, Valerie Fanelle
benchmark similarity 0.62
Abstract · 11 pages, 3 figures, 5 tables

A multi-step agent task fails when a single tool it needs is missing, yet the tools an agent may need are too many to list: the roughly 8,200 tools on the platform we study have definitions that together exceed the context window of any model we evaluate. We present Tool Recommendation, a deployed system that gives an agent all of them, hundreds of times more than a typical MCP server exposes, through a fixed interface of three tools. It runs at the gateway between agent and tools, executing every call under the end user's own credentials and checking it against the user's consent for each service and the tools the agent may use. We evaluate it on both a public retrieval benchmark and billed end-to-end session cost. On ToolRet, Tool Recommendation places every tool a query needs in the top ten 57% more often than BM25, a lexical ranking that commercial tool search offers, and compares favorably with published systems up to nine times its size. In paired multi-step workflows, its first recommendation covered every required tool in 93% of Codex sessions, and against a fixed list of 100 tools it cut input tokens by 17% with Codex and 40% with Claude Code and the total bill in all 14 runs, by up to 40%, with no change in task completion.

0.42

Reparo: Synthesizing and Validating Access Control Policies

cs.SE cs.CR ·2026-10-30 ·Ayesha Binte Mostofa, Gehao Zhang, Sam Stern, Juan Zhai, Pubali Datta
similarity 0.60
Abstract · 22 pages, 9 figures

Writing correct access control policies for software applications is difficult because it requires expertise in the authorization language and a thorough understanding of the application's authorization requirements. Existing synthesis and mining approaches assume explicit authorization inputs and a trusted reference policy. However, authorization logic is often distributed across code, comments, and documentation rather than captured in a single artifact, and developer-defined policies in real-world application repositories are often incomplete, outdated, or incorrect. We present Reparo, an LLM-aided pipeline for access control policy synthesis, repair, and validation. Reparo collects repository evidence for authorization intent and uses it to guide an LLM in generating an initial policy. It then uses official policy-language validation tools to identify and automatically repair errors. For semantic validation, a different LLM derives a behavioral oracle from the same repository evidence, consisting of concrete authorization requests with expected allow or deny decisions, which Reparo executes against the generated policy. We manually assess the oracle on a statistically sized sample of 390 test cases. We evaluate Reparo on 228 real-world GitHub repositories that use Cedar, OpenFGA, or Rego. In these repositories, 40.6-55.7% of the repository policies fail their language's validation tool, whereas Reparo produces 99.0-100% valid policies after repair. Agreement with the oracle is 98.2-99.6% for OpenFGA, 82.0-86.5% for Cedar, and 76.3-80.2% for Rego. Manual review confirms 91.5-96.9% of the sampled test cases as correct. Synthesizing and validating the policy for one repository takes about five minutes and costs $0.55-$1.57. These results show that repository evidence can support both access policy synthesis and independent validation, even when no reliable reference policy exists.

0.45

DualMine: Static-Dynamic REST API Constraint Discovery with Dual Validation

cs.SE ·2026-08-18 ·Tu Nguyen, Huy Nguyen, Juan C. A. Valenzuela, Thanh Nguyen, Tien N. Nguyen +1 more
similarity 0.60
Abstract · Submitted to a conference

REST API constraints capture semantic properties of API responses and are essential for automated test oracle generation, but they are difficult to discover reliably. Static approaches infer constraints from API specifications and documentation, but their results may be affected by incomplete, ambiguous, or outdated specifications. Dynamic approaches mine invariants from execution traces, but their results depend on execution coverage and may include coincidental properties that hold only for the observed executions. This paper presents DualMine, a hybrid framework for REST API constraint discovery that integrates specification-based constraint mining with runtime invariant mining. It first extracts candidate constraints from OpenAPI specifications using an LLM-based static miner and from request-response traces using dynamic invariant mining. It then performs asymmetric dual validation: runtime evidence is used to validate or refute specification-derived constraints, while specification-aware LLM reasoning is used to filter implausible log-derived invariants~without discarding plausible undocumented behaviors. Finally, it applies counterexample-guided refinement by performing targeted API executions to resolve uncertain, overlapping, or conflicting constraints. We evaluate DualMine on 39 real-world REST APIs and compare it against state-of-the-art static-only, dynamic-only, and constraint discovery approaches. The results show that it improves the quality of discovered constraints by reducing unsupported constraints, retaining complementary constraints missed by individual approaches, which helps detect 48 real REST API faults.

0.18

Invariant-Aware Harness Enforcement to Protect LLM Agents from Silent API Failures

cs.CL ·2026-11-16 ·Sajib Acharjee Dip, Brian Ouyang, Lala Motlhabi
code/data similarity 0.59
Abstract · 26 pages, 7 figures. Code: https://github.com/Sajib-006/silentguard . Data: https://huggingface.co/datasets/Sajib-006/AgentOrchBench

LLM agents answer questions by calling APIs and trusting the results. An API can fail silently. It ignores part of a request, such as a misnamed filter, yet returns HTTP 200 and a plausible but wrong result. Current fixes, such as checklists, reviewer models and LLM judges, ask the model to catch it. But models often see the problem and answer anyway. We present SilentGuard, which moves the decision from the model to the harness running tool calls. It tests each result against invariants that any correct API must satisfy and withholds values that break one. We find silent failures in 11 of 15 public research APIs, with counts inflated up to 691 times. Current fixes leave over 70% of answers wrong, and agents report 77.4% of counts a judge flags. Across 7 open models, SilentGuard cuts wrong answers on its target faults from 74.0% to 1.7%, with no loss on fault-free tasks. On three frontier models they fall from 70.2% to 0.5%. A warning from the same check leaves 70.1% wrong, so enforcement is the key. On new fault types, the harness alone cuts wrong answers from 61.1% to 24.5%. It avoids wrong answers by abstaining rather than recovering.

0.53

When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrieval

cs.LG cs.IR ·2026-08-17 ·Yiqi Liu, Joseph James, Yang Wang, Chenghao Xiao, Chenghua Lin
publishedbenchmark similarity 0.59
Abstract

Large-scale agents increasingly rely on retrieval to access external capabilities. We study this retrieval gate in structured tools and APIs, a measurable class of tool-backed executable skills that must be surfaced before an agent can plan, incorporate, or act. In this setting the retrieval layer can silently fail even when the capability corpus is fixed: on ToolRet, a retriever fine-tuned on one source-specific slice collapses on another source-specific slice of the same benchmark, with FT-1100 despite its higher lexical overlap with the gold tools. We call this failure mode source-style collapse. Query-side TF-IDF fingerprints flag source styles on which the fine-tuned retriever is likely to fail better than semantic or length-based proxies, giving a cheap signal for mismatch over a fixed tool corpus. We propose ToolScout, a source-aware routing method that uses this signal as a routing guard: on the mixed 4,996-query stream, TF-IDF-based routing raises coverage from 22.3% to 86.1%, and across five collapsed sources 20 matched examples raise the coverage-weighted global top-1 proxy from 1.3% to 53.9%. The same failure and routing behaviors persist when tools are rerendered as executable skill cards, which rules out raw API-schema format as the sole cause.

0.60

SLATE: A Large-Scale Benchmark for Long-Horizon API Selection in E-commerce Tool Use

cs.CL cs.AI ·2026-11-02 ·Rongzhe Wei, Ge Shi, Min Cheng, Sarthak Ghosh, Vaibhav Gorde +3 more
publishedbenchmarkcode/data similarity 0.59
Abstract · 20 pages, 13 figures, 6 tables. Camera-ready version for the NeurIPS 2026 Workshop on Agentic AI Benchmarks and Applications for Enterprise …

Enterprises increasingly want LLM agents to automate standard operating procedures over their software stacks -- mapping each step of a procedure to the right call among thousands of internal APIs, without hand-authored step-to-API mappings. The core difficulty is API selection at scale: choosing and composing the correct tools over a long horizon, judged only by whether the overall task succeeds. Current tool-use benchmarks do not test this regime -- they expose only a handful of tools, evaluate single-step invocation, or grade multi-step behavior with subjective LLM-as-a-judge scores that penalize valid alternative trajectories. We introduce SLATE (Synthetic Large-scale API Toolkit for E-commerce), a large-scale benchmark that tests large-toolset, long-horizon API selection in a realistic enterprise domain. SLATE couples (i) queries decomposed into hierarchical, tool-grounded procedures over a ~1,000-tool e-commerce library, (ii) a deterministic, context-aware simulator that returns recorded outputs for matching calls and default responses otherwise, and (iii) an end-to-end task-success metric -- a task succeeds when it reaches every reference outcome -- that credits diverse-but-valid trajectories while remaining fully automatic and reproducible. Experiments with six open decision models and the language models they are built on, from 3B to 12B parameters, show that even the best model completes only 5.0% of the tasks, establishing SLATE as a challenging testbed for reliable enterprise tool use. We release SLATE, its simulator, and the evaluation harness at https://geshijoker.github.io/slate/

0.34