CS Paper Radar

Localizing Post-Wire Semantic Changes in MCP Agent Frameworks

Aditi Patodiya

cs.SE ·submitted 2026-09-17 04:06 UTC · v1

Abstract

Valid Model Context Protocol (MCP) messages do not guarantee that an agent framework preserves the distinctions downstream software needs. We present a differential testing method that follows a fixed tool result through each framework's public interfaces and checks explicit consumer requirements. Applied to 18 designed fixtures in four pinned Python integrations, the method identifies 13 unique fixture-task divergences involving structured values, declared errors, and rich content. Google ADK satisfies every primary contract in its observed path; the other integrations show interface-specific changes or an execution failure. Whether a change matters depends on the consumer: treating absent optional fields as equivalent to null explains most of OpenAI's strict rich-content failures. In an exploratory replay, parsing JSON text recovers more structured values but also returns incorrect values and values for fields absent at the source. Documented settings help in conflicting-output stress cases without restoring a separate structured slot. Fault challenges expose an oracle weakness and test its repair on fresh cases. The study provides reproducible, interface-specific evidence from controlled cases, not production failure rates or model-behavior measurements. Its practical implication is that post-wire regression tests need to specify both the information required and how the consumer reads it.

Comment
10 pages, 7 tables, 1 figure. Submitted to SE4AgenticAI 2026. Reproducibility artifact: https://doi.org/10.5281/zenodo.22784143

Closest papers

Nearest neighbours by meaning across the whole index.

Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs

cs.SE cs.CLcs.LG ·2026-10-24 ·Amit Verma
similarity 0.72
Abstract · 36 pages, 2 figures, 16 tables. Under review at Empirical Software Engineering. Artifact: https://doi.org/10.5281/zenodo.23586834

Pipelines that require structured outputs from hosted language models often specify a Pydantic schema in the prompt, validate the response, and retry when validation fails. Although native structured-output modes provide an alternative, application-side validation remains important for enforcing custom constraints. Yet pipeline reliability is often summarized by a single validity rate, which obscures why outputs fail and which interventions may help. We introduce a fine-grained taxonomy of schema-conformance failures and a deterministic classifier that assigns failure categories from model responses and Pydantic validation errors. We apply this framework to 3,997 generation attempts across nine OpenAI and Anthropic model/API configurations, using questions from HotpotQA and FinanceBench. Our results show that similar validity rates can mask different failure patterns, and that dominant failure categories vary with schema constraints and task. Markdown-fenced JSON is the most common failure: among 681 fenced failures, only 48.0% pass validation after fence removal, while the remainder conceal additional violations, predominantly field-length violations. In an exploratory prompt ablation, variants that suppress fences appear to shift failures toward length violations, but the shift is consistent with exposing violations that were already present, illustrating how aggregate validity rates can obscure changes in failure composition. We derive four engineering practices for validator, schema, retry and evaluation design, three supported by our data and one, on retry feedback, proposed but not tested, and release the classifier, completions, and annotations.

0.29

Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

cs.CL cs.AIcs.SE ·2026-09-22 ·Lijuan Tang, Yuemeng Zheng
similarity 0.67
Abstract · 9 pages, 4 figures, 3 tables. Accepted at the 2nd Workshop for Research on Agent Language Models (REALM) @ EMNLP 2026

A coding agent must emit a valid tool call--a parseable invocation of a tool in the provided schema--before the harness can execute its chosen action. We study how local serving stacks affect this protocol step and show that measured outcomes can depend on the serving layer rather than model behavior alone. In Ollama, the default tools= request is gated per model by a static template flag: some models are accepted and return calls as text, some return native tool_calls, while Phi-3 and Gemma-3 are rejected before inference. In our harness, rejection and retry exhaustion are not preserved as structured failure metadata, so downstream analysis can misclassify them as model non-calls and naively report 0% fidelity. Adding a text tool list while retaining the native channel recovers much of the measured fidelity for accepted models, whereas a uniform text protocol reduces fidelity for Llama-3.2, which has native tool-call support. Cross-stack probes on Ollama, llama.cpp, vLLM, and SGLang show different handling of the same request. Constrained decoding removes parse failures but can induce non-termination, and turn-pooled versus per-instance estimates differ by up to about 55 points. We conclude with a checklist for treating serving behavior as part of the evaluation protocol.

0.55

SilentProbe: Measuring Silent Failure in Production APIs Used as Agent Tools

cs.IR cs.SE ·2026-08-29 ·Zongrong Li, Shengkun Ye, Feiyou Guo, Zuoyou Dang
code/data similarity 0.67
Abstract · 12 pages, 9 figures. Code and data: https://github.com/Jasper0122/silentprobe

An LLM agent calling a production API cannot distinguish a query that matched nothing from a query the server did not understand. Both return HTTP 200 with a parsable body, no exception to catch and no field to branch on. We ask what predicts which one occurred, and what it does to the agent. Auditing 721,320 parameters across 2,501 independently published OpenAPI documents, we find that 7.5% declare an enumeration and 15.2% declare any machine-checkable constraint at all, while 40.1% of documents state at least one constraint in prose that their schema does not encode. Executing 219 schema-derived perturbations against live commercial endpoints from 27 vendors, reached through a single aggregation layer (Monid) that publishes a schema and returns a run identifier for every call, we find that constraint form, not vendor identity, predicts honesty: machine-checkable constraints yielded an honest error in 111 of 111 cases, prose-only constraints failed silently in 44 of 61 (p = 2e-13). Twelve models across eight families then met these endpoints on ordinary tasks. A vocabulary that the description merely exemplifies was missed by every model on 88 of 88 attempts, while vocabularies written out in full were used correctly 88 to 91% of the time. Running the full agent loop, models detected the resulting silent failure in 12% of cases, repaired it in 0%, asserted a false negative to the user in 41%, and invented a figure in 12%. Promoting the vocabulary into the schema removes the failure, from 88 of 88 to 0 of 89. The fix is one line of schema rather than a better model. Code, schemas, perturbation sets, agent transcripts and per-call run identifiers are released at https://github.com/Jasper0122/silentprobe.

0.60

Replay Fidelity in LLM Agent Evaluation: Randomness, Hidden State, and Test Oracles in AppWorld

cs.SE cs.AI ·2026-11-08 ·You Liu, Quanchao Lu
similarity 0.64
Abstract · Empirical study; 11 pages, 3 figures, 5 tables; historical AppWorld census with a bounded BFCL consistency check

Reproducibility in large language model (LLM) agent evaluation depends on replay fidelity: which parts of an execution trace are reconstructed, controlled, and compared. This matters for agent debugging, failure analysis, and counterfactual evaluation of stateful tool use. We audit all 71 unsuccessful executions in a fixed historical AppWorld corpus using 284 isolated fixed-code replay attempts. Reconstructed declared settings admit 42 cases under printed-output, pre-oracle database-delta, and native-verdict/test-count checks; resetting the random-number generator before each interaction admits 68. The 26-case gap contracts to four under database/oracle checks, with one endpoint unknown. Mechanism studies identify four materialized-state discrepancies despite matching outputs and verdicts. In one admitted endpoint pair, an identical API call yields matching responses but different row effects; common future Python RNG removes that deterministic difference in two paired repetitions. A separate JWT credential-equality program and alignment controls expose authentication/write differences, with missing repetitions retained. For all 10 affected post-oracle export cases, native export from retained output trackers restores the reference comparison without changing original cached contents. A purposeful 12-case BFCL pilot shows no discrepancy under its tested public-state, return, and verdict checks. We preserve resource censoring and infrastructure deviations. The contribution is a bounded empirical characterization of replay eligibility, hidden state, and test-oracle integration, without estimating debugger accuracy, repair gains, or method-ranking distortion.

0.12

Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss

cs.SE cs.AI ·2026-11-11 ·Xin Liu, Xu Han, Yuxin Wang, Haoran Song, Mingjian Zhao +1 more
similarity 0.64
Abstract · Research report

LLM applications increasingly rely on translators and compatibility layers to move conversations across APIs. Yet a translated request can be accepted by the target API while silently losing information about instruction roles, tool interactions, error states, or generation controls. We introduce isloss, a deterministic oracle for testing conversation-information preservation across LLM API translations. It maps source and translated requests and replies to a shared intermediate representation, ConvIR, and applies explicit rules to distinguish legitimate representation changes from information loss. Unlike schema validation or reference-based differential testing, isloss requires neither a reference translation nor additional model calls. It also distinguishes avoidable loss from degradation caused by target API limitations. We evaluate isloss using specification-based fault injection, human review, and experiments on translators, gateways, and client libraries. On held-out tests, it detects 95/125 planted faults (76.0%) with 2/100 false alarms (2.0%). Across a synthetic request suite spanning six configurations of five translator codebases, 127/685 completed translations (18.5%) pass static target checks while silently changing prompt, role, or tool information in ways the target API could have preserved. In translated replies, 203/558 streams (36.4%) cause exceptions or silent corruption in at least one tested official OpenAI SDK. Targeted experiments across six models show that translation-induced changes can alter model behaviour when tasks depend on the affected information. These results establish conversation preservation as a distinct correctness requirement for reliable LLM interoperability, agent execution, and model evaluation.

0.35

AgentTrustCI: Build-Time Policy, Grounding, and Trajectory Tests for Java LLM Agents

cs.SE cs.AI ·2026-11-04 ·Sachin Gupta
published similarity 0.64
Abstract · Accepted for publication in the 2026 International Conference on Computer and Applications (ICCA)

Infrastructure security does not establish whether an LLM agent selected an allowed tool, grounded its answer in tool output, or followed a required workflow. AgentTrustCI is a Java/JUnit 5 framework that expresses these application-layer expectations as policy, grounding, and trajectory checks over recorded traces. A deterministic replay implementation supports repeatable build-time tests; a judge interface provides an extension point for optional semantic evaluation. On 55 synthetic cases spanning seven enterprise domains, the full pipeline correctly classifies all 30 injected faults and 25 passing controls, including ten non-trivial controls, and misses none of 12 designated critical cases. Each single-layer configuration correctly classifies 35 cases because the corpus deliberately isolates ten faults per layer. A grounded answer obtained through a disallowed sequence illustrates the additional coverage of trajectory checks. Repeated measurements quantify validator execution separately from process startup and reporting. These results establish coverage of the authored regression suite, not production accuracy. We describe a production-trace validation protocol, optional LLM-judge integration, and maintenance of expectations as workflows evolve. The contribution is a practical pattern for expressing behavioral contracts within an existing Java release pipeline.

0.50

DEIA: Evidence-Grounded Crash Repair with Change-Aware Test Design for HarmonyOS ArkTS

cs.SE ·2026-11-10 ·Xuhui Lu, Zhihao Liu, Xiaoke Fang, Yuhuan Fan, Yanzhe Tang +3 more
similarity 0.64
Abstract · Code and dataset: https://gitcode.com/harmony-dfx-detection/answer_B/tree/DEIA

Mobile crash repair requires more than identifying a source edit. Developers must establish how the edit relates to the reported failure and whether the modified application can be built, launched, and exercised without disrupting valid behavior. In HarmonyOS applications, this assessment also depends on platform-specific packaging and runtime execution. To address this need, we present DEIA, an evidence-grounded workflow for HarmonyOS crash repair. DEIA constructs a bounded source-evidence package, uses a large language model to propose content-anchored edits, and relies on a deterministic writer to materialize and check the patch. It derives fail-to-pass (F2P) obligations from the reported fault and pass-to-pass (P2P) obligations from the potential impact of the patch. A stage-level ledger links the patch, test obligations, runtime observations, and candidate-generation cost. We evaluate DEIA on 30 repair tasks from 24 HarmonyOS applications and compare it with an OpenHands-based workflow on 29 matched tasks. DEIA's first-added-line location exactly matches the curated reference in 25/29 tasks (86.2%), compared with 19/29 (65.5%) for OpenHands, a difference of 20.7 percentage points. DEIA also uses 81.5% fewer tokens, makes 60.8% fewer model calls, and takes 34.9% less candidate-generation time on average. Only two of the 58 behavior-specific P2P obligations were executed, both for one repair pattern; comprehensive behavior preservation therefore remains unestablished.

0.35

From Stories to Bugs: Story-Based Semantic Fuzzing of Interactive Systems

cs.SE ·2026-10-31 ·Yeshayahu Weiss, Achiya Elyasaf, Gera Weiss
benchmark similarity 0.62
Abstract · 20 pages, 4 figures, 5 tables

Complex interactive systems hide business-logic failures along two complementary axes: deep state-dependent behaviors that require long valid prefixes and concurrency-sensitive failures that depend on the ordering of otherwise legal operations. We present story-based semantic fuzzing, a methodology that treats semantically constrained compositions of behavioral stories as primary search objects. Implemented through Behavioral Programming, semantic binding and state constraints preserve legal prefixes, while event eligibility, schedule selection, and replay expose failure-relevant interleavings. We evaluate the methodology on Magento and four stateful REST APIs. On Magento, overlap-aware ranking increased reproduction of a sporadic Shipping Fee race from 3.6% to 41.2%; a controlled ordering intervention reduced a second failure signature from 100/100 initial executions to 0/200 constrained executions. For the REST study, a versioned end-to-end pipeline took each frozen public OpenAPI contract as its sole benchmark-specific model-generation input and automatically generated the Basic and Complex stories, dependency structure, and interface bindings. Under a shared black-box task and sequence-local evaluation contract, Basic and Complex confirmed 0/11 and 11/11 semantic fault classes, while RESTler and EvoMaster confirmed 2/11 and 0/11. A paired binding ablation produced 250/250 valid dependent operations with binding and 0/250 without it, while supplementary prefix controls localized the difference to preservation of legal relational state. Together, the studies show that searching admissible semantic compositions provides a unified and effective representation for reaching deep states and exposing concurrency-sensitive business-logic failures.

0.36