CS Paper Radar

Maat: Independent Deterministic Contract-Based Governance for Multi-Agent LLM Workflows

Uliana Elina

cs.SE cs.AIcs.MA ·submitted 2026-09-27 23:23 UTC · v1
benchmarkcode/data

Abstract

Large-language-model multi-agent systems (LLM-MAS) introduce a characteristic reliability problem: an error produced by one agent can be accepted as context by downstream agents and propagate across the workflow. Many proposed safeguards rely on learned or LLM-based judges whose verdicts are themselves probabilistic; we ask whether a deterministic layer can instead stop contract-detectable handoff defects. We present Maat, a runtime governance layer that validates agent-to-agent handoffs against a versioned workflow contract, or anchor, with no language model in the validation or scoring path. We evaluate it in six controlled domain workflows (6-15 agents, 522 trials) with injected data-level defects and a deterministic seven-check rubric. Version 1 reported gains in all six workflows (2.9-26.5%). A post-publication audit found that three benchmark scorers credited any early halt as a prevented defect. On paired trials where the governed run completed or halted on a finding attributable to a verified defect, the rubric score changes by +7.7% to +29.1% in five workflows and is flat in software development; model-call cost falls 17-53% where attributable halts occur early. A hand review of all 94 governed-arm halts found 35 false alarms (37%), caused by validator defects rather than model behaviour; counting those halts as failed work, the governed arm scores below the ungoverned arm in four of six workflows. The results support deterministic handoff validation for contract-expressible defects and show that validator configuration and halt attribution must themselves be tested; they do not establish universal correctness, hallucination detection, or model-independent effectiveness.

Comment
16 pages, 6 figures, 6 tables. Benchmarks: https://github.com/Lorelys/maat-benchmarks ; CrewAI integration demo: https://github.com/Lorelys/maat-crewai-demo

Closest papers

Nearest neighbours by meaning across the whole index.

Structure Before Intelligence: Gating What Agentic Scientific Workflows Release

cs.AI q-bio.QM ·2026-10-14 ·Ashish Acharya, Aman Al Amin, Hong Qin
similarity 0.70
Abstract · 21 pages: 10-page main paper and 11-page supplement; 2 figures

An agentic scientific workflow can run to completion and still release wrong results: a silent numerical error, a stale input, or a defect in a shared upstream step can contaminate several outputs while a sibling branch stays correct. We treat trust in such results as a release decision. A harness around the workflow's directed acyclic graph (DAG) decides which outputs may be used: it enforces mandatory checks, contains failures to the affected branch, recomputes from tracked versions, adds validators, and releases a result only if a risk model estimates its probability of being invalid as low. Language-model agents work inside the harness, choosing corrective actions and reviewing releases, but cannot override it. On 60 held-out paired instances of two workflows with injected faults, we switched components on and off. Each supplied a different property. Containment and recovery raised the number of correct results. Validation supplied safety: fixed rules without validation released 37 invalid results, while validating every candidate released none. Risk estimation supplied efficiency, matching that safety with 31% fewer validators at somewhat lower yield, although better targeting than random validation was not shown. Lineage evidence helped on one workflow. The language model added no measurable benefit: whether invalid results escaped depended on the harness, not the model. This proof of concept points to a practical path: placing agents inside explicit release-control structure can keep released scientific results trustworthy, and it gives more capable agents a measurable baseline to improve on.

0.45

Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution

cs.AI ·2026-10-13 ·Fengrui Fan, Huahao Wang, Jingyi Fang, Yutao Hu, Yueming Wu +1 more
benchmark similarity 0.69
Abstract

LLM agent frameworks heavily rely on external skill assets to standardize tool usage, encode operational workflows, and supply executable runtime resources. Despite their widespread adoption, multi-component skill packages frequently suffer from cross-artifact inconsistencies, where documentation, scripts, and configuration files specify conflicting behavioral constraints. Although static auditing techniques can uncover such contractual discrepancies, static defect volume cannot reliably reflect real-world agent performance degradation. This gap leaves practitioners without rigorous evidence to assess the practical risk of flawed agent skills. This paper presents a systematic empirical study to demystify the runtime implications of skill inconsistencies. We conduct large-scale auditing over 575 community and benchmark skills, yielding 1,182 manually validated contract inconsistencies. We further build a controlled evaluation pipeline with 49 executable agent tasks and three representative LLMs across NoSkill, OriginalSkill, and FixedSkill experimental settings. Through fine-grained behavioral analysis of 441 paired execution trajectories, we quantify the actual performance impacts of resolved defects.

0.13

Who May Overrule the Agent? Evidence-Gated Authority in LLM Review Institutions

cs.AI cs.MA ·2026-09-12 ·Fuad Ali
benchmarkcode/data similarity 0.69
Abstract · 11 pages, 3 figures, 3 tables. Code, frozen outputs, and reproduction instructions: https://github.com/tofuadmiral/institutional-representat…

Multi-agent language-model systems often add a reviewer to improve an upstream agent's decision. Yet a reviewer can both repair errors and replace decisions that were already correct. We study this as a problem of institutional jurisdiction: conditional on the same proposal, task information, model, and review output, when is the reviewer authorized to make its recommendation binding? We compare broad override authority with an evidence gate that permits replacement only after the reviewer produces a mechanically valid record of a binding-constraint violation and a compliant repair. Across two locally run open-weight models, 96 spatial-policy tasks, and a 96-task portfolio-transfer benchmark, the gate sharply reduces corruption of correct proposals while retaining most repairs of violations. The directions of this tradeoff are partly implied by the rule; its magnitude depends on reviewer transitions and upstream proposal states. A prospective validation then gives the reviewer an explicit choice to retain, replace, or escalate. This eliminates correct-proposal corruption for Qwen, but Mistral still replaces all 48 correct proposals; Qwen also declines every welfare-only repair that Mistral frequently attempts. Thus interface design changes reviewer behavior but is not a model-independent safeguard. The result is not that restricted review is universally better. It is that reviewer competence, action interface, and binding jurisdiction jointly determine both the correction domain and the corruption surface.

0.30

Evaluating Governed LLM-Driven ML Experimentation: Lifecycle, Provenance, and Failure Metrics

cs.AI cs.LGcs.SE ·2026-11-18 ·Dowling Wong
publishedbenchmarkcode/data similarity 0.68
Abstract · 22 pages. Presented at the KDD 2026 Workshop on Evaluation and Trustworthiness of Agentic AI (ETAAI; non-archival). Code and trial ledgers: …

Autonomous agents that run ML experiments are usually judged by the best score they reach. That score says nothing about how it was reached: whether the agent stayed within the files it was allowed to edit, whether failed trials were kept or silently dropped, and whether each keep or discard decision can be traced to its evidence. We propose evaluating such systems on their experimental process as well as their outcome, and we build a governed harness that makes this possible. A control plane, not the LLM that proposes changes, owns each trial's lifecycle, checks its edit scope, decides keep or discard, and appends one record per trial to an append-only ledger. Governance metrics such as the invalid-action rate, the repeated-bad rate and provenance completeness are computed from that ledger. Across six ML tasks, from synthetic logistic regression to language-model training, and 1,445 trials, the same contract applied unchanged. The ledger exposed failures that a task score hides: a no-memory arm in which all 120 trials failed, a benchmark adapter whose bounded proposals ran out, and a reset failure that reproducibility hashes revealed, after which the affected campaigns were rerun. In a matched counterfactual, a loop that logs only successful trials left no record of 43 of 50 trials, while the governed ledger recorded all 50. Memory ablations show that memory's effect on repeated bad proposals depends on the task. We release the ledgers of all 1,445 trials, so that every number reported for them can be recomputed.

0.50

AgentTrustCI: Build-Time Policy, Grounding, and Trajectory Tests for Java LLM Agents

cs.SE cs.AI ·2026-11-04 ·Sachin Gupta
published similarity 0.68
Abstract · Accepted for publication in the 2026 International Conference on Computer and Applications (ICCA)

Infrastructure security does not establish whether an LLM agent selected an allowed tool, grounded its answer in tool output, or followed a required workflow. AgentTrustCI is a Java/JUnit 5 framework that expresses these application-layer expectations as policy, grounding, and trajectory checks over recorded traces. A deterministic replay implementation supports repeatable build-time tests; a judge interface provides an extension point for optional semantic evaluation. On 55 synthetic cases spanning seven enterprise domains, the full pipeline correctly classifies all 30 injected faults and 25 passing controls, including ten non-trivial controls, and misses none of 12 designated critical cases. Each single-layer configuration correctly classifies 35 cases because the corpus deliberately isolates ten faults per layer. A grounded answer obtained through a disallowed sequence illustrates the additional coverage of trajectory checks. Repeated measurements quantify validator execution separately from process startup and reporting. These results establish coverage of the authored regression suite, not production accuracy. We describe a production-trace validation protocol, optional LLM-judge integration, and maintenance of expectations as workflows evolve. The contribution is a practical pattern for expressing behavioral contracts within an existing Java release pipeline.

0.50

When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems

cs.SE cs.AIcs.CR ·2026-10-17 ·Shihao Weng, Longtian Wang, Yang Feng, Mingfei Cheng, Jia Liu +4 more
code/data similarity 0.68
Abstract · 18 pages

Software quality has long rested on specifications that state what code must do, and on testing and verification that check that the code does it. Agentic systems change this foundation: they replace parts of their logic, such as choosing the next tool, judging whether an action is allowed, or writing code to run, with calls to a large language model (LLM). The program still relies on what this logic must guarantee, but the model's behavior is not fixed by code, varies across runs, and can be steered by its input. The guarantee must therefore come from the program around each LLM call site, as a contract of pre-conditions, prompt constraints, and post-conditions. We present an empirical study of these contracts in 496 LLM call sites of 40 popular open-source agentic systems (13.4 MLOC, four languages). Using a multi-agent pipeline validated by 17,428 human judgments, we trace where every model output flows, derive an expected contract for each call site without access to its code, and compare it with the contract that the code implements. We find that 48% of LLM outputs reach control flow or execution, and 73% of these never reach a person. Programs enforce 78% of their constraints in code, but in the prompt, 67% of the constraints are only requested from the model. Although 82% of call sites are checked before and after the call, 63% of the expected constraints are unmet and only 5.3% of call sites satisfy their whole contract; ten recurring patterns explain 99% of the gaps. 38 mechanisms behind these gaps trigger at runtime, and a small code fix that enforces the expected constraint stops all 38. Our results show that the presence of checks says little about their adequacy, and we derive implications for agent developers, framework maintainers, and researchers.

0.31

AgentEval: Evidence-Bounded Assessment of Stateful Agents

cs.AI ·2026-10-15 ·Ruiqi Gao
benchmark similarity 0.68
Abstract · 23 pages, 6 figures

Task completion alone does not determine whether a stateful agent satisfied its obligations, and a recorded judgment does not guarantee that its supporting evidence remains valid. We present AgentEval, a system that carries a contract-bound assessment record through tool execution, state and event capture, report qualification, and case maintenance. It separates what the agent did, what the evidence establishes, and whether the conclusion is supported for present use. On 32 tool-world administrations, independent checks agree on all 30 complete episodes, distinguishing satisfied, violated, and unresolved obligations; two interrupted administrations are retained. Controlled interventions on 48 evidence views expose 36 binding faults missed by hash-only checks without changing agent behavior. Scenario Studio connects reviewed cases to 12 two-turn reports with 24 independently checked answers. A separate public-benchmark study on 192 held-out executions reveals ten completed return subgoals within broader task failures and isolates the effect of missing evidence. These studies show how an executable assessment record makes scope, evidential support, and report reuse inspectable across an agent evaluation workflow.

0.17

When "Must" Becomes "Maybe": Erosion of Workflow Constraints in LLM Agent Handoffs

cs.AI cs.MA ·2026-08-25 ·Yiheng Sun, Huifei Wang, Yancheng Zhu, Zhenyu Li, Zebin Zhao +1 more
similarity 0.68
Abstract · 16 pages, 4 figures. Revised title, abstract, and exposition; expanded discussion and reorganized appendix

Large language model (LLM) agents build on one another's work through summaries, plans, and other intermediate artifacts. These records let a downstream agent continue without revisiting the complete upstream context. We uncover a hidden loss in this process: handoffs can carry tasks forward while eroding judgments already established upstream. A requirement that must be satisfied before action becomes a caveat that the next agent may acknowledge and set aside. The issue remains visible, but its role has changed. We define operational state preservation as retaining an established judgment's role in constraining downstream action. Safety blockers provide a concrete instance: an unresolved prerequisite restricts the admissible actions until the condition is resolved. In our experiments, we confirm correct upstream detection, hold source permissions fixed, and vary the handoff supplied to a local-view executor. Direct-state controls preserve every blocker, whereas compression, plan assimilation, convergence, ownership deferral, and precedent substitution repeatedly weaken them. Ownership deferral produces 76.7% blocker deactivation; multihop compression produces 97.2%. Measuring the transmitted state separately from endpoint action exposes two routes to behavioral recovery. In fixed-artifact replays, restoring the full blocker description raises recognition from 6.2% to 99.2% and reduces forbidden action from 85.2% to 0.0%. An authorization-check prompt instead reduces forbidden action to 1.6% while leaving artifact deactivation at 95.3%. A successful endpoint can therefore conceal a damaged handoff. These findings make maintaining earlier judgments a core capability of reliable collaboration: each stage must carry forward both what earlier work established and what it requires of subsequent action. Semantic availability does not guarantee operational preservation.

0.51