CS Paper Radar

Structure for Reading, Prose for Writing: Asymmetric Structural Conditioning in Multi-Agent Document Authoring

Cheng Yu, Nikhil Mathew, Zhengjie Wang

cs.AI cs.IR ·submitted 2026-08-21 06:57 UTC · v1

Abstract

Multi-agent pipelines that author formal documents must both read a requester's forms and write against them. We report a deployed tender-response system, running an open-weights model under sovereignty constraints, and evaluate it against human-written bids the same organisation actually submitted. On a blind comparison where the system had no worked example available, an LLM judge rated its answers at least as good as the human-submitted answer on $40$ of $55$ ground-truth sections, better on $4$, missing on none, and flagged one unsupported claim in total. Classifying every gap the judge identified shows that $68\%$ were content absent from the system's own sources -- knowledge the human author held and the pipeline was never given -- so only $6$ of the $15$ adverse verdicts involve a deficiency the system could have avoided. A divergence from ground truth is more often an information-availability result than a writing-quality one, and evaluations that do not separate the two understate such systems. Against this backdrop we report a conditioning asymmetry. It is well established that rendering documents as structural markup rather than flat prose improves extraction, and we reproduce that on three reading tasks. The benefit does not transfer to conditioning: converting a bid's \emph{instruction} material from prose to nested XML dropped answer quality from $74\%$ to $48\%$ under a paired comparison. We further find that naming a forbidden construction concentrates rather than removes it -- $96\%$ of surviving defects fall in the two forms the prompt explicitly names -- and that coupling a stochastic annotation to a deterministic windowing function moves the extracted requirement count from $68$ to $51$ on a byte-identical file. Structure belongs where the model reads; prose and self-applied tests belong where it writes.

Comment
10 pages, 3 figures

Closest papers

Nearest neighbours by meaning across the whole index.

What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion

cs.CL cs.AIcs.IR ·2026-10-21 ·Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra
benchmark similarity 0.65
Abstract · 28 pages, 5 figures

Enterprise agents act on documents. Before an agent can follow a policy, answer a compliance question or file a ticket, some ingestion layer must turn a Word file or PDF into tokens - and that layer is where an agent deployed in the wild silently loses information. We show that one layer is lost by every mainstream converter - the meaning an author encodes in how text looks - and that the loss makes agents confidently wrong rather than visibly incomplete. We build a task from 76 real policy documents: recover the requirements the author singled out, scored against the file's own OOXML properties by a deterministic matcher with no LLM judge. Formatting-blind ingestion collapses from micro F1 0.690 to 0.150 (docling) and 0.075 (plain text), a gap that replicates across six answering models from two vendors - five Claude models spanning the Opus, Sonnet and Haiku tiers, and OpenAI's GPT-5.6. An agent that cannot see the cue guesses, and docling's false-positive rate on requirements the author did not flag is four times ours. A span-level leave-one-out localises the loss to two cues - highlight (+0.302) and colour (+0.262), both Holm-adjusted p < 1e-4 - while bold, italic, monospace and strikethrough each sit within +/-0.003 of zero. The same asymmetry appears in PDF. Colour is also the cue no mainstream converter preserves and no parsing benchmark scores, and it is common: 32.8% of 6,460 documents carry a deliberate colour or highlight, 10.3% on an obligation. A pre-registered test shows that typing a cue for the agent adds +0.085 beyond merely preserving it (Holm-adjusted p = 0.049). We validate the labels against rendered page images through an independent modality (kappa = 0.72) and bound what wording alone can achieve (AUC 0.664). For agent builders the practical finding is that ingestion, not reasoning, is where this reliability is won or lost, and that the fix is cheap and measurable.

0.17

When Is a Multi-Agent Code Judge Actually Grounded? Two Label-Free Measurements, and a Judge That Declines to Guess

cs.AI cs.CLcs.SE ·2026-09-23 ·Salma Roshdy Aly, Hussein Assaf, Ziad Kobti
publishedposition similarity 0.64
Abstract · Accepted at the 21st Women in Machine Learning Workshop (WiML), NeurIPS 2026

When one language model judges whether another's code is correct, it does not report the absence of evidence. It returns a confident verdict with reasoning attached, indistinguishable from a verdict it had grounds for. Multi-agent verification, which decomposes a judgment into checkable claims and verifies each against evidence, is a promising response and works well when the evidence is a set of retrieved documents. We argue such methods require two things of their evidence: it must be independent of the answer under review, and it must differ between the two candidates being compared. The second condition holds automatically with retrieved documents and stops holding in code judging. Running MARCH, a published framework unmodified over 80 condition-by-cell measurements on two code judging benchmarks, we find it declares both solutions equally good on 78 to 95% of comparisons, reaching 4.4% accuracy where the same model asked directly reaches 43.7%. Neither easier problems nor a larger judge changes this. Two measurements taken from the pipeline's own logs explain it without needing labels. Gating on one of them, the pipeline declines the comparisons it cannot make and raises its accuracy from 20.7 to 36.9% while still answering half of all comparisons. The contribution is not a more accurate judge, but a label-free way to tell when a judge has no basis for its answer.

0.38

Who Verifies the Verifiers? Accountable Semantic Consensus for AI-Mediated Web Information Verification

cs.CR cs.DC ·2026-10-24 ·Yanna Jiang, Mingyuan Li, Xu Wang, Guangsheng Yu
theory similarity 0.61
Abstract

Large language models (LLMs) increasingly decide whether Web evidence supports online claims, often by voting over several assessors whose errors are presumed independent and benign. On the open Web, retrieved pages can steer the assessors that read them, and assessors from one model family fail together. We prove that even for independent assessors whose every output sequence is possible on both valid and invalid claims, any protocol that decides only from their outputs and always reaches an agreed verdict never rejects invalid claims or rejects valid ones with positive probability. We present Astral, which pins all assessors to one committed snapshot of the Web evidence. With up to f assessors faulty, Astral accepts a verdict only when 2f+1 assessors attest to it above a confidence threshold and no signed challenge stands. On five public benchmarks with two of seven assessors adversarial, Astral lowers pooled attack success to 21.5-53.1%, against 54.7-77.8% for majority voting. The cost is coverage, as without attack 16.0-61.7% of claims receive a binary verdict and the rest are disputed for audit. Our code runs at https://anonymous.4open.science/r/Astral-55F1.

0.30

Calibration Is Not Verification: Answer-Level Conformal Guarantees for Mixture-of-Agents

cs.MA ·2026-09-22 ·Nada Rahali, Zijia Wang, Zhi-Song Liu
benchmark similarity 0.61
Abstract · 11 pages, 3 figures; revised manuscript with updated experiments and analysis

Web answer synthesis with Mixture-of-Agents (MoA) requires reliable factual support for every atomic claim, yet agreement among agents is not verification. Correlated proposers can repeat a shared misconception, and a correct specialist fact receives little support when the other proposers stay silent. We introduce C-MoA, which decomposes MoA outputs into atomic claims, scores inter-agent semantic support, and calibrates claim retention at the level of whole answers. Under exchangeability and a policy frozen before calibration, it provides distribution-free marginal control of the probability that an answer retains any claim unsupported by the reference labels. Because calibration only selects a cutoff on a fixed ranking and cannot repair shared errors, we also study CONTRA-MoA, an optional extension that adds blinded near-miss tournaments, leave-one-agent-out stability, and availability-aware fusion, and we derive a competence--discrimination law for its verifier. Across 300 splits of the FactScore biography benchmark, the mean rate of fully supported answers (Fact) exceeds all six nominal targets, but only $1.7\%$--$9.8\%$ of supported claims are retained and $30\%$--$75\%$ of answers are empty. At $α=0.10$, mean agreement retains $3.1\%$ of supported claims, about $3.4$ times the $0.9\%$ retained by the best single-proposer score. On the medical Golden Dataset, it retains $37.3\%$ at a mean Fact of $0.898$, slightly below the $0.90$ target, and on NQ mean Fact reaches $0.904$. On a separate balanced FactScore cohort, memory-only counterfactual scoring is close to chance (AUC $0.522$ on 182 claims), and raw maximum fusion lowers the agreement AUC from $0.687$ to $0.652$. These results separate the statistical calibration of factuality from informative claim verification and clarify the role of each in multi-agent Web synthesis.

0.27

The Anatomy of a Blind Spot: What an LLM Judge Must Be Shown, and How Much

cs.AI cs.CL ·2026-10-11 ·Nitish Kumar, Prateik Pothuneedi, Greg Bandak
similarity 0.61
Abstract · 50 pages, 19 figures, 7 tables

Evaluation of text generated by large language models (LLMs) is increasingly delegated to LLMs acting as judges, yet what such a judge must be shown remains unresolved. We treat a judge whose reports are acted on as a regulator, in the sense of cybernetics, and ask what it must be shown, how much, in what form, and what its reports must get right. On seven public benchmarks we varied only what the judge was shown. Across three judges whose prices differ 10- to 17-fold, what the judge was shown mattered more than which judge it was. When a system revised its answers on feedback from a judge denied the material they had to respect, it ended below its starting point on instruction following and question answering. On instruction following the loss grew with each round, and it persisted when the judge saw the user's request but not the constraints and with a second judge family. On news summarisation, a correct reference summary beside the source undid most of the source's contribution, whereas on single-answer questions a reference helped. About a tenth of the source did almost all of its work, and a small entailment graph matched an LLM judge at finding errors in grounded text. We committed 83 testable clauses before running; of the 81 that ran, 32 met their bar and 50 went in the predicted direction, and most quantitative predictions from the theory failed. We report every shortfall and derive design rules tied to measured effects.

0.17

Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA

cs.IR cs.AIcs.CLq-fin.GN ·2026-09-14 ·Luis M. Sánchez
benchmark similarity 0.60
Abstract · 28 pages, 9 figures. Frozen evidence archive: https://doi.org/10.5281/zenodo.22310532

Frontier models score well on shallow document/chart reading tasks. In a controlled data-room audit, moving evidence into buried conditions reduced accuracy, increased forced declarations, increased tool calls, and increased cost per correct answer. Confidence and benchmark calibration did not fully capture wrong answers; a documented production incident shows fabricated structural claims can be mixed with accurate numeric tables. Agentic evaluations need claim-level receipts (statement-level provenance, not answer-level scores), condition-aware scoring, and human-adversarial verification - an auditing discipline, not a leaderboard. The setting we measure is financial due diligence; the setting we are building toward next is defense staff work, where the same buried-evidence shape appears. In both, the model is not a party to the consequences; the person who signs is. In plain terms: in the documented cases we examine, agents can pair accurate numbers with confident fabricated explanations, and the burden of proof must therefore move from the model to the evidence trail.

0.27

LabourCrew: A Multi-Agent RAG Framework for Trustworthy Adversarial Deliberation and Statutory Reasoning over Labour Law

cs.CL cs.AIcs.IR ·2026-08-18 ·Fatema Tuj Johora Faria, Mukaffi Bin Moin, Jubayer Al Mahmud, M. F. Mridha, Md. Alam Hossain
similarity 0.60
Abstract

In statutory question answering, every claim must be traceable to evidence, not merely relevant, since unverifiable labour-rights answers carry serious legal consequences. Current systems fall short: single-pass RAG cannot detect insufficient evidence, while multi-agent legal-debate systems treat grounding as a prompting convention, letting agents cite unretrieved evidence. To address this gap, we introduce LabourCrew, a multi-agent RAG framework built around three grounding mechanisms: StatuteGraph, a graph index that explicitly links chapter, section, proviso, and cross-reference structure rather than fixed-length spans; an Evidence Exchange Protocol that confines advocates and an interpreter to an evidence ledger, making citation to unretrieved text impossible, while a fault-tolerant supervisor board runs advocates in parallel so individual failures degrade rather than crash the system; and a Calibrated Trust Gate that replaces categorical accept/reject decisions with a trust score, thresholded via conformal risk control for a distribution-free bound on the false-accept rate. We evaluate on LabourActQA, a 500-item Bangla question set from the Bangladesh Labour Act, 2006, spanning seven reasoning categories and three difficulty tiers. The framework drives the empirical false-accept rate to 0.081, within the target level ($α= 0.10$), achieves the highest Answer Relevancy among HyDE RAG, Graph-RAG, and Hierarchical RAG (0.862 $>$ 0.839, 0.815, 0.828), and degrades gradually rather than catastrophically as question difficulty increases. These results show that calibrated abstention, not retrieval quality alone, is what makes legal question answering auditable in low-resource statutory domains.

0.21

Copies or Sources? Measuring How LLM Aggregators Count Restated Evidence in Multi-Agent Systems

cs.AI ·2026-10-05 ·Jianxin Gao, Runze Li, Tianyi Yu, Liangwei Ren, Bohan Chen +1 more
similarity 0.59
Abstract

Multi-agent systems built on large language models (LLMs) restate observations as a matter of course: relays forward them, shared boards repeat them and discussion rounds echo them. An aggregator that pools such messages should count sources, not statements. We convert a reported probability into units of independent readings, which assigns every restatement a copy weight, 0 for an aggregator that counts sources and 1 for one that counts every statement, and yields the implied decision under any cost structure. Three testbeds hold the evidence fixed and vary how it is restated: message logs with an exact Bayesian oracle, web documents with appended copies, and logs written by LLM agent teams under four communication protocols. Across four models from three providers, a forwarded copy counts for 0.06 to 0.42 of a new reading, mostly because some replies count every statement. On 5% to 40% of logs that state one reading three times, the reported belief implies an early commitment that the oracle never makes. The models that count copies least and most on controlled logs do so on web copies and agent-written logs as well. A one-paragraph declaration of what a copy contributes brings the copy weight on controlled logs to 0.08 or less. A rule that has agents refer to readings instead of restating them cuts belief-implied early commitment from 11.2% to 1.1% and preserves genuine corroboration.

0.37