CS Paper Radar

Beyond the Document: Compiled Context and Governed Agents for Any Work Product

James Rhodes, George Kang

cs.SE cs.AIcs.IR ·submitted 2026-11-09 15:20 UTC · v1

Abstract

A document preserves the visible result of professional work but often discards the research, calculations, assumptions, evidence, and authorization boundaries behind it. We study a different unit of delivery: a Portable Agent Format (PAF) package that carries a work product together with compiled context, evidence, provenance, and rules for subsequent interaction. To test whether this context remains useful beyond the process that produced it, we evaluate four new packages, created after the hosted and offline readers were fixed: an investment proposal, fund research, financial-statement analysis, and an engineering incident review. Two packages were produced by specialized agents and two through a general-purpose assistant's packaging skill. Across 400 frozen questions and 12,000 evaluated responses, a hosted governed reader succeeds on 96% of questions versus 86% with instructions plus raw-file retrieval (Holm-adjusted p < 10^-7). On the 260 answerable questions per condition, success is 94% versus 79%; the gap varies by package. Source-material chat and instructions alone reach 48% and 74% overall. A separate 0.8-billion-parameter offline reader succeeds on 69% overall, but 54% on answerable questions. Its -1 response rate is 2.9%, combining materially wrong and unauthorized answers, versus 56% with the source and instructions alone. Ablations suggest advantages for the integrated selector and package-adapted writer, without isolating representation from training effects. The same run exposes a failure: offline retrieval resolves none of the 40 historical-record questions in one financial-data package. These findings support treating compiled knowledge as part of the delivered work product, while leaving independent standards interoperability and end-to-end governance conformance unproven.

Comment
8 pages, 1 figure, 4-package held-out evaluation; 12,000 evaluated responses

Closest papers

Nearest neighbours by meaning across the whole index.

Scientific Agent Skills: A Library of Procedural Knowledge for Research Agents

cs.CL cs.AI ·2026-08-30 ·Timothy Kassis, Vinayak Agarwal, Yuhuan He, Darshil Patel, Aubrey M. Brueckner
code/datahighly cited similarity 0.66
Abstract · 31 pages, 16 figures, 2 tables, 1 listing. v2: adds a figure of how the documented workflows compose skills, quotes the skill clauses behind…

A language-model agent asked to analyse an experiment will usually return working code. Whether the analysis is defensible is a different question. A defensible analysis depends on procedural choices: which test the field accepts, which identifier namespace is authoritative, and which caveats must accompany a result. We present Scientific Agent Skills, an open library of 163 such procedures in 16 areas of practice, including genomics, cheminformatics, medical imaging, study design and scientific communication. Each skill is a directory built around a versioned, human-readable instruction file. An agent loads the file only when a task calls for it; the directory often also contains reference material and runnable scripts. We report no task-level evaluation and no host selection rate. We measure two properties of the documentation corpus: the always-resident descriptions of all 163 skills cost 7.1% of a 200,000-token window, and the median documented workflow fits within 23.9% of it, although 29 of 46 would overflow if every reference file were loaded. Openly licensed and available at https://github.com/K-Dense-AI/scientific-agent-skills.

0.55

From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation

cs.SE cs.AIcs.HC ·2026-08-20 ·Zhijun Gao, Jing Chen
similarity 0.66
Abstract · 14 pages, 1 figure, 10 tables

Technical documentation is written for human developers, but an increasing share of software changes is now authored by autonomous coding agents. Which documents they consult, when, and what follows remain unknown. We conduct a behaviour-grounded study of agent-documentation interaction across two public datasets: 557 agentic coding sessions from SWE-chat, yielding 94,813 development events including 3,033 documentation interactions; and 33,097 agentic pull requests from AIDev, with 690,260 classified file-level change records. Four findings challenge current documentation practice. First, agents' documentation work is dominated by agent-facing artefacts: instruction files and working notes account for 60.5% of all documentation interactions, versus 10.6% for classical technical documentation and 1.3% for API references. Second, the link between consultation and code editing is unresolved: the adjacent transition probability is 0.002 and the unadjusted three-event lift 1.05, whereas a stage-adjusted model places it above unity (OR 1.33 [1.09, 1.62]); documentation creation is elevated unadjusted (lift 1.67) but its adjusted interval includes unity. Third, no explicit documentation-based validation sequence was observed, and consultation is associated with less immediate testing (lift 0.23, cluster CI 0.08-0.45; adjusted OR 0.39 [0.25, 0.60]). Fourth, consultation is self-initiated (70.2%) far more often than failure-driven (7.5%), and documentation trails code: among multi-commit pull requests changing both, code is touched first 4.7x more often. From these traces we derive a descriptive model of agent-documentation interaction as a two-lobed cycle rather than a linear journey, and show that two widely assumed properties of "agent-friendly" documentation - actionability and verifiability - lack consistent behavioural support. We release our pipeline, coding scheme, and event-level data.

0.39

DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation?

cs.SE cs.AI ·2026-09-30 ·Frances Liu, Manny Silva, Paige Calvert, Ayu Adiati, Sarah Sanders
benchmarkcode/data similarity 0.65
Abstract

We introduce DoGBENCH (Documentation Generation Benchmark), to our knowledge, the first benchmark for generating and maintaining real user-facing software documentation. It asks whether an agent can produce documentation that experienced technical writers would accept in review. The benchmark contains 292 items from open source projects, including Helm, PostHog, and Mautic. Each item gives the agent a pre-change repository and a trigger, such as a code pull request or a reported documentation gap. The agent must first decide whether the documentation needs an update. For items that need one, the agent must produce an acceptable patch in one attempt. For items that do not need updates, the agent must abstain. Task-specific rubrics, validated with project maintainers, score each patch on accuracy, completeness, reader guidance, placement, and repository conventions. The composite score combines patch quality with correct abstention, and a score of 100 means an agent meets every requirement for the task. Scores should not be interpreted as a percentage of an expert's capability. We evaluated seven agents. The highest-scoring agent reached 47.3 out of 100 on the 117-item held-out split. In a separate audit of 1,267 patches, the most common failure modes were task-completion gaps (45.5%), technical inaccuracies (36.6%), and incomplete conceptual or reference coverage (32.5%). Analysis of the corresponding trajectories identified three key patterns associated with these failures: (1) describing interfaces without examining how readers use them (36.0%), (2) missing decisive evidence and filling the gaps with plausible assumptions (33.1%), and (3) stopping after finding the first plausible documentation surface and leaving other affected pages stale (30.1%).

0.29

Cross-Provider Review as a Runtime Contract for Coding Agents: A Controlled Pilot and Fault-Injection Study

cs.SE cs.AI ·2026-10-07 ·Bowen Xu, Boyu Chen
code/data similarity 0.63
Abstract · 11 pages, 1 figure, 3 tables. Project page: https://openterminalagent.surge.sh/

Coding agents increasingly share a workstation while drawing on separate providers and subscription allowances. A second agent can inspect a completed answer, but the call spends another pool and may provide no substantive finding. We describe an advisory cross-provider review contract: distinct resource pools, bounded execution, restricted reviewer capabilities, complete input delivery, usable semantic output, explicit failure states and durable per-attempt evidence. In a controlled, agent-authored pilot of 20 paired development turns, eight had a material reviewer finding (95% exact interval 19.1-63.9%). A boundary-condition scan across both reviewer backends reproduced a previously discovered false success on partial input: four truncation levels passed historically and failed after repair. The scan also found and repaired cancellation during process reaping. In real CLI probes, Claude had no writing tools; Codex attempted writes in five of five read-only trials, each write tool failed, and no disposable repository changed. These tests cover specified paths and versions, not field reliability. A preregistered shadow study of metadata-only review allocation accrued 25 formal observations before an exact-runtime regression found a third defect: a reviewer exiting nonzero with a well-formed verdict was counted as complete. Exit status was not recorded per attempt, so exposure cannot be resolved retrospectively. The 25 formal and two pending records remain an audit cohort; the measurement-valid cohort restarted at zero and collection has begun. No gate result is reported.

0.39

What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion

cs.CL cs.AIcs.IR ·2026-10-21 ·Yikai Zhao, Saurabh Pandey, Pradeep Kumar Misra
benchmark similarity 0.63
Abstract · 28 pages, 5 figures

Enterprise agents act on documents. Before an agent can follow a policy, answer a compliance question or file a ticket, some ingestion layer must turn a Word file or PDF into tokens - and that layer is where an agent deployed in the wild silently loses information. We show that one layer is lost by every mainstream converter - the meaning an author encodes in how text looks - and that the loss makes agents confidently wrong rather than visibly incomplete. We build a task from 76 real policy documents: recover the requirements the author singled out, scored against the file's own OOXML properties by a deterministic matcher with no LLM judge. Formatting-blind ingestion collapses from micro F1 0.690 to 0.150 (docling) and 0.075 (plain text), a gap that replicates across six answering models from two vendors - five Claude models spanning the Opus, Sonnet and Haiku tiers, and OpenAI's GPT-5.6. An agent that cannot see the cue guesses, and docling's false-positive rate on requirements the author did not flag is four times ours. A span-level leave-one-out localises the loss to two cues - highlight (+0.302) and colour (+0.262), both Holm-adjusted p < 1e-4 - while bold, italic, monospace and strikethrough each sit within +/-0.003 of zero. The same asymmetry appears in PDF. Colour is also the cue no mainstream converter preserves and no parsing benchmark scores, and it is common: 32.8% of 6,460 documents carry a deliberate colour or highlight, 10.3% on an obligation. A pre-registered test shows that typing a cue for the agent adds +0.085 beyond merely preserving it (Holm-adjusted p = 0.049). We validate the labels against rendered page images through an independent modality (kappa = 0.72) and bound what wording alone can achieve (AUC 0.664). For agent builders the practical finding is that ingestion, not reasoning, is where this reliability is won or lost, and that the fix is cheap and measurable.

0.17

Frustratingly Easy Agentic RAG via Portable Skills: A Case Study in Customer Support

cs.AI ·2026-10-10 ·Istok Bako, Bonan Min, Tiffany Shao, Claire Cheng, Yashar Mehdad
top venue similarity 0.63
Abstract · Accepted to EMNLP 2026 Industry Track. 15 pages, 1 figure, 5 tables

Enterprise workflow automation typically requires retrieving a complete and accurate set of relevant information from large operational document corpora (SOPs, policy documents) given a specific scenario or request. In customer support (CS), for example, human specialists must determine the correct sequence of resolution steps for each case, a task that demands precise grounding in policy documentation. Meeting this bar in production typically requires complex document chunking, indexing, and retrieval pipelines, often layered with learned re-rankers, query rewriters, and bespoke agentic RAG infrastructure. We report a deployment case study of a simpler alternative: a portable skill consisting of a natural-language instruction, lightweight tools (a retrieval API, format-checking scripts), and a corpus enriched with an LLM-generated navigation layer, which is a structured "Wikipedia-style" index over the underlying documents. This agentic RAG approach is straightforward to implement (1-2 engineer days), portable across capable agentic harnesses (e.g., Claude Code), and performs strongly on the challenging task of CS resolution workflow generation, achieving 99.2% procedural equivalence to human-approved ground truth and 83.3% evidence sufficiency. It is also roughly an order of magnitude slower per case than our deployed production assistant (~90 s vs. ~8-9 s), so we claim no latency advantage. Our claim is scoped to that setting: a mature, well-curated corpus, a latency-tolerant workflow, and human review of every case. There, the skill did the work of a complex RAG stack, and maintenance became a matter of editing text and data rather than retraining models.

0.41

AIPOCH Open-Science: An Open-Source AI Research Workbench with Reproducible .science Packages

cs.SE cs.AI ·2026-10-12 ·Wen Ouyang, Bocheng Huang, Qianyu Yao, Fei Sun, Minying Deng +74 more
code/data similarity 0.62
Abstract · 23 pages; 2 main figures, 3 tables, a graphical abstract and Supplementary Figure S1

AI-assisted research leaves instructions, computations, artifact revisions and checks whose relationships matter when another researcher receives the work. We present AIPOCH Open-Science, an open-source research workbench with portable .science session packages, and specify an exchange contract for these recorded relationships. Version and assessment targets remain attributable while receiving-workspace identifiers can be remapped, eligible contents can be omitted explicitly, and imported history remains separate from a writable fork. A controlled session tested an incorrect first version, a reviewed correction, an unreviewed revision and a planned but unexecuted step. We compared complete and partial exports with source records extracted independently of the exporter, then inspected imports into separate projects within the same installation. Both conditions retained the tested version and review associations, kept the unreviewed revision distinct from earlier assessments and represented excluded content as unavailable. A documentation-rich ordinary directory also supplied evidence for all seven static inspection questions. A separate registry case connected a table to its recorded task, execution and review; offline execution of its retained script and fixed input reproduced two saved CSV tables byte for byte. The study contributes an implemented exchange contract and a controlled examination of its preservation rules, content-selection constraints and interpretation boundaries.

0.35

Where Accountability Lives: Mapping Human Responsibility to Workflow Artifacts in Agentic Software Development

cs.SE ·2026-08-16 ·Sabry E. Farrag, Youssef Saber
similarity 0.62
Abstract · 30 pages, 5 tables. Source collection of 118 archived documents and 12 processing scripts deposited at Zenodo, doi:10.5281/zenodo.21965182

Coding agents author commits, open pull requests and push code in production repositories. Responsibility is settled in two layers that do not refer to each other: platform controls gating what an agent may do, and provider terms allocating responsibility for its output. Objective: Where the two layers disagree at a workflow event, and what each states there about authority, execution, verification, consequence and record. Method: A qualitative document study of 121 items archived byte-exact: documentation for four agentic coding tools, the platform controls on their output, and eighteen policy documents from seven providers, read by deductive content analysis against an a-priori system of five dimensions and nine workflow events. An independent second coder blind-recoded the verification-mechanism classification, a complete enumeration and not a sample (Cohen's kappa 0.81, n = 13, on whether a mechanism compels; 0.75, n = 12, on who performs it). Results: Cursor's terms make the user responsible for evaluating the use of any suggestion; its documentation describes a product that performs that evaluation and records it as an approval. No collected artifact records that the verification the terms make the user's duty took place. At merge two tools compel a person, one documents an agent approving below a configured risk threshold, and one only advises. Six of the fifteen mechanisms that compel a check have a default the vendor states; nine have one this study inferred. Readers who had not made them refuted five of the eight absence claims; three survived, one materially qualified. Conclusions: The terms attach duty and consequence to output as a class, the platform records events, nothing records the duty discharged. One vendor documents a product that forms the approval judgement, and the terms allocate consequence against the artifact regardless.

0.39