CS Paper Radar

Do AI Coding Agents Write Infrastructure as Code Like Humans? An Empirical Study of GitHub Pull Requests

Amine Barrak, Emna Ksontini, Charles Wehbe

cs.SE ·submitted 2026-10-25 22:20 UTC · v1

Abstract

Infrastructure-as-Code (IaC) configurations define cloud resources, networking, and access control, and AI coding agents increasingly write them. Little is known about how agent contributions differ from those of humans and automation bots, how maintainers evaluate them, and whether they introduce different sustainability concerns. We study 349,156 IaC pull requests (PRs) from 49,241 GitHub repositories, distinguishing humans, automation bots, silent agents, and disclosed agents. Bots account for 32.9% of ostensibly human PRs, and 12.3% of the remaining PRs contain agent traces, a share that grows over the study period. Agent PRs are about four times larger than human PRs and more often modify core infrastructure modules, whereas bots mainly maintain dependencies. Silent-agent PRs have higher overall merge rates than human PRs (87.4% vs. 78.9%) but are merged 4.0 percentage points less often within the same repositories. Disclosed agents create new infrastructure more often than humans (34.0% vs. 21.4%), and even for the same change type within the same repository, their PRs are 8.4 points less likely to be merged; silent-agent PRs show comparable acceptance for most change types. Finally, in 2,645 matched agent-human Terraform PR pairs, agent and human PRs introduce sustainability smells at similar rates (10.4% vs. 10.3%; paired odds ratio 1.02, 95% CI 0.83-1.25), most commonly missing remote state and monolithic configurations.

Closest papers

Nearest neighbours by meaning across the whole index.

Engineering Signals of Human-AI Collaboration in the Agentic Coding Era: A Longitudinal Analysis of 33,228 Pull Requests from vLLM and SGLang with Implications for Biomedical AI Agents and Bioinformatics Pipeline Developmen

cs.SE cs.AIcs.ETcs.HC+1 ·2026-08-14 ·Jiada Li, Xuesong Ye, Olamide Olowoniyi
code/data similarity 0.71
Abstract · 24 pages, 9 Figures

The rapid adoption of AI coding assistants and autonomous agentic development systems has coincided with major changes in the pace and structure of open-source software engineering. Yet empirical longitudinal evidence of these changes at the team level remains limited. We present a descriptive longitudinal analysis of seven engineering metrics: pull request (PR) throughput, cycle time, contributor diversity, PR comment density, merge rate, new-author participation, and PR size. Metrics were computed from all merged PRs in two high-velocity AI infrastructure repositories, vLLM (February 2023-June 2026; 18,290 PRs) and SGLang (January 2024-June 2026; 14,938 PRs). We segment development into four eras aligned with major changes in AI-assisted software development and examine human- and bot-authored activities. Both projects show substantial increases in development velocity and AI-developer collaboration signals. PR throughput increased 21x in vLLM and 17.9x in SGLang, while bot-authored PRs accounted for less than 0.2% of this growth, indicating that the increase was overwhelmingly human-driven. In the latest era, median cycle time was 1.04 days for vLLM and 0.62 days for SGLang, while P90 cycle times reached 16.8 and 14.3 days, respectively. Monthly unique authors increased steadily in both projects, suggesting broader contributor participation. PR comment density increased 4.2x in vLLM and 3.8x in SGLang, with bot comments contributing an estimated 15-20% of the increase. In contrast, PR size remained relatively stable across eras. Overall, AI-assisted development is associated with higher throughput, broader contributor participation, and increased AI-developer collaboration signals in high-velocity open-source software development.

0.26

AgentLogs: A Dataset for Opening the Black Box of Cloud Agents on GitHub

cs.SE cs.AI ·2026-08-29 ·Jonan Richards, Kosei Horikawa, Youmei Fan, Yutaro Kashiwa, Mairieli Wessel
dataset similarity 0.70
Abstract

Coding agents now complete development tasks in real software projects and submit the results as pull requests. A pull request shows the outcome of an agent's execution but not the process behind it, such as which files the agent read, which commands it ran, or how many tokens it consumed. This is recorded only in the agent's log, and no existing dataset provides such logs of real-world use, including failed agent executions, across a wide range of repositories. We introduce AgentLogs, a large-scale dataset of activity on GitHub's cloud agent platform on public repositories, collected without requiring opt-in from developers. AgentLogs comprises 363,624 tasks and 635,886 sessions, each session an execution of the agent, in 40,565 popular public repositories, with 75,631,136 log entries that record each session step by step, including prompts, intermediate reasoning, tool calls (e.g., file edits, git operations, and GitHub interactions), and token usage, together with the linked pull requests, timelines, workflow runs, and users. Sessions are included regardless of their outcome. AgentLogs enables research on agent behavior and failure modes, efficiency and cost, task formulation, human-agent collaboration, and repository adoption. A demonstration video can be found at: https://youtu.be/ZnFpjuK9Azg

0.29

Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild

cs.SE ·2026-09-12 ·Obada Kraishan
code/data similarity 0.69
Abstract · 9 pages, 5 figures, 2 tables

Autonomous coding agents now open pull requests in public repositories at a scale that was out of reach two years ago, yet little is known about what happens to that code after it lands. This paper studies 37,623 provenance-labeled pull requests (PRs) from five commercial agents (OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code) and a matched human baseline, drawn from 2,807 GitHub repositories between December 2024 and July 2025. We combine the AIDev dataset with 58,792 cached GitHub API responses to measure security smells in added code, structural maintainability, post-merge churn, revert rates, and human review behavior. Three results stand out. First, quality differences are vendor-specific rather than uniform: Codex-authored PRs were reverted about half as often as human PRs (6.1% vs. 11.5%, odds ratio 0.50), while Devin PRs were reverted more often (14.5%, odds ratio 1.31). Second, agent code pooled across vendors was less likely than human code to contain a security smell (odds ratio 0.63), driven by fewer hardcoded credentials and eval-style constructs. Third, review effort concentrates unevenly: Copilot PRs drew the most human reviews and change requests, and Claude Code PRs waited the longest for a first human review (median 12.6 hours). All pipeline code, statistical reports, and figures are released for replication.

0.20

AgentTorrent: A Dataset of Context and Harness Engineering Artifacts on GitHub

cs.SE ·2026-11-11 ·Sebastian Baltes, Seyedmoein Mohsenimofidi, Levi Böhme, Vahram Nersesyan, Muhammad Auwal Abubakar +3 more
code/data similarity 0.68
Abstract · 5 pages, 2 figures, 2 tables, under review

Over the past months, developers have moved from formulating detailed prompts to engineering the context that agentic coding tools read and to configuring the underlying harnesses. For example, developers commit context files, skills, hooks, and MCP configurations to their repositories, and these files control which instructions a harness reads and which actions it can take. Developers may also commit the specs, plans, and task files that agentic coding tools and spec-driven development frameworks write for agents to implement. The AgentTorrent dataset records these files for six proprietary and open-source agentic coding tools in 10,944 engineered software projects on GitHub. It contains 62,374 configuration artifacts, the files that these artifacts import, 27,593 agentic planning documents, and more than 1.2 million commits that credit an AI tool. All heuristics used for data collection are grounded in tool documentation or derived from GitHub repositories, and we evaluate their detection accuracy with each collection. We use open-weight models to classify whether GitHub repositories host engineered software projects. Unlike most datasets, AgentTorrent links configuration artifacts, imported files, agentic planning documents, and commits that credit an AI tool within the same repositories. Configuration artifacts, planning documents, and credited commits occur together in 388 repositories. For example, our dataset allows researchers to study whether a tool is credited in commits that add instructions for that same tool and how instruction files import and contradict each other. Our data collection pipeline supports regular updates and automatically checks vendor documentation and reports unknown artifacts and agentic coding tools. The dataset and the pipeline are open-source and are archived on Zenodo.

0.36

On the Maintenance and Co-evolution of Agent Plugins: An Empirical Study of Claude Code Plugin Marketplaces

cs.SE cs.AI ·2026-08-28 ·Ahmed Hereiz, Yingzhe Lyu, Hao Li, Bram Adams, Ahmed E. Hassan
code/data similarity 0.68
Abstract · Under review

AI coding agents, software tools that automate development tasks through reasoning and tool use, are increasingly extended through plugin marketplaces, yet the structure, maintenance, and co-evolution dynamics of these emerging repositories remain empirically unexplored. Unlike traditional software packages that deliver functionality through source code, agent plugins deliver functionality through a combination of natural-language instruction files, scripts, and configuration files, raising the question of whether these plugins are maintained artifacts that co-evolve across components, or one-off artifacts that developers write once and do not need to revisit. To study the maintenance and co-evolution of agent plugins, we conduct an empirical study of 1,926 repositories hosting Claude Code plugin marketplaces, analyzing 8,351 plugins and 77,773 commits across 2,018 marketplaces. We find that the marketplace is expanding rapidly, plugin-touching commit activity growing 8.8x over six months after the October 2025 launch, and plugins targeting Software Engineering tasks accounting for 61.3% of all plugins. Plugin development is predominantly feature-driven, with feature commits occurring at more than twice the rate of conventional open-source software (OSS) (39.6% vs. 17.2%). Claude co-authors 34.9% of all commits, and four commit types (docs, perf, style, and refactor) carry substantially different meanings in plugin repositories than in traditional software. Most component types evolve independently, but within skills directories, natural-language instruction files and implementation scripts co-evolve at above-chance rates, with 78% of co-changes being functionally coupled, representing a new class of maintenance dependency not observed in traditional software engineering.

0.41

Who Delegates to AI? Evidence from Agent Configurations in Github

cs.AI cs.CY ·2026-08-19 ·Hyeongjae Lee, Jihyang Cheon, Lanu Kim
similarity 0.68
Abstract

A growing body of literature measures the extent to which occupations are exposed to AI, yet existing measures capture where AI could perform tasks rather than whether workers have actually adopted it. We introduce a distinct tier of exposure, delegated exposure, which records whether a worker has committed a task to AI by embedding it into a structured workflow. We operationalize this concept through the Agentic Adoption Index (AAI), measuring how closely an occupation's tasks align with the agentic routines that practitioners have built and shared. Using semantic embeddings of roughly 888,000 agent skill specifications from public GitHub repositories, we compute their similarity to nearly 18,000 O*NET task statements and aggregate these scores to the occupational level. We present three main findings. First, the occupations where task delegation concentrates differ sharply from those identified as most vulnerable by pre-AI automation frameworks. Second, the AAI aligns more closely with measures of technical capability than with measures of current conversational LLM use. Third, for occupations requiring a bachelor's degree or less, the AAI increases alongside average wage levels; however, this relationship reverses for occupations requiring a master's degree or higher, where adoption declines among higher earners. These patterns replicate on an independently collected corpus of agent skills from the Manus Skills Marketplace. This lower adoption among highly educated, high-earning workers may reflect tasks that inherently resist advance specification or professional discretion over the pacing of workflow codification. Distinguishing these mechanisms will require longitudinal measurement.

0.38

A Few Pages of Markdown: Committed AI Configuration and Lower Quality Cost after Coding-Agent Adoption

cs.SE cs.AI ·2026-08-26 ·Yegor Denisov-Blanch, Shyam Agarwal, Pavel Azaletskiy, Hao He, Rylan Schaeffer +3 more
top venue similarity 0.68
Abstract

Coding agents increase development velocity but also technical debt. Prior work reports only average effects across adopters, hiding wide differences between teams. We introduce RAMP (Repository AI Maturity Profile), a four-level cumulative maturity model grounded in version-controlled artifacts that teams commit to configure AI tools. RAMP runs from behavioral rules and coding standards through named agent definitions to multi-agent orchestration, with observed practice concentrated in the first three levels. Across 441 repositories the levels behave as a cumulative scale, and independent human annotation reproduces RAMP's repository-level labels on 97% of a held-out sample. Adoption is cumulative, forward-only, and set-and-forget: 73.8% of artifacts are committed once and never modified. Re-estimating an existing agent-adoption panel within each stratum, agents accelerate development regardless of maturity (28-38% more commits), but quality diverges: among agent-first repositories, where the contrast is identified, those without committed AI configuration show roughly twice the increase in cognitive complexity (+53% versus +27%) and 1.7x the increase in static-analysis warnings. Because maturity is observational, correlated engineering discipline or model capability may explain part of the gap; we present these findings as hypothesis-generating and release RAMP as a reusable instrument.

0.50

Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requests

cs.SE ·2026-09-22 ·Wannita Takerngsaksiri, Nhat Duong, Scott Barnett
code/data similarity 0.66
Abstract · 21 pages, 6 figures, 7 tables

AI coding agents now author a large share of pull requests (PRs) merged into popular open-source projects. A merged agent PR is usually considered finished work; yet, prior studies have reported issues in agent code after the merge (e.g., code smells and static-analysis issues). However, little is known about how often a merged agent PR is fixed afterward, and who actually authors the fixing. In this paper, we follow 6,774 merged agent PRs across five AI coding agents (OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code) from the AIDev-pop dataset (open-source repositories with at least 500 stars) into their follow-up fixes, against a baseline of 5,044 contemporaneous human PRs from the same repositories. We link each merge to its candidate fixes, verify every candidate with human annotators and an LLM judge that matches human-level agreement (binary Cohen's Kappa=0.78 against a human-human K=0.77, Direct-fix precision 90%), and attribute the fixing work at the PR and the commit level. Our findings show that (1) merged agent PRs attract verified fixes at 1.62 times the odds of merged human PRs in the same repositories over the same period of time; (2) 69.6% of verified fixes in agent merges come from the same agent; and (3) 76.4% of the verified fix PRs are agent-authored throughout all commits. These results show that agents currently largely finish their own job, but their merges still require fixing more often than human merges.

0.48