CS Paper Radar

AI-to-AI Code Reviews of GitHub Pull Requests

Niruthiha Selvanayagam, Taher A. Ghaleb

cs.SE ·submitted 2026-08-21 17:17 UTC · v1
top venuedataset

Abstract

AI coding agents are increasingly integrated into software development workflows, operating on both sides of the pull-request (PR) process: AI authoring agents create or modify PRs, while AI reviewers evaluate them. This creates a closed loop in which one AI coding agent reviews a contribution attributed to another. We construct a large-scale dataset of AI-to-AI code review by linking AI-attributed PRs with AI-attributed review events from CodAGE, a public dataset of coding-agent-generated GitHub events. Our dataset contains 248,641 unique AI-attributed PRs that received at least one AI-attributed review. Of these, 45,269 received cross-product review and 208,145 received same-product review; 4,773 PRs received both. Cross-product AI-to-AI review occurred in approximately 1.6% of identified agent-authored PRs but was substantial in absolute terms, and its volume increased by more than two orders of magnitude from 2025-Q1 to 2025-Q3. Reviewer output varied across author-reviewer configurations. CodeRabbit labeled 35.0% of its comments on Claude Code-authored PRs as refactor comments, compared with 10.5% on Copilot-authored PRs, although this difference may reflect characteristics of the PRs rather than the reviewer. For three of four dual-role reviewers, mean comments per PR were 58-65% higher in the same-product group, although effect sizes were small or negligible and the difference was concentrated in the upper tail. Among pairs with complete, nonnegative timestamps, the observed median latency was 1.2 minutes for cross-product pairs and 4.7 minutes for same-product pairs; differential timestamp availability and reviewer composition limit this comparison. Overall, closed-loop AI-to-AI review is increasing but remains a minority of identified agent activity, with review output varying across authoring-agent groups and product configurations.

Comment
Accepted at the 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026), Emerging Results, Vision, and Reflection Papers Track

Closest papers

Nearest neighbours by meaning across the whole index.

Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requests

cs.SE ·2026-09-22 ·Wannita Takerngsaksiri, Nhat Duong, Scott Barnett
code/data similarity 0.75
Abstract · 21 pages, 6 figures, 7 tables

AI coding agents now author a large share of pull requests (PRs) merged into popular open-source projects. A merged agent PR is usually considered finished work; yet, prior studies have reported issues in agent code after the merge (e.g., code smells and static-analysis issues). However, little is known about how often a merged agent PR is fixed afterward, and who actually authors the fixing. In this paper, we follow 6,774 merged agent PRs across five AI coding agents (OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code) from the AIDev-pop dataset (open-source repositories with at least 500 stars) into their follow-up fixes, against a baseline of 5,044 contemporaneous human PRs from the same repositories. We link each merge to its candidate fixes, verify every candidate with human annotators and an LLM judge that matches human-level agreement (binary Cohen's Kappa=0.78 against a human-human K=0.77, Direct-fix precision 90%), and attribute the fixing work at the PR and the commit level. Our findings show that (1) merged agent PRs attract verified fixes at 1.62 times the odds of merged human PRs in the same repositories over the same period of time; (2) 69.6% of verified fixes in agent merges come from the same agent; and (3) 76.4% of the verified fix PRs are agent-authored throughout all commits. These results show that agents currently largely finish their own job, but their merges still require fixing more often than human merges.

0.48

Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild

cs.SE ·2026-09-12 ·Obada Kraishan
code/data similarity 0.73
Abstract · 9 pages, 5 figures, 2 tables

Autonomous coding agents now open pull requests in public repositories at a scale that was out of reach two years ago, yet little is known about what happens to that code after it lands. This paper studies 37,623 provenance-labeled pull requests (PRs) from five commercial agents (OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code) and a matched human baseline, drawn from 2,807 GitHub repositories between December 2024 and July 2025. We combine the AIDev dataset with 58,792 cached GitHub API responses to measure security smells in added code, structural maintainability, post-merge churn, revert rates, and human review behavior. Three results stand out. First, quality differences are vendor-specific rather than uniform: Codex-authored PRs were reverted about half as often as human PRs (6.1% vs. 11.5%, odds ratio 0.50), while Devin PRs were reverted more often (14.5%, odds ratio 1.31). Second, agent code pooled across vendors was less likely than human code to contain a security smell (odds ratio 0.63), driven by fewer hardcoded credentials and eval-style constructs. Third, review effort concentrates unevenly: Copilot PRs drew the most human reviews and change requests, and Claude Code PRs waited the longest for a first human review (median 12.6 hours). All pipeline code, statistical reports, and figures are released for replication.

0.20

Reviewing Beyond the Change: Repository Size and Existing Code Across Agent and Human Pull Requests

cs.SE ·2026-10-11 ·Sabry E. Farrag
similarity 0.72
Abstract · 45 pages, 3 figures, 7 tables. Data and code: https://doi.org/10.5281/zenodo.23290164

Reviewers sometimes name existing repository items outside the change under review. We ask whether such references are more common in larger repositories, and whether the pattern differs between agent-authored and human-authored pull requests. An instrument resolves names in inline review comments against the base-commit repository; size-matched negative controls estimate chance matches. Among 92,042 comments on agent pull requests in 2,034 GitHub repositories, 6.5 per cent name an existing item outside the change. An internally pre-specified log-linear model finds an odds ratio of 1.35 per tenfold increase in code files (interval 1.15 to 1.59), and size-matched negative controls indicate that chance matching does not account for the rise. Exploratory analyses suggest that the association is concentrated from small to mid-sized repositories and is imprecise among large ones. Human pull requests in 1,845 of these repositories show a similar association. A pre-specified comparison finds a small agent excess just within the bound set as negligible, but the evidence does not establish a robust agent-specific difference. The instrument misses many references, so reported shares likely understate prevalence, and the validation cannot estimate how recall changes with repository size.

0.29

Agents on the Patch Path: Mining AI-Associated Security Remediation in GitHub Security Advisories

cs.SE ·2026-11-06 ·Claudio Segal, Paulo Segal, Hudson Silva Borges, Paulo Silveira, Daniel S. Menasché +6 more
similarity 0.72
Abstract

Autonomous coding agents are increasingly involved in software development, but their participation in real-world security remediation remains poorly understood. We investigate observable evidence of coding-agent involvement in commits referenced by reviewed GitHub Security Advisories (GHSAs), characterize the associated vulnerabilities, and examine review timing in relation to advisory disclosure routes. We develop a local-Git mining pipeline that identifies agent evidence through known agent accounts, explicit commit-message attribution, repository instruction files, and agent configuration artifacts. Linking these findings to GHSA-referenced commits yields 469 valid candidate advisories across 189 repositories, including 448 with direct attribution and 21 supported only by repository-level evidence. Claude is the most frequently observed tool family, followed by Copilot and Cursor. We further compare review timing against other advisories and historical baselines using an exact route--timing decomposition. Relative to advisories published in the same months, candidate advisories exhibit a modestly longer mean review lag, primarily associated with within-route timing rather than disclosure-route composition. Compared with a historical baseline, they exhibit substantially shorter mean review lags, with both route composition and within-route timing contributing to the difference. These findings demonstrate that coding-agent involvement is already observable across hundreds of security advisories and repositories, including high- and critical-severity vulnerabilities. By distinguishing direct attribution from repository-level adoption evidence and separating route-composition effects from within-route timing differences, we provide an empirical characterization of agent-associated remediation in the GHSA review pipeline.

0.17

Level Up! AI-Driven Challenges and Adaptations to Code Review Practices

cs.SE ·2026-11-13 ·Kaia Newman, Christian Bird, Jenna Butler, Emerson Murphy-Hill
survey similarity 0.71
Abstract

AI can write code faster than humans can review, causing a bottleneck and necessitating changes to code review practices. This bottleneck is hotly debated, giving rise to two extreme and orthogonal imagined futures: one where humans continue to be firmly entrenched in the review process, and one where AI agents merge pull requests entirely without human involvement. Yet, these futures remain rooted in folk wisdom. Through a mixed-methods study of interviews ($n = 24$) and a survey ($n = 2{,}590$) at a large, multinational software company with widespread access to AI, we identify AI-driven code review challenges and strategies reported as effective to meet them. We found that AI enables, necessitates, and complicates a shift in the human reviewer's role toward high-level concerns (e.g., design and intent). We found that AI-assisted authoring results in increased burden for reviewers due to challenges such as a lack of author stewardship and PR volume and size, potentially leading to downstream debt. Our survey showed AI review usage was not correlated with less difficulty keeping up with PR volume, indicating using AI for review alone is not sufficient for solving AI-driven review challenges. Developers found the most useful strategies to be organizations supporting and resourcing review, pushing review burden upstream, and prioritizing reviewer attention on the concerns that matter most.

0.25

What Open-Source Projects Tell Their Coding Agents: A Census of AGENTS.md and CLAUDE.md

cs.SE cs.AI ·2026-10-22 ·Pawel Zarzycki
similarity 0.70
Abstract · Corrected abstract metadata formatting and restored missing numerical values; paper unchanged

Coding agents can be guided with instruction files in a repository. We examine this emerging practice through a census of 22,854 popular, active GitHub repositories in thirteen languages, treating each repository's root 'AGENTS$.$md' and 'CLAUDE$.$md' as one instruction set. Agent instructions appear in 26.3% of repositories and 59.6% of the most-starred; where both files exist, 'CLAUDE$.$md' usually points to 'AGENTS$.$md'. We analyse 341,852 statements using model-assisted annotation and human review. The annotations reveal a practical mix of project knowledge and rules about code, building and workflow. Collaboration rules name concrete requirements, from test plans to commit formats, yet 74.0% of directives give no reason. Projects disagree on AI attribution: commit-message rules more often prohibit it than require it, while pull-request rules more often require disclosure. A third of instruction sets have a generator-associated signature, a tool-inserted block, shared text or an AI-marked creation commit; four in five are edited after creation. Prespecified tests associate repository popularity with rules on pull requests and AI contributions. Exploratory analyses link activity to process-oriented content and show that LLM and agent projects are over-represented among adopters. The models agree much more closely with each other than with the human reviewer's topic assignments, a caution for automated annotation. We release the annotations, taxonomies and pointers to the analysed file versions.

0.42

CodeHealth Predicts the Token Cost of Agentic Code Review: A Pre-Registered Study of 465 Agent-Authored Pull Requests

cs.SE cs.AI ·2026-10-15 ·Lo Gullstrand Heander, Markus Borg, Emma Söderberg, Christofer Rydenfält
similarity 0.69
Abstract · 32 pages, 6 figures, 16 tables

Context: Organizations increasingly delegate code review to multi-agent large language model (LLM) pipelines billed per token. Low code quality raises human maintenance effort, but its effect on agent review cost has not been measured. Objective: We test whether baseline CodeHealth predicts the token cost of agentic review and fix beyond pull request (PR) footprint. Method: In a pre-registered observational study, we sampled 465 agent-authored PRs from 246 repositories in AIDev, jointly stratified by CodeHealth and footprint under repository caps. A frozen five-agent review-and-fix pipeline processed every PR with three different LLMs. Per LLM, we fitted a mixed model of log tokens on CodeHealth and log PR footprint, with a repository random intercept. Results: For all three LLMs the CodeHealth coefficient was negative, with 95% CI excluding zero; each CodeHealth point was associated with 3.5% to 6.1% lower token cost. CodeHealth was unrelated to PR footprint, so PR size does not explain the association. Two of three registered smells predicted fix cost; batch fixing left median CodeHealth unchanged. Exploratory analyses located the association in the review stage, where agents iterated more on unhealthy code without writing more comments. Adjusting for the size of the scored files absorbed the association, which we interpret as mediation by file size. Conclusions: A quality metric calibrated against human maintainability also predicts the inference cost of multi-agent review and fix. The effect is modest per point and not carried by PR size, so the refactoring case extends to agent-reviewed code when it changes structure and file size, not only the score. Data Availability: https://doi.org/10.5281/zenodo.22127920

0.25

Reliable Mining of Heterogeneous AI Coding-Agent Data: Context-Aware Prediction and Resource-Aware Ranking

cs.SE ·2026-11-15 ·Haoran Yu, Lifei Liu, Danping Zhang
similarity 0.69
Abstract

Operational records from AI coding agents differ across agents, repositories, tasks, and workflows. We study 71,677 pull requests (PRs) from six agents and show that pooling these records can reverse descriptive and predictive conclusions. PRs carrying a co-authorship trailer have a 55.6% merge rate, compared with 77.5% without the trailer, but removing the dominant agent flips this -21.8 percentage-point gap to +23.7 points. We treat this Simpson-type reversal as a data-veracity problem and represent source, environment, and workflow context in a feature cascade. Gradient-boosting merge-prediction AUC increases from 0.774 to 0.832; the advantage persists under temporal evaluation (0.801) and unseen repositories (0.807). The co-authorship coefficient changes from -1.09 to +1.44 after conditioning, whereas repository-clustered AIPW intervals show no stable adjusted trailer association. A strict PR-creation-time model reaches 0.741 AUC in standard evaluation and 0.710 on unseen repositories, although its temporal AUC falls to 0.641. Finally, a separately cross-fitted analysis of an observed-mode functional estimates a +12.3-point doubly robust gain at a 30% allocation budget on the non-Codex population, only 0.5 points above risk ranking. This last result is retrospective decision support over metadata states, not a causal effect of assigning human review.

0.26