CS Paper Radar

What Will Replace Pull Requests? Rethinking Software Collaboration for AI Agents

Christoph Treude, Sebastian Baltes

cs.SE cs.HC ·submitted 2026-10-26 07:00 UTC · v1
code/data

Abstract

The emergence of artificial intelligence (AI) coding agents has put pull requests under pressure: agents open millions of pull requests per month, while some open-source projects have begun to restrict or disable external contributions. We investigate which alternatives to pull requests practitioners are proposing or adopting and what these changes mean for empirical software engineering research. We reviewed 50 gray-literature documents from 31 organizations or individuals published between October 2025 and October 2026, coding the problems each named, the mechanisms it proposed or adopted, which of five roles of pull requests (integration, review, rationale, accountability, and external contribution) each mechanism takes over, and the evidence offered. Most responses retained pull requests in some form, whether by adding tooling, attaching a new artifact, or restricting access, and restriction was the response most often reported as adopted. Several proposed alternatives moved review and rationale into agent conversations and session records, but only Zed reported using a replacement in practice. Only two documents outlined a replacement for external contribution, and neither reported adoption. Pull requests are changing in ways that pull-request datasets will not capture: reviews conducted in agent conversations leave no pull-request comments, and projects that disable pull requests disappear from samples of newly opened ones. We outline a research agenda on the effectiveness of the emerging alternatives, their consequences for open-source participation, and the adaptation of empirical methods.

Closest papers

Nearest neighbours by meaning across the whole index.

An Empirical Study of Maintainer-Identified AI Slop Pull Requests in Free and Open Source Software Projects

cs.SE ·2026-11-11 ·Hugo Raskin, Xavier Devroey, Benoît Vanderose
code/data similarity 0.71
Abstract · 12 pages

The FOSS ecosystem has always relied on outside contributions, notably through issues or Pull Requests (PRs), to sustain itself. With the advance of GenAI, the barrier to contributing to FOSS projects has never been lower, leading to the emergence of AI slop PRs: low-quality, low-value generated PRs. Past studies have examined AI slop PRs through theoretical, platform-level, or practitioner-account frames, but not at the project level. To this end, this study investigates the incidence of maintainer-identified AI slop PRs in FOSS projects by focusing on PRs flagged as slop by maintainers. Using a mixed-methods approach, we trace the origins of maintainer-identified AI slop PRs, quantify their volume in FOSS projects, describe their characteristics, and highlight maintainers' reasons for flagging or closing them. Our results show that the volume of maintainer-identified AI slop PRs increased in mid-2025, peaking in May 2026. These PRs are mostly additive, rarely merged, and typically closed within 3 days. The authors of these PRs disclosed the use of AI in 14% of cases, with 1% disclosing after a maintainer's request. Most PRs are closed or marked as slop without any reason provided by the maintainer. Based on our open coding, reasons range from concrete elements, like build failures, to ideological or intuitive (i.e., "looks AI to me"). This exploratory study captures a phenomenon in its early stage and highlights GenAI's potential impact on FOSS, emphasizing the pressure it puts on maintainers and the need to shift current open-source development practices.

0.45

Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild

cs.SE ·2026-09-12 ·Obada Kraishan
code/data similarity 0.71
Abstract · 9 pages, 5 figures, 2 tables

Autonomous coding agents now open pull requests in public repositories at a scale that was out of reach two years ago, yet little is known about what happens to that code after it lands. This paper studies 37,623 provenance-labeled pull requests (PRs) from five commercial agents (OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code) and a matched human baseline, drawn from 2,807 GitHub repositories between December 2024 and July 2025. We combine the AIDev dataset with 58,792 cached GitHub API responses to measure security smells in added code, structural maintainability, post-merge churn, revert rates, and human review behavior. Three results stand out. First, quality differences are vendor-specific rather than uniform: Codex-authored PRs were reverted about half as often as human PRs (6.1% vs. 11.5%, odds ratio 0.50), while Devin PRs were reverted more often (14.5%, odds ratio 1.31). Second, agent code pooled across vendors was less likely than human code to contain a security smell (odds ratio 0.63), driven by fewer hardcoded credentials and eval-style constructs. Third, review effort concentrates unevenly: Copilot PRs drew the most human reviews and change requests, and Claude Code PRs waited the longest for a first human review (median 12.6 hours). All pipeline code, statistical reports, and figures are released for replication.

0.20

Engineering Signals of Human-AI Collaboration in the Agentic Coding Era: A Longitudinal Analysis of 33,228 Pull Requests from vLLM and SGLang with Implications for Biomedical AI Agents and Bioinformatics Pipeline Developmen

cs.SE cs.AIcs.ETcs.HC+1 ·2026-08-14 ·Jiada Li, Xuesong Ye, Olamide Olowoniyi
code/data similarity 0.71
Abstract · 24 pages, 9 Figures

The rapid adoption of AI coding assistants and autonomous agentic development systems has coincided with major changes in the pace and structure of open-source software engineering. Yet empirical longitudinal evidence of these changes at the team level remains limited. We present a descriptive longitudinal analysis of seven engineering metrics: pull request (PR) throughput, cycle time, contributor diversity, PR comment density, merge rate, new-author participation, and PR size. Metrics were computed from all merged PRs in two high-velocity AI infrastructure repositories, vLLM (February 2023-June 2026; 18,290 PRs) and SGLang (January 2024-June 2026; 14,938 PRs). We segment development into four eras aligned with major changes in AI-assisted software development and examine human- and bot-authored activities. Both projects show substantial increases in development velocity and AI-developer collaboration signals. PR throughput increased 21x in vLLM and 17.9x in SGLang, while bot-authored PRs accounted for less than 0.2% of this growth, indicating that the increase was overwhelmingly human-driven. In the latest era, median cycle time was 1.04 days for vLLM and 0.62 days for SGLang, while P90 cycle times reached 16.8 and 14.3 days, respectively. Monthly unique authors increased steadily in both projects, suggesting broader contributor participation. PR comment density increased 4.2x in vLLM and 3.8x in SGLang, with bot comments contributing an estimated 15-20% of the increase. In contrast, PR size remained relatively stable across eras. Overall, AI-assisted development is associated with higher throughput, broader contributor participation, and increased AI-developer collaboration signals in high-velocity open-source software development.

0.26

How Do Coding Agents Optimize Software and Report Performance Validation? A Large-Scale Empirical Study of Open-Source Pull Requests

cs.SE cs.PF ·2026-10-02 ·Huiyun Peng, Ricardo Calvo, Kelechi G. Kalu, James C. Davis
publishedbenchmarkcode/data similarity 0.71
Abstract · 46 pages, 11 figures. Extends our MSR 2026 Mining Challenge paper. Huiyun Peng and Ricardo Calvo contributed equally. Replication package: h…

Software performance optimization requires identifying bottlenecks and improvement opportunities, and empirically evaluating the effects and trade-offs across performance objectives. Autonomous AI coding agents submit performance-oriented pull requests (PRs) to open-source projects, yet it remains unclear whether these changes reflect established optimization practices and adequately substantiate their claimed value. Extending our MSR 2026 Mining Challenge pilot study of 407 performance PRs, we analyze a sample of 1,130 agentic and 1,130 human-authored performance PRs collected through June 2026. We compare adoption outcomes and patch characteristics between the two groups, and examine their optimization practices and validation behaviors. The results show that agentic PRs are merged less often than human-authored PRs (54.4% vs. 73.3%), but the two groups use similar optimization strategies and report validation at comparable rates. Agentic PRs rely more on static reasoning and less on benchmarks (46.1% vs. 57.2%), although benchmark use converges by the end of the study period. Agentic PRs also report validation for code-smell refactorings as often as for resource-targeting changes, whereas human-authored PRs do so less often. Across both groups, about half of validated PRs report no quantitative performance metric, and relevant trade-offs are rarely quantified. Agentic optimization practice has become increasingly similar to human practice, but the supporting evidence remains incomplete. Establishing whether agent-proposed optimizations are supported by rigorous, quantitative, and multidimensional evidence remains an important challenge. These findings motivate evaluation infrastructure and review criteria that measure intended effects and relevant costs rather than treating the presence of validation alone as sufficient.

0.40

VeeReview: Measuring Build-Verify Pairs in Agent-Authored Pull Requests from a V-Model Perspective

cs.SE cs.CL ·2026-10-21 ·Gabriel Dautovi, Jie JW Wu
positioncode/data similarity 0.69
Abstract · 47 pages, 9 figures, 11 tables. Under review at Empirical Software Engineering. Replication package: https://zenodo.org/records/22884519

Context: Reviewing a pull request requires verification evidence that addresses the proposed change. Coding agent benchmarks and studies of pull request outcomes do not directly assess this relationship. Objective: We test whether the V-Model's build and verify pairs can measure verification evidence in pull requests and support comparisons between agent-authored and human-authored changes. Method: VeeReview classifies four build and verify pairs using repository signals and a language-model judge. Three pairs use model judgements constrained by deterministic signals, and a rule over the diff and repository tree decides Module Design-Unit Test. We evaluated VeeReview against a 25-pull request reference set labelled by two raters and applied it to 1,454 matched pairs of agent and human pull requests from 741 repositories. Results: VeeReview assigned lower build-only rates to agent-authored pull requests on all four pairs, contrary to the preregistered hypothesis (mixed-effects odds ratios 0.52 to 0.86). Architecture-Integration was not significant. Standardization for linked issues, shipped tests and size reduced matched-rate differences by 34% to 60%, and the rule-decided pair's standardized interval included zero. The instrument did not meet its preregistered calibration criterion: agreement with adjudicated labels was 4% to 60% against a required 80%, and inter-rater agreement was weak (kappa = 0.154). Conclusions: The rule-decided pair shows that a process measure judged by a language model partly reflects how a pull request is presented rather than the verification work behind it. We report the cohort differences as observations made with this instrument, and we argue that such measures need a mechanically decided component and a validated human reference before they can support comparisons of verification practice.

0.30

Testing with Coding Agents: A Large-Scale Comparison of Agent-Supported and Human-Authored Pull Requests

cs.SE ·2026-10-26 ·Roberto Milanese, Francesco Salzano, Hovhannes Muradyan, Jingjing Wu, Remo Pareschi +2 more
similarity 0.69
Abstract

Context. Coding agents increasingly contribute tests through pull requests that developers then maintain, so how testing work is divided affects long-term maintainability. Objective. We compare pull requests in which developers and coding agents collaborate (agent-supported) with pull requests without detectable agent involvement (human-authored) along five dimensions: the frequency and extent of test changes, their type and context, test structure, test smells, and code coverage. Method. We analyze 114,134 pull requests merged between January 2025 and April 2026 in 391 Java repositories, identifying five coding agents through pull-request-level and commit-level signals. Since agent adoption varies across projects, we compare the two groups within repositories with mixed-effects models that control for change size. Results. Pooled across repositories, both groups change test code almost equally often; within repositories, the odds that an agent-supported pull request does so are 1.65 times higher. Agent-supported pull requests add tests more often (odds ratios 1.50-1.58) and modify (0.70-0.76) and delete (0.51-0.77) them less often. We detect no difference in most structural characteristics of the tests they add, which more often have an oracle, nor, in the 95 repositories we could build, in the coverage of their production changes; in test-only pull requests, however, their tests cover less of the code they exercise. Differences in test smells are borderline and not reproduced for individual test methods. Conclusions. Agent participation is associated with a different mix of testing work, which pooled comparisons across projects can hide or reverse; its relationship with test quality remains uncertain.

0.17

"We Permit the Use of AI, but [...]": The Landscape of AI Policies in Popular Open Source Projects

cs.SE ·2026-09-07 ·Andre Hora, Romain Robbes, Stefano Zacchiroli
code/datahighly cited similarity 0.68
Abstract · Currently under review at ACM Transactions on Software Engineering and Methodology (TOSEM)

Open source communities are converging on a new governance artifact: the AI contribution policy. These policies barely existed a few months ago and are now being written and adopted. We analyzed 281 AI contribution policies, and manually classified them along the six dimensions; to study how policies change, we also tracked 92 dedicated AI policy files over time. We answer four research questions on (1) AI usage allowance, (2) AI disclosure practices, (3) AI slop countermeasures, and (4) AI policy evolution. We find that, first, permission is the norm rather than the exception: 83.3% of policies permit or encourage AI in code contributions. But permission comes with conditions, as 67.3% require a high level of human involvement and 43.4% assign accountability. Second, AI disclosure is required by 48.8% of policies, most often in pull request descriptions and commit messages, but what must be disclosed varies widely. Third, we identify ten countermeasures against AI slop, targeting pull requests, users, and autonomous agents. Finally, policies are not static: half of the dedicated AI policy files have already been revised since creation. Our results give maintainers and researchers a baseline and a labeled corpus for studying the impact of AI policies.

0.58

Documentation Drift in AI-Generated Pull Requests

cs.SE ·2026-10-19 ·Will De Michele, Kla Tantithamthavorn
published similarity 0.67
Abstract · Accepted at APSEC 2026 (Early Research Achievements)

AI coding agents now contribute to software repositories at scale, yet their documentation practices remain uncharacterised. We analyse 29,669 AI-generated pull requests from 2,585 repositories in the AIDev dataset, measuring documentation co-change: whether code-file modifications are accompanied by a documentation-file update in the same PR. We find that 63.2% of AI PRs change code without updating any documentation. This varies sharply by task type: 55.2% for features and 77.5% for refactoring. Undocumented code files are followed by a fix to the same file 8.6 to 11.7% more often than documented files across 30 to 90-day windows. A raw PR-level comparison shows the opposite because documented PRs are 3.9 times larger and therefore attract more fixes. Controlling for size exposure reverses this result. Documented changes also merge faster despite their larger size (median: 40 seconds versus 184 seconds), although merging is largely automated in this dataset. Documentation drift also compounds: within 90 days, a file changed without documentation is later re-documented at roughly half the rate of a file changed with documentation (24% versus 48%). This suggests that documentation debt tends to accumulate rather than be revisited.

0.23