CS Paper Radar

Level Up! AI-Driven Challenges and Adaptations to Code Review Practices

Kaia Newman, Christian Bird, Jenna Butler, Emerson Murphy-Hill

cs.SE ·submitted 2026-11-13 20:47 UTC · v1
survey

Abstract

AI can write code faster than humans can review, causing a bottleneck and necessitating changes to code review practices. This bottleneck is hotly debated, giving rise to two extreme and orthogonal imagined futures: one where humans continue to be firmly entrenched in the review process, and one where AI agents merge pull requests entirely without human involvement. Yet, these futures remain rooted in folk wisdom. Through a mixed-methods study of interviews ($n = 24$) and a survey ($n = 2{,}590$) at a large, multinational software company with widespread access to AI, we identify AI-driven code review challenges and strategies reported as effective to meet them. We found that AI enables, necessitates, and complicates a shift in the human reviewer's role toward high-level concerns (e.g., design and intent). We found that AI-assisted authoring results in increased burden for reviewers due to challenges such as a lack of author stewardship and PR volume and size, potentially leading to downstream debt. Our survey showed AI review usage was not correlated with less difficulty keeping up with PR volume, indicating using AI for review alone is not sufficient for solving AI-driven review challenges. Developers found the most useful strategies to be organizations supporting and resourcing review, pushing review burden upstream, and prioritizing reviewer attention on the concerns that matter most.

Closest papers

Nearest neighbours by meaning across the whole index.

Who Pays the Review Cost? Triage, Fairness, and Accountability in AI-authored Pull Requests

cs.SE ·2026-10-08 ·Md Shamimur Rahman, Khairul Alam, Banani Roy, Chanchal K. Roy
survey similarity 0.73
Abstract · 30 pages

AI coding agents are moving from local code assistance into pull-based workflows, where generated contributions must be reviewed, explained, and maintained within existing project norms. Although recent work has begun to characterize AI-authored pull requests (AIPRs), less is known about how reviewers govern their entry into review, how AI authorship reshapes credibility and fairness, and what intake mechanisms protect review sustainability. We report a mixed-method questionnaire survey of 239 practitioners from 31 countries with code-review experience and varying exposure to AIPRs. In the scenarios and self-reports elicited by the survey, AI authorship was not a categorical rejection signal. Instead, respondents described review effort as conditional on whether an AIPR arrived as an accountable contribution, with bounded scope, project-grounded rationale, validation beyond Continuous Integration (CI), contributor responsiveness, and identifiable post-merge ownership. This conditional logic extended to newcomer AIPRs, where respondents emphasized visible participation in the current review process over profile-level reputation alone. Qualitative responses further described shifts in mentoring, scrutiny, deferral, and routing when human stewardship was difficult to observe. We conceptualize missing rationale, validation, and ownership as reviewability debt, the work reviewers must absorb when generated code lacks sufficient human grounding. These findings reframe AIPR governance around making human judgment observable before generated contributions consume scarce reviewer attention.

0.20

AI-to-AI Code Reviews of GitHub Pull Requests

cs.SE ·2026-08-21 ·Niruthiha Selvanayagam, Taher A. Ghaleb
top venuedataset similarity 0.71
Abstract · Accepted at the 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026), Emerging Results, Vision, and Re…

AI coding agents are increasingly integrated into software development workflows, operating on both sides of the pull-request (PR) process: AI authoring agents create or modify PRs, while AI reviewers evaluate them. This creates a closed loop in which one AI coding agent reviews a contribution attributed to another. We construct a large-scale dataset of AI-to-AI code review by linking AI-attributed PRs with AI-attributed review events from CodAGE, a public dataset of coding-agent-generated GitHub events. Our dataset contains 248,641 unique AI-attributed PRs that received at least one AI-attributed review. Of these, 45,269 received cross-product review and 208,145 received same-product review; 4,773 PRs received both. Cross-product AI-to-AI review occurred in approximately 1.6% of identified agent-authored PRs but was substantial in absolute terms, and its volume increased by more than two orders of magnitude from 2025-Q1 to 2025-Q3. Reviewer output varied across author-reviewer configurations. CodeRabbit labeled 35.0% of its comments on Claude Code-authored PRs as refactor comments, compared with 10.5% on Copilot-authored PRs, although this difference may reflect characteristics of the PRs rather than the reviewer. For three of four dual-role reviewers, mean comments per PR were 58-65% higher in the same-product group, although effect sizes were small or negligible and the difference was concentrated in the upper tail. Among pairs with complete, nonnegative timestamps, the observed median latency was 1.2 minutes for cross-product pairs and 4.7 minutes for same-product pairs; differential timestamp availability and reviewer composition limit this comparison. Overall, closed-loop AI-to-AI review is increasing but remains a minority of identified agent activity, with review output varying across authoring-agent groups and product configurations.

0.42

From Delegation to Action Approval: An Exploratory Study of Developers' Practices in Collaborating with AI Coding Agents

cs.SE ·2026-11-06 ·Yinan Wu, Dominik Wermke, Ze Shi Li, Kathryn Thomasset Stolee, Bowen Xu
similarity 0.70
Abstract

Developers now collaborate with AI coding agents that write code and take actions on their behalf. Beyond reviewing this code, developers decide what to delegate and which agent actions to approve. Some of these actions, such as running shell commands or reading files, can have effects that never appear in the resulting code. How developers make these decisions and how much effort this requires are not well understood. To better understand this collaboration, we surveyed 203 developers who use AI coding agents. We found that careful code review does not necessarily extend to action approval: 61% of developers who carefully reviewed most or all generated code still often approved at least one type of agent action without close review. Developers aware of security or privacy incidents or concerning practices did so more often than those who were not. We also found that autonomy varied more between developers than between development activities. Although developers reported a lighter workload, supervising agents remained mentally demanding. When asked what would make them grant more autonomy, developers wanted more than reliability: more than half of their responses called for agents that are bounded, observable, or reversible. Our findings suggest that as agents take on more of the coding, the challenge shifts toward supporting the developers who supervise them.

0.10

Technical Debt Risks in AI-Assisted Software Development: Insights from Practitioners

cs.SE ·2026-11-11 ·Nazia Jahan, Nafisa Alam, Sourov Debnath, Arafat Rahman, S. M. Waesh Ahmed +2 more
similarity 0.68
Abstract

AI coding assistants increasingly support software development, raising concerns about technical debt associated with AI-generated code and development practices. We investigate these concerns through a sequential mixed-methods study involving 137 survey respondents and 20 follow-up interviews with software professionals in Bangladesh. Drawing on ten established technical debt dimensions, we examine perceived risks, contributing circumstances, maintenance consequences, validation practices, and developer understanding. Respondents expressed the strongest concerns about People Debt (73.0\%), Management Debt (70.1\%), and Architecture Debt (68.6\%), suggesting that risks extend beyond code quality. Participants associated ambiguous requirements, insufficient project context, and unrequested AI-generated changes with architectural inconsistencies, rework, and maintenance difficulties. Their experiences revealed contrasting outcomes: AI assistance supported refactoring, maintainability improvements, and testing, but also introduced rework and context misalignment. Although 89.8\% reported regularly verifying AI-generated code, participants expressed concerns about AI dependence, reduced code understanding, and review effort. They also emphasized the need for stronger architectural awareness, requirements clarification, and validation support in coding assistants. Our findings suggest that managing technical debt risks requires attention to project context, implementation decisions, and developer understanding alongside generated code quality.

0.11

How Do Coding Agents Optimize Software and Report Performance Validation? A Large-Scale Empirical Study of Open-Source Pull Requests

cs.SE cs.PF ·2026-10-02 ·Huiyun Peng, Ricardo Calvo, Kelechi G. Kalu, James C. Davis
publishedbenchmarkcode/data similarity 0.67
Abstract · 46 pages, 11 figures. Extends our MSR 2026 Mining Challenge paper. Huiyun Peng and Ricardo Calvo contributed equally. Replication package: h…

Software performance optimization requires identifying bottlenecks and improvement opportunities, and empirically evaluating the effects and trade-offs across performance objectives. Autonomous AI coding agents submit performance-oriented pull requests (PRs) to open-source projects, yet it remains unclear whether these changes reflect established optimization practices and adequately substantiate their claimed value. Extending our MSR 2026 Mining Challenge pilot study of 407 performance PRs, we analyze a sample of 1,130 agentic and 1,130 human-authored performance PRs collected through June 2026. We compare adoption outcomes and patch characteristics between the two groups, and examine their optimization practices and validation behaviors. The results show that agentic PRs are merged less often than human-authored PRs (54.4% vs. 73.3%), but the two groups use similar optimization strategies and report validation at comparable rates. Agentic PRs rely more on static reasoning and less on benchmarks (46.1% vs. 57.2%), although benchmark use converges by the end of the study period. Agentic PRs also report validation for code-smell refactorings as often as for resource-targeting changes, whereas human-authored PRs do so less often. Across both groups, about half of validated PRs report no quantitative performance metric, and relevant trade-offs are rarely quantified. Agentic optimization practice has become increasingly similar to human practice, but the supporting evidence remains incomplete. Establishing whether agent-proposed optimizations are supported by rigorous, quantitative, and multidimensional evidence remains an important challenge. These findings motivate evaluation infrastructure and review criteria that measure intended effects and relevant costs rather than treating the presence of validation alone as sufficient.

0.40

Cognitive Debt in the GenAI Era - Listening to Practitioners' Voices on the Loss of Understanding

cs.SE ·2026-10-27 ·Theocharis Tavantzis, Bianca Trinkenreich, Michaela Greiler, Júlia Condé Araújo, Marina Condé Araújo +4 more
similarity 0.67
Abstract · 12 pages, 3 figures. Manuscript under review at ICSE-SEIP

Fast-paced AI-assisted software development risks reducing developers' understanding of the systems they build. This growing gap between what the system does and what developers understand about it, described as ``cognitive debt'', is not new, but AI has exacerbated it. This has led to a recent surge in blog posts and online discussions, in which practitioners recognize and write about cognitive debt, and share their experiences well before academic studies can capture them. To understand their concerns and inform future research and practice, we conduct a grey literature review of 76 sources, focusing on how cognitive debt is conceptualized and manifests, its contributing factors, its consequences, and the practices to prevent, detect, and mitigate it. We find that cognitive debt can accumulate unnoticed behind working code and passing tests, particularly when developers accept AI-generated solutions without building understanding. Practitioners report consequences such as harder maintenance and debugging, fragmented knowledge in the team, and reduced productivity. They propose practices such as documenting design rationale, review-sized changes, comprehension gates, and knowledge-sharing rituals, along with signals like reliability metrics to detect it. Our findings give software teams a structured view of emerging practical approaches in AI-assisted development and stress the need to preserve human understanding alongside faster code production.

0.55

Comprehension Audits to Mitigate Risks from Automated AI Research

cs.CY cs.AIcs.SE ·2026-10-07 ·Ronald J. Bodkin, Bahrad A. Sokhansanj, Gillian K. Hadfield
code/data similarity 0.67
Abstract · 23 pages, 2 figures, 11 tables

AI is already writing a majority of code for frontier AI labs. This creates a safety risk if there is insufficient human oversight. Existing work proposes minimum comprehension thresholds and unaided checks to mitigate this. To our knowledge, however, there is currently no published frontier-AI assurance regime that requires demonstrated evidence that the responsible humans understand what they are building as a precommitted condition for continuing development or usage. We propose comprehension audits, a novel development-process assurance mechanism in which the responsible people explain R&D contributions to auditors to demonstrate understanding. With independent administration and graded reports, they provide a gate: development of a contribution stops based on a failure to demonstrate human understanding until remediated, with escalating consequences for repeated failures. Our analysis of leading open-source AI projects finds increased output of code with reduced human review commentary rates per line of code, with far lower rates for automated fleet accounts. We advocate for labs to conduct them with embedded independent auditors.

0.20

Vibe Coding: Practice, Performance, Productivity, and Risk -A State-of-the-Art Review

cs.SE ·2026-08-20 ·Dominik L. Michels, Mutaz Abu Ghazaleh, Francois Lazzari, Nabil Kassem, Jonathan Klein
surveyposition similarity 0.66
Abstract

Vibe coding - AI-assisted software development in which the developer describes intent in natural language and validates results by running rather than reading the generated code - was named by Andrej Karpathy in February 2025 and produced its first body of empirical evidence within seventeen months. This state-of-the-art review assembles that evidence across a cross-disciplinary corpus spanning software engineering, human-computer interaction, labour economics, security research, governance, and education. We survey the model landscape, the tool ecosystem, and the performance record by task type, finding the early benchmarks saturated but task-level capability uneven: reliable code generation alongside weak fault detection and hard-to-audit documentation. The productivity record is at first contradictory: peer-reviewed field experiments report +26% more tasks per week, independent randomised trials measure a 19% slowdown, and team-level telemetry shows code-review time up +441%. We argue these readings are consistent once measurement method, scope, and time horizon are held constant, and identify six patterns behind the dispersion, among them effect-shrinkage under broader measurement, self-report diverging from independent measurement, output volume conflated with productivity, and bold claims walked back once tested over longer horizons. We further document security failures in deployed applications, code-quality degradation visible in large-scale code and developer telemetry, unsettled copyright exposure, and evidence of skill atrophy. The review closes with the open research questions and one falsifiable conjecture: that the gains are real on new code and shrink or reverse on mature codebases, which would account for most of the disagreement in the record.

0.20