CS Paper Radar

From Cosmetic to Structural: A Paired Experiment on Metric-Guided Agentic Refactoring

Nadim Hagatulah, Markus Borg, Emma Söderberg

cs.SE ·submitted 2026-10-23 22:28 UTC · v1
code/data

Abstract

The role of coding agents is expanding beyond writing new code toward maintaining existing systems as AI teammates. Yet recent studies show that agent refactorings favor superficial, consistency-oriented edits over structural transformations needed to resolve Technical Debt (TD). We investigate if explicit static-analysis feedback closes this gap. Through the CodeScene MCP server, we give an open-source agent (gptme) access to code smell reports and CodeHealth (CH) scores (1--10, higher is better), and compare guided against unguided refactorings in a paired experiment covering $24{,}620$ single-file competitive programming solutions in C++, Java, and Python. We find that guidance shifts the agent from performing cosmetic edits toward structural operations, raising the rate of effective refactorings by $21$--$26$ percentage points. The benefit depends on initial code quality, with language-dependent break-even points at CH $4.9$--$7.1$, below where the structural transformations pushed by guidance increasingly risk regression, indicating where the agent needs more support to refactor. Guided runs also shift the failure profile toward logical test failures, introduce new coupling smells, and consume $\sim$$3$--$5\times$ more tokens over $\sim$$2\times$ as many interaction turns. Our results indicate that AI teammates can benefit from similar static-analysis feedback that human developers already use, enabling structural transformations toward more effective TD remediation, at a higher token cost and more interaction turns.

Comment
41 pages, 5 figures, 14 tables. Under Review

Closest papers

Nearest neighbours by meaning across the whole index.

REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring

cs.SE cs.AI ·2026-08-21 ·Muhammad Waseem, Aakash Ahmad, Pekka Abrahamsson
code/data similarity 0.76
Abstract · Preprint. 450 Java files from 15 open-source systems; 1,350 model-pass outputs across three LLM configurations. The accompanying replication…

Large Language Models (LLMs) offer new opportunities for automated code refactoring. However, generated changes must reduce targeted quality problems without introducing new issues or altering behaviour-relevant code structures. We introduce REFINE (Refactoring with Evidence-aware Flow for Integrated ageNtic Execution), a tool-agnostic, evidence-aware multi-agent approach for generating Java file-level refactoring candidates. REFINE combines static-analysis-guided smell identification, smell-informed planning, LLM-based transformation, automated re-analysis, preservation checks, and structured reporting. We evaluate REFINE on 450 Java files from 15 open-source systems, producing 1,350 model-pass outputs using OpenAI GPT-5.5, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.8. REFINE reduces detected code smells by 68.26%, 72.79%, and 68.49% across the three configurations, respectively, with the strongest reductions observed for major smells. A matched 150-file direct-prompt baseline shows that REFINE achieves a higher median code-smell reduction with smaller edits and fewer public-method removals. However, broader quality improvements are inconsistent, and preservation checks reveal residual risks, including assert/fail-call changes and public-method removal. Therefore, REFINE outputs should be treated as refactoring candidates requiring compilation, testing, dependency analysis, and human review before adoption in repository- or system-level settings.

0.23

SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

cs.CL cs.SE ·2026-08-10 ·Yuling Shi, Jinghan Xu, Kelin Fu, Wenhao Zeng, Shilin He +10 more
benchmarkcode/datahighly cited similarity 0.73
Abstract · Published as a conference paper at COLM 2026

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.5 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.

0.64

SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

cs.CL cs.AIcs.SE ·2026-08-24 ·Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao +5 more
benchmarkhighly cited similarity 0.69
Abstract

Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs ($5.4\%$) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores $47.0/100$. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, $58\%$ reach $99\%$ of the fixed checks, yet only $26\%$ reach $100\%$. Agent capability differs across migration categories: agents score $31.4$ on build toolchain rewrites but only $5.6$ on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.

0.32

Rasoi: Visual Diffs for Evaluating Structural Changes Made by Coding Agents

cs.SE cs.HC ·2026-11-12 ·Riya Sahni, Aryan Kaul, Anushka Bhatnagar, Xuanming Zhang, Vivian Liu +2 more
similarity 0.68
Abstract

Coding agents let developers modify software rapidly through natural language. However, as agents take on larger changes, developers can lose track of how the codebase is evolving. These edits may look correct on the surface while restructuring the system in unexpected ways: duplicating logic, changing modules outside the requested scope, removing functionality developers meant to keep, or introducing architectural drift. We investigate how visual diffs can help developers understand and evaluate the structural changes that coding agents make. We present Rasoi, a design probe that maps how a codebase's architecture evolves after each agent-made edit, letting developers inspect which architectural elements and relationships were added, modified, or removed across multiple levels of abstraction. We deployed Rasoi in a week-long study with 12 developers working on their own codebases across varied technical domains. We found that visual diffs helped developers see whether changes stayed within scope, notice unexpected restructuring, and understand how modified parts related to the system as a whole. Our findings suggest that the value of oversight tools for coding agents lies less in making developers trust the agent and more in helping developers feel confident in their own evaluation of agent-made changes. We discuss implications for visual tools that support code comprehension and human oversight in AI-assisted software development.

0.24

Your Wish Is My Commit: Over-Compliant Refactoring in AI Coding Agents

cs.SE cs.HC ·2026-10-26 ·Haocong Jiang, Sadra Sabouri, Sepand Haghighi, Souti Chattopadhyay
similarity 0.67
Abstract

Large language models are known to be sycophantic, over-complying with users even when doing so is unproductive. When these models act as coding agents, this can be costly: an agent that keeps editing after no edit improves the code writes churn, technical debt, and review burden into the codebase. Yet the consequences of sycophancy in software engineering remain largely understudied. To understand them, we run a controlled study of iterative refactoring, an open-ended task with no stopping criterion, where any edit an agent makes once the code is already clean reflects compliance rather than improvement. Across algorithmic (n=50) and object-oriented (n=10) suites, we repeatedly ask two widely used agents, Codex and Claude Code, to refactor further across five models. Over 3,000 turns, we define six behavioral signals of over-compliance. We hypothesized that a capable agent would stop once it recognized an optimized codebase; none did. Every model kept editing, rewriting up to 31x the original codebase before first stopping and resuming edits even after declaring the task complete. Over-compliance is a costly, multi-turn behavior that compounds with every turn, and our six signals give developers and tool builders a way to detect and curb it.

0.37

FrontierRefactor: Benchmarking Multi-Scale Performance-Oriented Codebase Refactoring

cs.AI ·2026-10-19 ·Shijian Wang, Fanqing Meng, Jiajun Chen, Qiguang Chen, Linxin Song +6 more
benchmark similarity 0.67
Abstract

Autonomous codebase refactoring requires coding agents to restructure working software while strictly preserving externally observable behavior and prioritizing execution performance and compute efficiency over stylistic preference. Existing benchmarks: refactoring benchmarks assess functional correctness while overlooking runtime regressions, whereas code performance benchmarks target isolated micro-kernels that ignore codebase-scale complexity. We observe that refactoring scale is governed by the scope of the observable contract, the caller-visible interface fixing behavioral invariants while freeing private implementations for optimization. Building on this insight, we introduce FrontierRefactor, a benchmark of 60 qualified engineering tasks spanning four scales, from computational kernels to full repositories, across in-place optimization and cross-language rewriting. Evaluating frontier coding agents through a reference-free cascaded protocol across 1,080 execution runs reveals that correctness and speedup are largely decoupled: 41.2% of submissions satisfy functional equivalence yet fail to deliver verified performance improvements, with 19.3% suffering performance regressions and 21.9% trapped in measurement noise. Refactoring paradigms reveal orthogonal engineering bottlenecks: while cross-language rewrites gain speedups from native compilation after passing functional checks, agents struggle with contract fidelity, with pass rates dropping from 65.0% on modules to 32.6% at package scale; conversely, in-place optimization reliably preserves behavioral contracts at 86.5% but rarely achieves acceleration without physical performance profiling. Trajectory analysis shows that agents fall victim to uncalibrated micro-benchmarking illusions, an operational blind spot that multi-turn execution-timing feedback directly resolves, raising strict success from 16.0% to 40.0%.

0.21

Continuous Autonomous Refactoring: A Research Roadmap for AI-Driven Code Quality Maintenance

cs.SE ·2026-09-01 ·Xin Sun, Daniel Ståhl, Kristian Sandahl, Christoph Kessler
top venue similarity 0.66
Abstract · The paper was accepted in ICSME 2026 in the Visions and Emerging Results Track

Large language models have shown promising capabilities in code refactoring, but existing approaches remain limited to method-level tasks. In this paper, we envision LLM-based refactoring as a continuous component of software maintenance rather than a tool invoked only for occasional manual refactoring. Under this vision, AI agents continuously monitor, evaluate, and improve codebases against explicit and evolving notions of software quality. We present a roadmap organized around five dimensions: the multi-objective optimization problem, quality definition and evaluation, multi-timescale integration of heterogeneous signals, architecture and design pattern, and trust in autonomous refactoring. We further identify integration into continuous delivery pipelines and cost considerations as cross-cutting concerns. For each dimension, we analyze the underlying challenges and pose open research questions. These dimensions define a research agenda for advancing autonomous refactoring from isolated code improvements to system-level quality maintenance.

0.36

From Software History to Executable Regression Tasks: A System for Evaluating Software Agents

cs.SE ·2026-10-18 ·Ruikai Shi, Zhengran Zeng, Rui Xie, Shikun Zhang, Wei Ye
similarity 0.66
Abstract · 22 pages

As software agents increasingly generate and modify code, evaluating their ability to review changes and discover defects becomes important. Such evaluation needs executable project environments for investigation and reliable defect references supported by reproducible behavior. Existing code review benchmarks provide limited support for this combination. We present a system that constructs executable regression instances from historical introducing--fixing commit pairs and developer tests. The system extracts candidate tests from fixing-side changes, then uses constrained agents to transplant them and configure version-specific environments while preserving production code and test expectations. Independent replay and evidence-based review verify whether tests that fail on the introducing version and pass on the fixing version support the proposed regression relationship. Accepted instances retain both versions' replay recipes, validated tests, and execution evidence. Repair-supported analysis and human review add root-cause locations, tested repairs, and causal explanations. These instances support bug detection and localization in code review, and enable bug repair and regression-test generation on the same defects. The system constructs 172 validated instances from 522 candidates across 11 Python projects. We evaluate three software agents with ten models and additional reasoning-effort settings on 83 manually selected, enriched instances across the three tasks. Precise root-line localization remains challenging, while baseline combinations achieve repair and regression-test generation success rates of up to 68.7% and 77.1%, respectively. Leading configurations differ by task, and higher task performance does not necessarily yield greater API-cost efficiency. Our approach provides reusable executable evidence for studying agents' defect discovery, repair, and testing capabilities.

0.17