CS Paper Radar

Your Wish Is My Commit: Over-Compliant Refactoring in AI Coding Agents

Haocong Jiang, Sadra Sabouri, Sepand Haghighi, Souti Chattopadhyay

cs.SE cs.HC ·submitted 2026-10-26 22:12 UTC · v1

Abstract

Large language models are known to be sycophantic, over-complying with users even when doing so is unproductive. When these models act as coding agents, this can be costly: an agent that keeps editing after no edit improves the code writes churn, technical debt, and review burden into the codebase. Yet the consequences of sycophancy in software engineering remain largely understudied. To understand them, we run a controlled study of iterative refactoring, an open-ended task with no stopping criterion, where any edit an agent makes once the code is already clean reflects compliance rather than improvement. Across algorithmic (n=50) and object-oriented (n=10) suites, we repeatedly ask two widely used agents, Codex and Claude Code, to refactor further across five models. Over 3,000 turns, we define six behavioral signals of over-compliance. We hypothesized that a capable agent would stop once it recognized an optimized codebase; none did. Every model kept editing, rewriting up to 31x the original codebase before first stopping and resuming edits even after declaring the task complete. Over-compliance is a costly, multi-turn behavior that compounds with every turn, and our six signals give developers and tool builders a way to detect and curb it.

Closest papers

Nearest neighbours by meaning across the whole index.

From Cosmetic to Structural: A Paired Experiment on Metric-Guided Agentic Refactoring

cs.SE ·2026-10-23 ·Nadim Hagatulah, Markus Borg, Emma Söderberg
code/data similarity 0.67
Abstract · 41 pages, 5 figures, 14 tables. Under Review

The role of coding agents is expanding beyond writing new code toward maintaining existing systems as AI teammates. Yet recent studies show that agent refactorings favor superficial, consistency-oriented edits over structural transformations needed to resolve Technical Debt (TD). We investigate if explicit static-analysis feedback closes this gap. Through the CodeScene MCP server, we give an open-source agent (gptme) access to code smell reports and CodeHealth (CH) scores (1--10, higher is better), and compare guided against unguided refactorings in a paired experiment covering $24{,}620$ single-file competitive programming solutions in C++, Java, and Python. We find that guidance shifts the agent from performing cosmetic edits toward structural operations, raising the rate of effective refactorings by $21$--$26$ percentage points. The benefit depends on initial code quality, with language-dependent break-even points at CH $4.9$--$7.1$, below where the structural transformations pushed by guidance increasingly risk regression, indicating where the agent needs more support to refactor. Guided runs also shift the failure profile toward logical test failures, introduce new coupling smells, and consume $\sim$$3$--$5\times$ more tokens over $\sim$$2\times$ as many interaction turns. Our results indicate that AI teammates can benefit from similar static-analysis feedback that human developers already use, enabling structural transformations toward more effective TD remediation, at a higher token cost and more interaction turns.

0.33

When Models Edit Too Much: On the Fidelity of Minimal Code Edits

cs.SE cs.AIcs.CL ·2026-09-03 ·Tongyao Zhu, Wei Hern Lim, Min-Yen Kan
top venuehighly cited similarity 0.65
Abstract · EMNLP 2026 (Main)

Large language models (LLMs) are increasingly used to edit existing code, but correctness alone is not enough: useful repairs should also be minimal, reviewable, and faithful to the original implementation. We study over-editing, the tendency of a model to rewrite code beyond what is required to fix a bug. We construct an evaluation framework from 400 BigCodeBench problems by injecting controlled AST-level corruptions into reference solutions, giving each repair task a known minimal patch. Across frontier LLMs, over-editing is widespread even among strong models like GPT-5.5: high Pass@1 can coexist with unnecessarily large edits and added cognitive complexity. A preservation instruction substantially reduces this behavior, lowering average excess Levenshtein distance from 0.195 to 0.131, reducing added cognitive complexity by 26.6%, and increasing Pass@1 by 2.3 points. However, these gains do not simply follow from a larger reasoning budget or larger models. We next ask whether minimal editing can be learned directly during post-training. We observe that supervised fine-tuning overfits to seen corruption patterns, whereas reinforcement learning gives the best out-of-domain edit-fidelity and performance-retention trade-off. These results position edit fidelity as a distinct axis of code-repair quality and show that it can be measured and learned.

0.71

Continuous Autonomous Refactoring: A Research Roadmap for AI-Driven Code Quality Maintenance

cs.SE ·2026-09-01 ·Xin Sun, Daniel Ståhl, Kristian Sandahl, Christoph Kessler
top venue similarity 0.63
Abstract · The paper was accepted in ICSME 2026 in the Visions and Emerging Results Track

Large language models have shown promising capabilities in code refactoring, but existing approaches remain limited to method-level tasks. In this paper, we envision LLM-based refactoring as a continuous component of software maintenance rather than a tool invoked only for occasional manual refactoring. Under this vision, AI agents continuously monitor, evaluate, and improve codebases against explicit and evolving notions of software quality. We present a roadmap organized around five dimensions: the multi-objective optimization problem, quality definition and evaluation, multi-timescale integration of heterogeneous signals, architecture and design pattern, and trust in autonomous refactoring. We further identify integration into continuous delivery pipelines and cost considerations as cross-cutting concerns. For each dimension, we analyze the underlying challenges and pose open research questions. These dimensions define a research agenda for advancing autonomous refactoring from isolated code improvements to system-level quality maintenance.

0.36

Metrics-in-the-Loop: Complexity Feedback Reshapes Agent-Written Code

cs.AI ·2026-11-18 ·Nicolas Chapados, Jean-François Gagné, Nouha Dziri
similarity 0.62
Abstract · Agent Evaluation Science Symposium, Fall 2026, New York City

Coding agents now write whole codebases in long autonomous runs, guided mainly by whether their tests pass. Yet passing tests alone does not reveal accumulating complexity or gaps in test coverage, which then persist as agents extend their own code. We introduce metrics-in-the-loop, an inference-time intervention that gives agents in-flight feedback: cheap, automatically computed code metrics injected into the agent's context during the trajectory. After each code-changing agent turn, we insert short feedback messages naming changed functions whose cyclomatic complexity exceeds a limit (complexity feedback) or changed lines that are skipped by tests (coverage feedback), giving the agent the opportunity to refactor code or strengthen tests before continuing. We run extensive experiments and ablations comparing several interventions across three long-horizon coding benchmarks and four strong models. We find that complexity feedback significantly reduces erosion, a measure of complexity concentration of code, without affecting held-out correctness of the code. Broadly, our experiments investigate a practical form of agent "re-railing", where timely, actionable measurements redirect agents before weaknesses become the foundation they keep building on. As agents scale to codebases with millions of lines, such interventions become critical to sustaining effective development and long-term code maintainability.

0.23

Agent Skill Evolution: How Revisions Affect Coding Agents

cs.SE ·2026-10-04 ·Jiajie Wang, Yutong Zhao, Tianlin Li, Huashan Chen, Jinfu Chen +2 more
similarity 0.62
Abstract

Agent Skills, the SKILL.md files that tell an LLM coding agent how a project works, are revised like code, yet what a revision does to the agent is unknown. From 2,608 first/last revision pairs of 3,159 Skills, we characterize how Skills evolve and how they change together with the configuration of the agent's harness. We then focus on rule changes, revisions that add or remove a rule we can check automatically, such as "run allium check". We measure their effect on 21 models in single answers and on four agents in a sandbox, and their cost on 20 of these models and the four agents. Most revisions (55%) change a rule or procedure, and commits that revise a Skill change harness files such as CLAUDE.md more often than other commits of the same size. Across 16 open-weight models, an added rule raises compliance in a single answer by +0.41 on average. Across the four agents, the rate at which the agent takes the required action rises by +0.23 on average (+0.16 to +0.36), and for the three agents that blind judges assessed, final correctness rises by +0.10 on average (+0.06 to +0.14). The gain comes mainly from rules that name a command or path the old Skill did not mention. Real tools load a Skill's body only when the agent decides it needs it. In that setting the four agents keep about half of the action gain on average (51%), and the three open models about 38%. A revision adds 18-19% input tokens to a single answer and no detectable cost to an agent episode, while loading a Skill's body raises the tokens of an episode by 50% on average.

0.49

Do Agents Still Follow the Rules? Benchmarking Rule Compliance under Agent Context Compaction

cs.CR cs.AIcs.CL ·2026-11-02 ·Yiwei Zha, Qingyu Meng, Herbert Bos, Min Chen
benchmarkcode/data similarity 0.62
Abstract

LLM agents compact long histories to fit their context windows, but the compacted context must still carry the operating rules that govern later actions. Prior work shows that violations rise after compaction, but measures compliance outside a complete agent harness and does not identify where each violation comes from. We introduce ARCBench(Agent Rule Compliance Benchmark), a configurable benchmark built on the open-source pi agent harness. Its tool-calling agent loop lets us measure rule compliance from actions taken during execution, bringing the evaluation closer to deployed agent workflows. Using ARCBench, we examine where violations after compaction come from, and how different compaction configurations and adversarial settings change their sources. We find that compaction rarely deletes the rule: most violations after compaction occur while the rule is still in the compacted history, with the agent treating a later request as permission, ignoring the rule, and proceeding anyway. Where violations come from depends on the configuration: how the rule is delivered decides whether the agent would break it even without compaction, while the compaction strategy and budget decide whether the rule survives. The strongest compactor-targeted attack more than doubles violations by corrupting the rule in the summary and changing how the agents behave, whereas the strongest defense restores the rule in most summary but still violates. Finally, retaining a rule does not guarantee compliance: compacted runs that retain the rule violate more than twice as often as runs with the full history.

0.24

Teaching Agents to Code Reliably

cs.AI cs.LG ·2026-10-02 ·Muhammad Ahmed Mohsin, Myeongsoo Kim, Kangrui Ruan, Shweta Garg, Varun Kumar +1 more
position similarity 0.62
Abstract · Submitted to ICLR 2027

Autonomous coding agents solve repository issues by reading code, running commands, editing files, and submitting patches. Extra inference-time compute yields gains only when it produces a useful repair and supplies reliable evidence for choosing one. Three behaviors decide both, and we argue they are teachable rather than byproducts of scale, so a policy can carry them instead of a scaffold. Location diversity remains narrow, since attempts return to the same site and extra samples add no coverage. Edit diversity is left unexploited, since methodologies that differ resolve complementary issues no single run reaches. Verification misleads, since a test the agent writes for its own patch accepts many incorrect ones. Directing search by execution feedback and scoring each patch against its own reverted tree resolves 52.8% of SWE-bench Verified using 48.1% of the agent-steps an eight-sample baseline spends. Training moves these behaviors into the policy. On the 270 issues held out from SFT and RL training, weighted supervised fine-tuning raises pass@1 from 31.9% to 35.2% and pass@8 from 46.7% to 51.1%. A reinforcement objective then trains the verifier against gold-labeled repairs and incorrect variants, crediting the assertions that detect them. It raises pass@1 to 43.0% and pass@8 to 60.7%, lifts verifier precision from 26.8% to 41.7%, and more than halves false acceptance. Resolution improves on two of three out-of-distribution suites and verifier precision on all three, and the gains hold at 7B, 14B, and 30B against published coder baselines.

0.19

Rebuild Dossier: Mechanically-Enforced Specs for Agentic App Rebuilds, and What Model-Tier Failures Reveal

cs.SE cs.AI ·2026-08-22 ·Parker Fawcett
code/data similarity 0.62
Abstract · 48 pages, 1 figure. Code and evaluation artifacts: https://github.com/Parker-Fawcett/rebuild-dossier (archived at DOI: 10.5281/zenodo.220368…

An AI agent's rebuild is only as good as the process that produced it. Prior work found that once a model is strong enough, a multi-agent rebuild pipeline loses to the simplest approach: giving the model the original code and one instruction (AgentModernize). We present rebuild-dossier, an open-source tool that locks an application's real interface - its exact inputs and outputs - before any code is written, then enforces one-test-at-a-time building through automated checks, not written instructions alone. Three results shape this evaluation, with differing amounts of evidence. First, in a small comparison, the compliant agent failed a held-back test while the rule-breaking agent passed everything - proof that a passing suite doesn't certify correctness when tests can be gamed. Second, we tested whether this beats simply giving the weaker model the source and one instruction: tied on a small app, but lost outright on a larger one where the automated check wasn't even running - pointing to the check mechanism, not interface-locking, which held up separately. Third, every claim here is checked at three levels - the agent's own report, an automated log, and the actual files produced - catching real errors, including a bug in our own logging code, that a single level would have missed. These risks reproduce on a different model and toolchain: a stronger model followed our process three times running, something the weaker model never managed. The tool is public, MIT licensed, and reproduces end to end against our own applications.

0.32