CS Paper Radar

AppLooper: An Agentic Application Engineering Loop for Accountable Release with Virtual-User Feedback

Zihong He, Chen Liang, Hai-Ning Liang

cs.HC ·submitted 2026-08-14 08:55 UTC · v1
code/data

Abstract

Much existing research on coding agents organizes application development as an iterative loop of requirement interpretation, implementation, tool execution, evaluation, and repair. As these loops run longer, requirements may drift; users may lose awareness of the current state and rationale for changes; and generated applications may remain insufficiently grounded in target users' contexts and needs. Application engineering therefore requires a mechanism connecting owner intent, target-user experience, development changes, and responsibility for release. We present AppLooper, a human--coding-agent--virtual-user application engineering loop for accountable release. An application owner confirms frozen requirements, supplies feedback, inspects candidates, and retains final release authority. A development agent produces and revises versioned candidates. A virtual-user agent cohort executes interface scenarios grounded in target users and contexts of use. Besides, an owner-intent simulation agent retests only requirements, constraints, and feedback explicitly confirmed by the owner, abstaining when evidence is insufficient. A testing agent performs read-only developmental checks by reproducing reported failures, running existing regression tests, and exercising the current candidate through its browser interface. The orchestration layer groups the resulting findings and routes them into development revision, targeted retesting, and owner inspection. AppLooper binds requirements, feedback sources, interface targets, development changes, retesting outcomes, owner interactions, and release decisions to specific versions. It thereby extends sustained coding-agent iteration into a traceable and reviewable lifecycle in which humans retain final responsibility for release. Source code is available at https://github.com/ZihongHe/applooper.

Closest papers

Nearest neighbours by meaning across the whole index.

Who Verifies the Builder? Toward Independent Assurance for Agent-Produced Software

cs.SE cs.AI ·2026-10-16 ·Sion Israel Sion, Alain April
code/data similarity 0.71
Abstract · 10 pages, 2 figures, 4 tables. Artifact: https://github.com/besionisrael/assure-harness

Coding agents can now implement a specification, write and run tests, repair failures, and declare the result ready for release. When the same agentic system produces both the software and the evidence of its correctness, a passing test suite is no longer independent evidence: visible tests become an optimization target, code and tests may share one misinterpretation of the specification, and the builder's own tests may simply be incomplete. This paper argues that agent-produced software requires an assurance plane that is structurally independent of the build plane. We characterize four gaps between stakeholder intent and production (requirement, conformance, evaluation, and model gaps) and identify which of them verification can close. We propose a framework that separates building from verifying, introduces a Verification Contract stating what must be demonstrated before release, and organizes verification in a specification-first phase without access to the implementation and an implementation-aware phase based on mutation analysis, property-based testing, and fuzzing, with an assurance report that leaves the release decision to a human. We evaluate the framework in a pre-registered study on three stateful services, two of them written by third-party models, and an EvalPlus pilot. The builder's programs were correct, but its own tests missed subtle interactions: a verifier in a fresh context detected more of the 46 seeded faults than the builder's tests (Holm-corrected p = 0.006) and surfaced the ambiguities of the specification, at the price of rare false oracles. A verifier from another provider did not add detection.

0.20

Reality Is the Final Verifier: On Two Key Gaps in Agentic Software Engineering

cs.SE cs.AI ·2026-09-10 ·Alexander Krentsel, Shubham Agarwal, Mert Cemri, Shu Liu, Sidharth Sankhe +3 more
similarity 0.71
Abstract

Software development follows an implementation-verification loop in which developers or agents iteratively revise an implementation until an evaluator, such as a test suite, accepts it. The evaluator checks the implementation against a set of requirements under a model of the deployment environment. Yet even a formal proof that the implementation satisfies the requirements under the model cannot guarantee acceptable behavior after deployment. Requirements only approximate stakeholder intent, and the model only approximates the real deployment environment. We call these together - requirement gap and model gap - the two-gap framework, which unifies the main failure modes of agentic software engineer-ing: reward hacking exploits omissions in the requirements or model, while hallucination widens the gaps by fabricating requirements or environment assumptions. Because neither gap can generally be certified closed in an open, changing world, the goal shifts from closing them to continuously narrowing them. We therefore propose an assurance-revision loop that uses deployment evidence to revise the requirements, model, or evaluator when stakeholders reject the resulting behavior. We then cast assured agentic development as a resource-allocation problem over human judgment, agent capability, and compute. The two principal bottlenecks mirror the two gaps: human judgment for the requirement gap and faithful, costly evaluation for the model gap. Reality remains the final verifier: acceptable behavior under actual deployment conditions is the ultimate test, while predeployment evaluations remain proxies for it.

0.39

PersonaCodeBench: Evaluating Coding Agents in Persona-driven Human-Interactive Sessions

cs.AI ·2026-11-03 ·Haixin Wang, Jianheng Hou, Hongyu Shen, Wenzhe Fan, Wei-Chieh Huang +25 more
benchmark similarity 0.68
Abstract · 40 pages, 9 figures

Coding agents must deliver correct software while respecting users' expertise, preferences, and working conditions. Existing benchmarks increasingly incorporate interaction, yet personalized requirement fulfillment remains underexplored. We introduce PersonaCodeBench, which turns user characteristics into verifiable obligations in persona-driven multi-turn coding sessions. Our persona pool spans roles, technical abilities, interaction preferences, and practical constraints, yielding task-grounded evolved requirements for implementation, permissions, testing, explanation, and delivery. At runtime, a controller governs user simulator responses and requirement disclosure, using online requirement evaluation to provide feedback and advance the interaction. Requirements emerge progressively, with their content, disclosure conditions, and acceptance criteria fixed before evaluation. A completion verifier and a personalization verifier separately assess native correctness and persona fulfillment. Joint pass requires both adherence to the declared budget and compliance with validity conditions. We evaluate all major frontier agents, establish a comprehensive leaderboard, and conduct detailed experimental analyses to validate effectiveness. PersonaCodeBench supports reproducible evaluation of coding agents' ability to meet explicit, personalized user expectations alongside coding objectives.

0.10

UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users

cs.AI ·2026-10-21 ·Zhan-Yu Wu, Ji Luo, Alyssa Wu, Junjielong Xu, Richeng Xuan +3 more
similarity 0.68
Abstract

As coding agents become more capable, developers increasingly use them to fix bugs through multi-turn interactions. Existing benchmarks do not adequately cover this scenario. Most present a complete issue report in a single turn without interaction. Their evaluation can produce both false positives and false negatives, accepting incorrect patches or rejecting correct ones. We introduce UserSWE for evaluating coding agents' ability to fix repository-level bugs through multi-turn interactions with dynamic users and execution-grounded verdicts at every turn. Its construction pipeline turns repository history and existing single-turn benchmarks into multi-turn tasks. At each turn, information pools constrain what the simulated user can reveal, while a persona chain sampled from a transition model calibrated on 20,859 development sessions governs how it speaks. Tests run after every turn, and a pair of agents from different model families writes and gates each verdict, using test results as primary evidence and model judgment only for aspects the tests cannot resolve. We evaluate eleven agents on 72 tasks from 42 repositories and find that agent rankings change with the interaction budget. Claude Fable 5.1 resolves the most tasks in one turn, GPT-6-Astra resolves the most by the eighth turn, and Claude Opus 5 leads at most intermediate budgets. These results indicate that single-turn performance is insufficient to characterize interactive coding ability.

0.18

Testing with Coding Agents: A Large-Scale Comparison of Agent-Supported and Human-Authored Pull Requests

cs.SE ·2026-10-26 ·Roberto Milanese, Francesco Salzano, Hovhannes Muradyan, Jingjing Wu, Remo Pareschi +2 more
similarity 0.67
Abstract

Context. Coding agents increasingly contribute tests through pull requests that developers then maintain, so how testing work is divided affects long-term maintainability. Objective. We compare pull requests in which developers and coding agents collaborate (agent-supported) with pull requests without detectable agent involvement (human-authored) along five dimensions: the frequency and extent of test changes, their type and context, test structure, test smells, and code coverage. Method. We analyze 114,134 pull requests merged between January 2025 and April 2026 in 391 Java repositories, identifying five coding agents through pull-request-level and commit-level signals. Since agent adoption varies across projects, we compare the two groups within repositories with mixed-effects models that control for change size. Results. Pooled across repositories, both groups change test code almost equally often; within repositories, the odds that an agent-supported pull request does so are 1.65 times higher. Agent-supported pull requests add tests more often (odds ratios 1.50-1.58) and modify (0.70-0.76) and delete (0.51-0.77) them less often. We detect no difference in most structural characteristics of the tests they add, which more often have an oracle, nor, in the 95 repositories we could build, in the coverage of their production changes; in test-only pull requests, however, their tests cover less of the code they exercise. Differences in test smells are borderline and not reproduced for individual test methods. Conclusions. Agent participation is associated with a different mix of testing work, which pooled comparisons across projects can hide or reverse; its relationship with test quality remains uncertain.

0.17

Agent Trajectories as Evidence for Software Quality Assurance

cs.SE cs.AI ·2026-11-17 ·Jonathan Ullrich, Julien Siebert, Michael Kläs, Adam Trendowicz, Andreas Jedlitschka
similarity 0.67
Abstract · 3 pages, submitted to ICSE2027 AMPLIFY Workshop https://conf.researchr.org/home/icse-2027/amplify-2027#co-create-software-engineering-resear…

Autonomous software agents increasingly plan, implement, and test software with limited human oversight. Their decisions may introduce implicit assumptions that final code and passing tests do not adequately expose. We propose analyzing observable agent trajectories alongside software and agent specifications and AI generated artifacts as evidence for targeted quality assurance. The aim is to prioritize consequential decisions for scrutiny, investigate their possible contribution to software problems, and inform improvements to specifications and development practices. Industrial experience motivates the idea; available execution records and preliminary analysis methods make it plausible. However, trajectories alone do not establish that decisions are justified, outcomes acceptable, or corrective changes effective. We ask how this evidence can support review, diagnosis, and continuous quality improvement, and when simpler QA approaches suffice. Perspectives from requirements engineering, verification and validation, agent systems, causal inference, empirical software engineering, and human factors could help develop and evaluate the idea.

0.16

Toward Evidence-Based Agentic Software Engineering

cs.SE ·2026-10-26 ·Sebastian Baltes, Christoph Treude
similarity 0.67
Abstract · 5 pages, 1 figure, 1 table, under review

Task success rates and execution costs of agentic coding tools depend on both model and harness. The harness manages context, provides tools, and runs the agent loop. In empirical software engineering (SE) studies, comparisons that change model and harness together cannot isolate their effects. We qualitatively annotated 325 experiments reported in 146 peer-reviewed full papers published between 2025 and 2026. We examined the reported configurations and whether comparisons isolated harness effects, including harness configuration and version, model, tasks, and user interaction. The exact version of every harness used was reported for only 15 experiments. For five, the paper reported when the experiments were run, and for 31, authors shared execution traces for every session. Of the 226 experiments comparing harnesses, 35 never varied the harness in isolation. Based on this analysis, we derive requirements for a reference harness that the SE community can use for experiments. We call on the community to report all degrees of freedom of experiments with agentic coding tools and on SE venues to pilot dedicated tracks or special issues around a specified reference harness release. In these tracks, papers would hold the harness version fixed and vary features through separately versioned extensions to isolate feature effects and test their interactions.

0.34

WatchPoint: Executable User Feedback for Real-World Agentic Web Development

cs.SE cs.AIcs.CL ·2026-08-10 ·Guanqun Yang, Wei Yang, Xueqing Liu
benchmark similarity 0.67
Abstract

When a professional web developer's code fails a test, they do not simply re-read the stack trace. They open the application in a browser, click buttons, inspect computed styles, and run diagnostic commands to understand what went wrong. Existing feedback mechanisms for coding agents rely on screenshots, LLM-as-a-judge scoring, or natural-language corrections, but few interact with the live application the way a developer would. We introduce WatchPoint, a simulated-user system that mimics real developer behavior by generating and executing diagnostic scripts against the running application, producing structured observations that guide the coding model's retry. Unlike prior approaches that target single-file edits or evaluate using non-executable metrics, we operate on Web-Bench, a benchmark of 50 multi-file web projects comprising 1,000 sequentially dependent tasks, verified by deterministic end-to-end tests. WatchPoint recovers 57.6% of the tasks it diagnoses, and a controlled user study confirms the simulation's realism: human testers achieve a comparable recovery rate (54.5%), providing evidence that automated diagnostic scripts can substitute for interactive human testing on sequential web development tasks. We further identify a pattern of capability gaps that governs when simulated-user feedback is helpful and when it should be withheld.

0.04