CS Paper Radar

Evaluating Automated Testing on an Open-Source Web Application Using Cypress

Quoc-Binh Nguyen, Truc-Ly Phan Nguyen, Ngoc Hong Tran, Dung Hai Dinh

cs.SE cs.ET ·submitted 2026-08-20 12:28 UTC · v1
publishedcode/data

Abstract

End-to-end automated testing is increasingly used in web software development to ensure system quality and shorten response times during development. However, the true effectiveness of automated testing depends on many factors including execution time, stability of test results, and maintainability of the test suite as the application continues to evolve. In this paper, we evaluate the effectiveness of end-to-end automated testing using the Cypress framework for an open-source web application. We deployed the experiment with 27 test cases. The test's effectiveness is measured by execution speed, reliability, and maintainability. The experimental results show that the Cypress-based end-to-end test suite has short and stable execution times. It suits frequent runs during software development. The majority of test cases achieved consistent results across multiple runs, while flakiness only occurred in a few tests which involve complex interactive functions. Furthermore, the study highlights the impact of element locator strategies and Page Object Model (POM) architecture on test suite maintainability, demonstrating that resilient data-cy attributes significantly reduce maintenance overhead when UI changes occur.

Comment
Accepted by the 9th International Conference on Research in Engineering and Technology (RET 2026)
Journal ref
9th International Conference on Research in Engineering and Technology (RET 2026)

Closest papers

Nearest neighbours by meaning across the whole index.

WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective

cs.SE cs.AI ·2026-09-14 ·Chenxu Liu, Zilu Zou, Peizhong Gao, Jiawen Tao, Zhexin Zhang +7 more
benchmark similarity 0.65
Abstract

Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evaluation remains challenging. Static benchmarks can credit functionality that exists in source code but is unreachable at runtime. Interactive benchmarks exercise the application, yet incomplete exploration can cause them to miss implemented functionality and confound application defects with agent execution failures. To address these limitations, we propose WebCraftBench, an interactive benchmark for evaluating web application generation from a software testing perspective. WebCraftBench instruments each generated application and uses code coverage to guide an agent in exploring its functionality through user-simulated interactions. It then abstracts the interaction trace into a state-transition graph and evaluates the application along three dimensions: visual aesthetics, usability, and requirement alignment. By separating exploration from scoring, WebCraftBench collects runtime evidence without constraining exploration to predefined acceptance criteria. WebCraftBench comprises 369 real-world user requirements and 5,088 acceptance criteria. Evaluation of 17 frontier LLMs reveals distinct strengths across the three dimensions, with no model leading on every dimension. On 197 validated sessions sampled from an internal arena, WebCraftBench achieves 85.3\% agreement with human preferences, with agreement generally increasing as the score difference between paired applications grows. Further experiments show that coverage guidance improves exploration coverage and the model rankings remain stable when the judge model is replaced.

0.31

WebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution Harness

cs.SE cs.AIcs.MA ·2026-10-02 ·Yun-Yun Tsai, Yuning Mao, Shiqi Wang, Junfeng Yang, Sinong Wang
benchmark similarity 0.63
Abstract · 43 pages

Evaluating WebUI code generation at scale is difficult: outputs may compile and look plausible yet fail under user interaction, and prior benchmarks largely rely on free-form prompts with static checks (build success, screenshots) that miss functional correctness. We introduce WebUIProof, an execution-oriented benchmark that provides structured specifications and dense, executable interaction tests for WebUI generation across two task families: general WebUIs (e.g., dashboards, game, interactive tools) and 3D interactive simulation (e.g., particle/galaxy systems, physics dynamics). WebUIProof includes a UI-agent harness that runs executable interaction tests in a headless browser using an iterative plan--act--observe loop: it locates DOM elements, performs actions, observes resulting UI/DOM changes, and checks the specified assertions. We evaluate across eight commercial LLMs and observe frequent failures on interaction-based requirements even when pages render successfully, especially on 3D simulation interfaces. Finally, we show the UI-agent harness can provide outcome-level training signals. Training compact models (e.g., Qwen2.5 14B and MIMO 7B) with RL rewards derived from executable interaction tests improves functional completion while reducing build failures.

0.19

TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications

cs.SE ·2026-10-09 ·Kazuki Egashira, Niels Mündler-Sasahara, Mark Niklas Müller, Martin Vechev
benchmarkcode/data similarity 0.62
Abstract

Coding agents are increasingly used to create large software projects. While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks remain limited to individual functions or bug fixes. In particular, generating complete test suites for complex software from scratch remains unexplored. To address this gap, we introduce TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. Each mutant is generated from a single requirement of the specification and refined to isolate that behavior: 73\% are killed by exactly one reference test and, on average, by only 0.2\% of the remaining tests. We find that even frontier coding agents struggle to generate strong test suites given our specifications, achieving mutation scores below 60\%, despite their suites being largely valid and 88\% of their tests passing on the reference implementation. We observe that generated test suites often lack relevant input combinations and are smaller than reference test suites. Meanwhile, implementations generated for the same specifications pass 82\% of the reference tests. These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.

0.53

On the Effectiveness of Snapshot Testing: An Empirical Study

cs.SE ·2026-10-21 ·Shun Fujita, Satoru Tomizawa, Yutaro Kashiwa, Bin Lin, Hajimu Iida
survey similarity 0.62
Abstract

Software testing is essential for ensuring quality, yet testing complex front-end user interfaces is often time-consuming and difficult. Snapshot testing has emerged as a low-cost regression testing method that detects unexpected changes by comparing program behavior before and after modifications. Despite its growing adoption, empirical evidence regarding its actual effectiveness remains scarce. To fill this gap, this study presents a large-scale empirical investigation into the effectiveness of snapshot testing involving 340 GitHub repositories and a survey of 60 experienced practitioners. Our quantitative results demonstrate that combining snapshot and traditional non-snapshot tests significantly enhances code coverage, achieving a median statement coverage of 86.7%. We find that the two approaches are highly complementary, typically exercising distinct regions of the production code. Furthermore, test suites incorporating snapshot tests detected 1.3 times more mutants than those relying solely on traditional tests. Our qualitative insights from the developer survey reveal that practitioners value snapshot testing for its ease of implementation, ability to prevent regressions, and transparency in checking results. However, reviewing large result sets and managing false positives remain significant hurdles. These findings guide the strategic integration and future improvement of snapshot testing in modern development pipelines.

0.19

Step-Aligned Script Generation with Verification and Recovery for Web Automation Testing

cs.SE ·2026-10-19 ·Hridya Dhulipala, Xiaokai Rong, Aashish Yadavally, Tien N. Nguyen
similarity 0.62
Abstract · 12 pages, 5 figures, 9 tables

Automated generation of end-to-end web tests remains costly: developers must identify stable selectors, encode multi-step workflows, attach assertions, and maintain scripts as interfaces evolve. Recent LLM-based web agents can complete natural-language tasks in a live browser session, but task completion is not the same as producing a test-ware artifact - code that is executable, replay-able from a reset state, inspectable by developers, and tied to an oracle. We present TESSARA, an adaptive test-script generation framework that converts natural-language test intent into Playwright scripts consisting of executable interaction code, goal-state assertions, and step-level provenance linking each step to its generated code fragment, selector evidence, execution outcome, and repair history. TESSARA grounds each planned step via Step-Aligned Scripting (SAS), enabling fine-grained failure localization, selective regeneration of failing suffixes, and reversibility-aware backtracking that avoids undoing likely commit points (e.g., form submissions). A three-level verification pipeline (syntactic, execution, and semantic/oracle), detects and repairs malformed code, interaction failures, and incorrect landing states. We evaluate TESSARA on three public web benchmarks, measuring script executability, oracle satisfaction, replay stability, and repair efficiency, and show substantial improvements over baselines and underlying VLMs used directly as script generators.

0.49

On the Reliability of LLM-Based Vulnerability Patching Benchmarks

cs.CR cs.SE ·2026-10-07 ·Dang K Le, Wenxuan Shi, Xinyu Xing
benchmarkcode/datahot direction similarity 0.61
Abstract

Large language models (LLMs) have shown strong potential for automated vulnerability patching, but current benchmarks can substantially distort reported performance. Drawing on extensive experience developing, running, and stress-testing such frameworks, we identify under-examined pitfalls across three dimensions: (1) agent-level factors, where prompting, tool availability, and detailed instructions can raise success rates without improving developer-aligned patch quality; (2) framework-level factors, where permission errors, infrastructure bugs, and timeout handling can silently suppress or inflate performance; and (3) dataset-level factors, where bug reports and single proof-of-concept (PoC) tests fail to capture whether patches address root causes or follow developer intent. We curate 112 historical bugs from 84 open-source C/C++, Go, and Rust projects, each with PoC tests, regression tests, and additional developer tests that assess alignment with the original developers' design principles. Through controlled experiments and case studies, we show that LLMs can achieve high PoC passing rates under ideal conditions, yet benchmark execution choices can materially change measured success. More importantly, developer-test passing rates remain low and improve only marginally with newer models, suggesting that models increasingly suppress symptoms without consistently producing upstream-quality fixes. These results show that benchmark scores are highly sensitive to evaluation design, and we provide practical guidelines for more rigorous, reliable, and reproducible evaluation.

0.31

WatchPoint: Executable User Feedback for Real-World Agentic Web Development

cs.SE cs.AIcs.CL ·2026-08-10 ·Guanqun Yang, Wei Yang, Xueqing Liu
benchmark similarity 0.61
Abstract

When a professional web developer's code fails a test, they do not simply re-read the stack trace. They open the application in a browser, click buttons, inspect computed styles, and run diagnostic commands to understand what went wrong. Existing feedback mechanisms for coding agents rely on screenshots, LLM-as-a-judge scoring, or natural-language corrections, but few interact with the live application the way a developer would. We introduce WatchPoint, a simulated-user system that mimics real developer behavior by generating and executing diagnostic scripts against the running application, producing structured observations that guide the coding model's retry. Unlike prior approaches that target single-file edits or evaluate using non-executable metrics, we operate on Web-Bench, a benchmark of 50 multi-file web projects comprising 1,000 sequentially dependent tasks, verified by deterministic end-to-end tests. WatchPoint recovers 57.6% of the tasks it diagnoses, and a controlled user study confirms the simulation's realism: human testers achieve a comparable recovery rate (54.5%), providing evidence that automated diagnostic scripts can substitute for interactive human testing on sequential web development tasks. We further identify a pattern of capability gaps that governs when simulated-user feedback is helpful and when it should be withheld.

0.04

EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

cs.AI cs.SE ·2026-10-12 ·David Nukrai, Bar Alon, Nativ Levy, Ido Andrew Atad, Ido Cohen +4 more
benchmarkcode/data similarity 0.61
Abstract · 24 pages, 6 figures. Code: https://github.com/two-delta/EvoWeb ; Dataset: https://huggingface.co/datasets/TD-Labs/EvoWeb

Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily evaluate development toward fixed specifications. We introduce EvoWeb, a novel benchmark comprising 320 cumulative requests over 32 web applications, grounded in real-world product features and structured by functional dependency graphs. Agents iteratively modify a persistent codebase, while a multimodal web agent tests each intermediate application in a live browser, assessing newly requested functionality, cross-request integration, and previously requested functionality. To characterize reliability throughout development, we integrate trajectory performance across user-tolerance thresholds rather than impose a fixed stopping criterion. Experiments across open- and closed-source frontier models show that EvoWeb reveals clear performance differences between models, with cross-request integration emerging as a key challenge. Further analysis shows that later requests with deeper or more numerous dependencies are solved less accurately, while runtime errors become more frequent over successive requests. EvoWeb provides a testbed for evaluating and improving coding agents in long-horizon, multi-request development.

0.60