CS Paper Radar

Evaluating Change Point Detection Methods for Software Performance Regression Analysis

Diego Elias Costa, Michele Tucci, Luca Traini, Daniele Di Pompeo, Thomas Bach, Francois Farquet, David Daly, Simon Eismann, Petr Tůma, Vittorio Cortellessa, André van Hoorn

cs.SE ·submitted 2026-10-06 19:22 UTC · v1

Abstract

Performance issues in software systems are a critical quality issue that can erode user trust, violate service-level agreements, and ultimately affect business efficiency. Consequently, software performance engineering has shifted its focus to developing robust techniques to detect performance regressions as early as possible in the development cycle. Performance regression analysis often relies on time series of performance measurements to detect significant changes in performance behavior. Change Point Detection (CPD) methods have been widely used to automate the identification of such changes in various domains, including finance, healthcare, and performance monitoring. However, the effectiveness of these methods for software performance measurements has not been thoroughly evaluated. In this paper, we present a comprehensive study to evaluate the effectiveness of various CPD methods on real-world software performance measurement datasets. We start by collecting performance measurement data from three large software systems and characterizing the unique properties of performance time series data. Then, we undertake a large-scale effort to annotate and evaluate the consistency of human annotators' identification of potential performance changes. Thereafter, we evaluate the accuracy of twelve distinct CPD methods in detecting potential performance changes, providing insights into their applicability and effectiveness in software performance regression analysis.

Closest papers

Nearest neighbours by meaning across the whole index.

Towards Automated Recommendation of Steady-State Detection Algorithms for Software Performance Benchmarks

cs.PF cs.SE ·2026-11-12 ·Martin Beseda, Vittorio Cortellessa, Federico Di Menna, Daniele Di Pompeo, Luca Traini
benchmark similarity 0.64
Abstract

Steady-state detection is a central step in software performance benchmarking, yet different detection methods rely on different assumptions and may perform best on different execution profiles. This paper formulates steady-state detection as an algorithm-recommendation problem: given a benchmark time series, predict which of two complementary detectors is more appropriate. The first, CP-SSD, identifies steady state through changepoint-based segmentation of the execution trace, whereas the second, KB-KSSD, uses a kernel-based approach to identify stable behaviour. We propose a data-driven recommender that characterises benchmark time series using automatically extracted features and combines supervised prediction with information derived from groups of time series exhibiting similar behaviour. The approach is evaluated on 407 labelled benchmark time series for which one of the two detectors can be identified as preferable with respect to a reference steady-state index. The proposed recommender selects the better detector correctly for 72.97% of the series, compared with 62.65% when always selecting the detector that performs best most often, corresponding to an absolute improvement of 10.32 percentage points. Paired statistical testing and bootstrap confidence intervals support this improvement. A comparison with a recommender using time-series features alone shows that the cluster-derived information mainly improves interpretability rather than predictive accuracy. Recommendations also remain highly stable when the unsupervised representation is learned from either the labelled subset or the full collection of 5,860 time series. These results demonstrate the feasibility of per-series steady-state detector recommendation while highlighting the need for broader labelled data, particularly for execution profiles favouring the changepoint-based detector.

0.23

Systematic Detection of Energy Regressions in Software Version Histories

cs.SE ·2026-10-14 ·Jérôme Maquoi, François Bechet, Luís Cruz, Benoît Vanderose, Xavier Devroey
similarity 0.62
Abstract · 17 pages, 4 figures, submitted to Information and Software Technology journal

Context: Green software engineering is emerging as a crucial response to the rising energy impact of information technology, especially in continuous development. However, challenges remain in devising automated methods to efficiently identify energy regressions across commits. In particular, little effort has been put into automatically detecting regressions at the commit level by identifying statistically significant changes in energy consumption. Objectives: This paper presents EnergyTrackr, an approach for detecting energy regressions across multiple commits. Methods: We empirically evaluated EnergyTrackr on 1,932 commits from five Java projects, assessing its ability to identify significant energy changes. We also compared a straightforward linear algorithm that analyzes the entire commit history with a more efficient dichotomous search algorithm. Results: The evaluation demonstrates the effectiveness of EnergyTrackr in identifying significant energy changes across commits. Our findings highlight the efficiency of the dichotomous algorithm in uncovering overall software energy trends, achieving results comparable to the linear approach while reducing resource consumption and processing time. Conclusion: We expect EnergyTrackr to help developers accurately monitor energy regressions and improvements in their projects and optimize their source code to reduce software energy consumption.

0.19

AdapBench: Adaptive Early-Stopping and Prioritization for Efficient Microbenchmarking in Cloud-based CI/CD

cs.SE cs.DCcs.PF ·2026-10-31 ·Sebastian Koch, Jonas Heisterberg, Nils Japke, Trever Schirmer, David Bermbach
publishedcode/data similarity 0.61
Abstract · 11 pages, 3 figures, 3 tables. Accepted at the 19th IEEE/ACM International Conference on Utility and Cloud Computing (UCC 2026). Code and da…

Cloud-based software systems frequently release new versions, a fact further accelerated through agentic coding. Each new release, however, brings the risk of performance regressions which can irritate end users, increase infrastructure costs, and increase carbon footprints. Therefore, developers strive to catch such regressions in testing stages, usually through so-called continuous benchmarking. The standard way of doing this is using microbenchmark suites which, however, take notoriously long to execute. Optimizing the execution of microbenchmark suites is, hence, an active field of research. In this paper, we propose AdapBench, an online scheduling system for reducing the execution duration of microbenchmark suites. AdapBench is the first approach which dynamically makes at-runtime decisions for prioritizing the most informative benchmarks and early-stopping each one once enough evidence has accumulated to accept or reject a performance change hypothesis. We evaluate our prototype on eight popular Java projects, commonly used by microbenchmark research, and four additional Go projects. At matched 90% change detections and false-positive rate compared to an optimistic uniform run, AdapBench saves 42% of the aggregate execution duration on the Java projects, with negligible scheduling overhead (below 0.001% of the total runtime). On the Go projects, AdapBench finds every performance change with zero false positives while saving 85% of the full run.

0.49

Characterizing Post-Incident Software Evolution

cs.SE ·2026-10-19 ·Matt Pope, Jonathan Sillito
similarity 0.58
Abstract

BACKGROUND: After a software incident, organizations attempt to improve the reliability of their software systems by conducting post-incident analysis and identifying actions to prevent or mitigate similar future incidents. Despite the importance of this activity, there is a limited understanding of the nature and characteristics of the software evolution triggered by the analysis process. AIMS: This study seeks to bridge this gap by providing the first empirical characterization of post-incident software evolution. We aim to establish an understanding of how these changes manifest, specifically focusing on how their characteristics (such as effort, location, and timing) differ from standard maintenance and evolutionary changes. METHOD: We conducted a mixed-methods analysis of a 10-year longitudinal dataset from Wikimedia, examining 1,302 tasks and 2,203 code reviews. We categorized these changes across five dimensions: target, effort, abandonment rate, prioritization, and time-to-resolve. RESULTS: Our results show that the post-incident evolution is distinct from other change types. Post-incident changes are more time-consuming, with a median completion time over twice that of other changes and are disproportionately concentrated in a server configuration repository. CONCLUSION: By identifying a few of the characteristics of post-incident evolution, this work provides a foundation for future research into reliability engineering and learning from failure. We highlight further avenues for exploration necessary to fully understand post-incident evolution. Lastly, our work enables researchers to move beyond high-level analysis and begin developing specialized methods and tooling tailored to the nature of post-incident evolution.

0.21

Discovering Performance Archetypes: Critical-Path-Aware Pattern Analysis and Regression Detection

cs.PF ·2026-09-15 ·Kaveh Shahedi, Heng Li, Maxime Lamothe, Foutse Khomh
top venue similarity 0.57
Abstract

Software performance analysis and prediction requires integrating multiple signals, as code structure alone cannot capture runtime behavior shaped by execution frequency, resource contention, and I/O patterns. We present a critical-path-aware performance analysis methodology that automatically discovers recurring performance patterns by synthesizing static code features, dynamic execution traces, and kernel-level resource data. In a preliminary study across six real-world C/C++ applications (SQLite, OpenSSL, Zstandard, FFmpeg, cURL, and jq), we first empirically confirm that static complexity metrics explain only 10.4% of the variance ($ρ^2$) in critical path execution time, quantifying a gap that, while theoretically expected, had not been measured systematically across applications. Motivated by this finding, we analyze nearly 80,000 critical execution paths and address two research questions. First, we discover 13 distinct performance archetypes: recurring behavioral patterns that appear consistently across different applications, independent of their domain or implementation. Five of these patterns are near-universal and appear in at least five of the six applications studied. Notably, three of these archetypes are present in all six applications, and together, these common patterns account for 56.4% of all observed paths. Each archetype maps to specific resource profiles and optimization strategies that transfer across domains. Second, we leverage these archetypes within a multi-signal regression detection framework that triangulates path structure, resource consumption, and archetype deviations, achieving an F1-score of 0.867 and a 60.4% improvement over resource-only methods.

0.37

Behavioral Regression Detection for Pull Requests via Structured Oracles

cs.SE ·2026-10-26 ·Yiwen Fu, Binhang Qi, Xiao Li, Runlin Liu, Hailong Sun +1 more
benchmark similarity 0.56
Abstract

Software regressions are unintended behavioral changes introduced by code modifications and are hard to detect at pull-request (PR) level. Existing methods rely on incomplete tests or directly analyze PRs via LLMs, lacking structured behavioral reasoning and explicit oracles. We present ORION, a staged framework for PR-level regression detection. ORION extracts PR intent and identifies impacted functions via AST-based impact analysis, then synthesizes function-level structured behavioral oracles encoding preserved/changed behaviors, setup constraints and control-flow hints from PR intent, static risk signals and code changes. Guided by these oracles, ORION generates executable tests, conducts oracle-coverage-guided augmentation and static repair, and differentially runs tests on pre- and post-change code. It classifies PRs with deterministic rules and constrained LLM reasoning. We build RegBench-Py, the first Python PR regression benchmark with 154 verified real-world PRs. ORION achieves F1-scores of 0.6125 (DeepSeek-Chat-V3.2) and 0.5578 (Claude-4.5-haiku), outperforming baselines including Testora, Codex and Claude Code. Compared with PatchGuru, ORION supports broader multi-function PRs while remaining competitive on single-function cases. On 1,459 merged PRs, ORION finds 9 reproducible regression candidates, 8 confirmed by maintainers. It takes ~15 minutes per PR, showing practicality for CI/CD screening.

0.19

SafeLLM4SE: Statistical Evaluation and Reporting for LLM-based Software Engineering Systems

cs.SE cs.LG ·2026-09-29 ·Francisco Ortin
benchmarkcode/data similarity 0.56
Abstract · This work has been submitted for possible publication. Copyright may be transferred without notice, after which this version may no longer b…

Large language models (LLMs) are increasingly used for software engineering tasks, yet their stochastic behavior challenges the validity, reproducibility, and comparability of their evaluations. Conventional practices such as reporting a single output, an average score, best-of-N, or pass@k performance can obscure variability and estimation uncertainty, potentially leading to misleading conclusions about system reliability. This article presents SafeLLM4SE, a practical methodology and reporting standard for statistically principled evaluation of LLM-based software engineering systems. Rather than treating generated outputs as deterministic artifacts, SafeLLM4SE treats them as realizations of a stochastic process and distinguishes quality, stability, and estimation uncertainty. It combines adaptive sampling with confidence intervals, distribution-aware statistical comparisons, effect sizes, and a minimum reporting standard covering model configuration, reproducibility, evaluation procedures, and resource usage. SafeLLM4SE is also provided as an open-source software package available on PyPI, enabling researchers and practitioners to reproduce and extend the methodology. We illustrate its application by comparing two LLMs on HumanEval, a benchmark of programming problems assessed through functional tests.

0.25

From Software History to Executable Regression Tasks: A System for Evaluating Software Agents

cs.SE ·2026-10-18 ·Ruikai Shi, Zhengran Zeng, Rui Xie, Shikun Zhang, Wei Ye
similarity 0.56
Abstract · 22 pages

As software agents increasingly generate and modify code, evaluating their ability to review changes and discover defects becomes important. Such evaluation needs executable project environments for investigation and reliable defect references supported by reproducible behavior. Existing code review benchmarks provide limited support for this combination. We present a system that constructs executable regression instances from historical introducing--fixing commit pairs and developer tests. The system extracts candidate tests from fixing-side changes, then uses constrained agents to transplant them and configure version-specific environments while preserving production code and test expectations. Independent replay and evidence-based review verify whether tests that fail on the introducing version and pass on the fixing version support the proposed regression relationship. Accepted instances retain both versions' replay recipes, validated tests, and execution evidence. Repair-supported analysis and human review add root-cause locations, tested repairs, and causal explanations. These instances support bug detection and localization in code review, and enable bug repair and regression-test generation on the same defects. The system constructs 172 validated instances from 522 candidates across 11 Python projects. We evaluate three software agents with ten models and additional reasoning-effort settings on 83 manually selected, enriched instances across the three tasks. Precise root-line localization remains challenging, while baseline combinations achieve repair and regression-test generation success rates of up to 68.7% and 77.1%, respectively. Leading configurations differ by task, and higher task performance does not necessarily yield greater API-cost efficiency. Our approach provides reusable executable evidence for studying agents' defect discovery, repair, and testing capabilities.

0.17