CS Paper Radar

Towards Automated Recommendation of Steady-State Detection Algorithms for Software Performance Benchmarks

Martin Beseda, Vittorio Cortellessa, Federico Di Menna, Daniele Di Pompeo, Luca Traini

cs.PF cs.SE ·submitted 2026-11-12 09:51 UTC · v1
benchmark

Abstract

Steady-state detection is a central step in software performance benchmarking, yet different detection methods rely on different assumptions and may perform best on different execution profiles. This paper formulates steady-state detection as an algorithm-recommendation problem: given a benchmark time series, predict which of two complementary detectors is more appropriate. The first, CP-SSD, identifies steady state through changepoint-based segmentation of the execution trace, whereas the second, KB-KSSD, uses a kernel-based approach to identify stable behaviour. We propose a data-driven recommender that characterises benchmark time series using automatically extracted features and combines supervised prediction with information derived from groups of time series exhibiting similar behaviour. The approach is evaluated on 407 labelled benchmark time series for which one of the two detectors can be identified as preferable with respect to a reference steady-state index. The proposed recommender selects the better detector correctly for 72.97% of the series, compared with 62.65% when always selecting the detector that performs best most often, corresponding to an absolute improvement of 10.32 percentage points. Paired statistical testing and bootstrap confidence intervals support this improvement. A comparison with a recommender using time-series features alone shows that the cluster-derived information mainly improves interpretability rather than predictive accuracy. Recommendations also remain highly stable when the unsupervised representation is learned from either the labelled subset or the full collection of 5,860 time series. These results demonstrate the feasibility of per-series steady-state detector recommendation while highlighting the need for broader labelled data, particularly for execution profiles favouring the changepoint-based detector.

Closest papers

Nearest neighbours by meaning across the whole index.

Evaluating Change Point Detection Methods for Software Performance Regression Analysis

cs.SE ·2026-10-06 ·Diego Elias Costa, Michele Tucci, Luca Traini, Daniele Di Pompeo, Thomas Bach +6 more
similarity 0.64
Abstract

Performance issues in software systems are a critical quality issue that can erode user trust, violate service-level agreements, and ultimately affect business efficiency. Consequently, software performance engineering has shifted its focus to developing robust techniques to detect performance regressions as early as possible in the development cycle. Performance regression analysis often relies on time series of performance measurements to detect significant changes in performance behavior. Change Point Detection (CPD) methods have been widely used to automate the identification of such changes in various domains, including finance, healthcare, and performance monitoring. However, the effectiveness of these methods for software performance measurements has not been thoroughly evaluated. In this paper, we present a comprehensive study to evaluate the effectiveness of various CPD methods on real-world software performance measurement datasets. We start by collecting performance measurement data from three large software systems and characterizing the unique properties of performance time series data. Then, we undertake a large-scale effort to annotate and evaluate the consistency of human annotators' identification of potential performance changes. Thereafter, we evaluate the accuracy of twelve distinct CPD methods in detecting potential performance changes, providing insights into their applicability and effectiveness in software performance regression analysis.

0.24

Discovering Performance Archetypes: Critical-Path-Aware Pattern Analysis and Regression Detection

cs.PF ·2026-09-15 ·Kaveh Shahedi, Heng Li, Maxime Lamothe, Foutse Khomh
top venue similarity 0.63
Abstract

Software performance analysis and prediction requires integrating multiple signals, as code structure alone cannot capture runtime behavior shaped by execution frequency, resource contention, and I/O patterns. We present a critical-path-aware performance analysis methodology that automatically discovers recurring performance patterns by synthesizing static code features, dynamic execution traces, and kernel-level resource data. In a preliminary study across six real-world C/C++ applications (SQLite, OpenSSL, Zstandard, FFmpeg, cURL, and jq), we first empirically confirm that static complexity metrics explain only 10.4% of the variance ($ρ^2$) in critical path execution time, quantifying a gap that, while theoretically expected, had not been measured systematically across applications. Motivated by this finding, we analyze nearly 80,000 critical execution paths and address two research questions. First, we discover 13 distinct performance archetypes: recurring behavioral patterns that appear consistently across different applications, independent of their domain or implementation. Five of these patterns are near-universal and appear in at least five of the six applications studied. Notably, three of these archetypes are present in all six applications, and together, these common patterns account for 56.4% of all observed paths. Each archetype maps to specific resource profiles and optimization strategies that transfer across domains. Second, we leverage these archetypes within a multi-signal regression detection framework that triangulates path structure, resource consumption, and archetype deviations, achieving an F1-score of 0.867 and a 60.4% improvement over resource-only methods.

0.37

TSB-SEG: A Systematic Time-Series Segmentation Benchmark

cs.LG ·2026-11-02 ·Félix Chavelli, Arik Ermshaus, Fan Yang, Patrick Schäfer, John Paparrizos +1 more
publishedbenchmarkcode/data similarity 0.59
Abstract · NeurIPS 2026, Track on Evaluations and Datasets. Code and results: https://github.com/fchavelli/TSB-SEG

Time-series segmentation, studied as either Change Point Detection (CPD) or State Detection (SD), underpins a broad range of monitoring and diagnostic applications. However, the field suffers from a persistent divide between the statistical community (focusing on CPD) and the machine learning and data mining communities (focusing on SD). Consequently, the literature remains fragmented: evaluations typically cover narrow methodological families, datasets are often small or homogeneous, and CPD and SD are rarely evaluated together. We address these gaps with TSB-SEG, a unified benchmark comprising 592 univariate and multivariate time series from 8 heterogeneous datasets and surveying 26 algorithms that span five decades of progress. To ensure a fair comparison, we introduce a two-pool design: a scalable main pool of tractable detectors used for the aggregate ranking, and an extended-scope pool for computationally intensive methods. All methods are unified under tsseg, an open-source library to ensure full reproducibility. Our systematic evaluation yields three insights: (a) no single detector dominates across domains, and per-dataset hyperparameter tuning remains decisive for performance, (b) CPD accuracy is a useful, though imperfect, proxy for SD accuracy, and (c) data characteristics should guide method selection. We provide actionable guidance based on data properties, error tolerances, and tuning budgets, identify impactful hyperparameters, and characterize algorithm robustness to these choices.

0.43

Less is more: learning what to profile for GPU kernel optimization

cs.LG cs.AI ·2026-10-27 ·Yihan Wang, Peiyu Liu, Runyu Chen, Wei Xu
code/datahot direction similarity 0.58
Abstract

Large language models have recently shown strong potential for automatically optimizing GPU kernels, yet their effectiveness depends critically on the hardware information available to the kernel optimization model. Existing systems typically rely on predefined profiling strategies, leaving what to profile largely outside the optimization process. We show that effective profiling is not simply about collecting more information: increasing the number of profiling metrics does not monotonically improve kernel optimization, while the most useful profiling information varies across kernels. Our theoretical analysis further shows that, under such heterogeneous profiling utility, kernel-conditioned selection can outperform any single fixed profiling configuration. Motivated by this observation, we propose AutoProfiler, a feedback-driven framework for learning a deterministic profiling tree from optimization feedback. The tree is iteratively refined through REWRITE and SPLIT operations, with an LLM generating candidate modifications and empirical evaluation determining their acceptance. Once constructed, the tree quickly gathers the required profiling information for any kernel, allowing the LLM rewriter to be invoked only once. Experiments on KernelBench show that our approach improves kernel optimization over fixed profiling configurations under constrained profiling budgets. These results suggest that hardware profiling should evolve from static metric collection toward profiling reasoning, making profiling itself a new dimension for improving kernel optimization. The codes are available at https://github.com/Satissss/AutoProfiler.

0.31

FORTE: A Production Framework for Feature Evaluation and Deprecation in Industrial Recommender Systems

cs.LG cs.AI ·2026-10-09 ·Vishnu Arun, Adam Obeng, Tianyou Wang, Zilin Ren, Mukuntha Narayanan Sundararaman +5 more
similarity 0.58
Abstract

Production recommender systems have feature-lifecycle asymmetry: adding new features must produce gains on some metric, while feature removal must not regress any metric. In practice, feature deprecation is hard to prioritize as a feature can fail to be removed if it interacts with other features, its importance changes daily, it cannot be evaluated before production models change or data drift occurs, it is in contention for A/B test space with other launches, it causes regressions in an online A/B test, or it is consumed by multiple models that need to coordinate its deprecation. We present FORTE, a model-agnostic production framework for discovering, validating, and removing features. Its efficient perturbation implementation allows daily feature evaluation. To account for interactions, it evaluates candidate feature sets constructed from all the low-rank features through grouped ablation and its offline results are sufficiently predictive of online A/B tests to allow deprecations to proceed to online testing with confidence. Lineage tracing identifies all downstream consumer models to realize the full savings of removing a feature from the training set. We validate FORTE over 2 large-scale recommender systems and find that it identifies deprecable feature sets which do not cause online regressions, while the impact on efficiency depends on model characteristics. On a search-relevance model, FORTE resulted in 3 nested candidate sets of 32, 48, and 59 features. Online guardrails rejected the largest treatment. The 48-feature set was selected for deprecation, removing 43.3% of the model's features. On a large ranking model, FORTE identified 8 candidates for removal. Retraining and A/B testing confirmed no regression. Its launch reduced the trainable parameters by 29.9%, improved serving throughput by 19.8-25.5%, and reduced annual training costs by hundreds of thousands of dollars.

0.21

AdapBench: Adaptive Early-Stopping and Prioritization for Efficient Microbenchmarking in Cloud-based CI/CD

cs.SE cs.DCcs.PF ·2026-10-31 ·Sebastian Koch, Jonas Heisterberg, Nils Japke, Trever Schirmer, David Bermbach
publishedcode/data similarity 0.57
Abstract · 11 pages, 3 figures, 3 tables. Accepted at the 19th IEEE/ACM International Conference on Utility and Cloud Computing (UCC 2026). Code and da…

Cloud-based software systems frequently release new versions, a fact further accelerated through agentic coding. Each new release, however, brings the risk of performance regressions which can irritate end users, increase infrastructure costs, and increase carbon footprints. Therefore, developers strive to catch such regressions in testing stages, usually through so-called continuous benchmarking. The standard way of doing this is using microbenchmark suites which, however, take notoriously long to execute. Optimizing the execution of microbenchmark suites is, hence, an active field of research. In this paper, we propose AdapBench, an online scheduling system for reducing the execution duration of microbenchmark suites. AdapBench is the first approach which dynamically makes at-runtime decisions for prioritizing the most informative benchmarks and early-stopping each one once enough evidence has accumulated to accept or reject a performance change hypothesis. We evaluate our prototype on eight popular Java projects, commonly used by microbenchmark research, and four additional Go projects. At matched 90% change detections and false-positive rate compared to an optimistic uniform run, AdapBench saves 42% of the aggregate execution duration on the Java projects, with negligible scheduling overhead (below 0.001% of the total runtime). On the Go projects, AdapBench finds every performance change with zero false positives while saving 85% of the full run.

0.49

Concept Drift Detection and Adaptive Retraining of Malware Classification Models

cs.LG cs.AIcs.CR ·2026-08-13 ·Christofer Washington Berruz Chungata, Martin Jurecek, Katerina Potika, William B. Andreopoulos, Mark Stamp
top venue similarity 0.57
Abstract · To appear as a chapter in the book "Artificial Intelligence for Cyber Defense in Emerging Threats", to be published by Springer by early 202…

Concept drift refers to changes over time in the statistical properties of data, as compared to the data that was used to train a learning model. Machine learning models for malware detection or classification are particularly susceptible to performance degradation caused by concept drift, as attackers constantly modify existing malware. In this chapter, we analyze two machine learning-based approaches to automated concept drift detection-a novel approach based on One-Class Support Vector Machines (OCSVM) and a previously-studied technique based on Minibatch K-Means (MK-Means). For comparison we also consider Maximum Mean Discrepancy (MMD), a statistical technique for detecting changes in multidimensional data. We conduct an extensive series of experiments comparing the effectiveness of four learning models, namely, Multilayer Perceptron, Random Forest, Support Vector Machines, and eXtreme Gradient Boosting. For each of these models, we consider three distinct scenarios: A static scenario where no model retraining occurs, a periodic scenario where models are constantly retrained irrespective of concept drift, and a drift-aware scenario where models are only retrained when concept drift is detected. Under the drift-aware scenario, we analyze the tradeoff between accuracy and training efficiency using Pareto Front analysis. We find that all three concept drift detection techniques achieve classification accuracy comparable to periodic retraining, while offering substantially greater efficiency in terms of the number of models that must be retrained. In addition, drift-aware retraining based on our OCSVM technique generally outperforms the MK-Means and MMD approaches. Overall, these results provide strong evidence that we can accurately detect concept drift in malware classification models.

0.31

Caprese: Change Impact Recommendation Using History and Runtime Analysis

cs.SE ·2026-11-11 ·Sadjad Tavakoli, Saba Alimadadi
code/data similarity 0.57
Abstract

Change impact recommendation identifies program entities potentially affected by source code changes. Historical co-changes and runtime dependencies provide complementary evidence, particularly in JavaScript, where dynamic and asynchronous behaviour complicates dependency analysis. We present Caprese, an open-source framework for function-level change impact recommendation in Node.js. Given a change set, it produces a ranked impact set of potentially affected functions. It supports repository-based recommendation through confidence-guided mining of recurring, change-set-relevant co-change patterns of varying lengths; dynamic recommendation through execution-derived dependencies involving calls, callbacks, events, timers, and shared state; and hybrid recommendation integrating both analyses. Its modular implementation supports reusable analyses. Evaluation on ten open-source applications and 100 selected change sets shows that, on average across projects, 77.4% of the relevant functions among the top 60 hybrid recommendations are identified exclusively by one analysis, demonstrating substantial complementarity. Dynamic recommendation achieves the highest mean precision among the top five recommendations (0.65), while repository-based recommendation performs best when the ranking is extended to 60 recommendations (0.42). Hybrid recommendation outperforms TARMAQ on average but does not consistently outperform the individual modes. Caprese is publicly available with source code, documentation, and evaluation artifacts.

0.55