CS Paper Radar

CTML: A Comprehensive Framework for Continuous Testing in Machine Learning Operations

Bestoun S. Ahmed

cs.SE ·submitted 2026-10-07 18:45 UTC · v1

Abstract

The increasing use of machine learning (ML) systems in production has highlighted some important weaknesses in traditional software testing methods. Unlike regular software, ML systems can fail in specific ways due to their reliance on data, including issues like data drift, concept drift, and performance decline. To address these issues, this paper introduces the Continuous Testing for Machine Learning (CTML) framework. This framework is designed specifically for MLOps environments and offers a thorough, automated solution. It establishes a standardized way to evaluate ML systems across four main areas: data quality, drift detection, model quality, and pipeline validation. The CTML framework consists of a modular setup that includes a Test Registry, Test Runner, Result Store, and Integration Layer. This design ensures it integrates smoothly with existing Continuous Integration/Continuous Delivery (CI/CD) pipelines and MLOps platforms. We put this framework to the test in a predictive maintenance case study, which showed how effective it is at identifying issues like data drift, performance degradation, and bottlenecks in the pipeline problems that traditional testing methods might overlook. The findings indicate that CTML successfully uncovers critical failures in the system across all testing dimensions, providing useful insights to help keep ML systems running smoothly in production. Overall, this work presents a new, integrated framework that connects academic research with industry best practices, pushing forward the standards in MLOps quality assurance.

Closest papers

Nearest neighbours by meaning across the whole index.

MLOps-Bench: Benchmarking AI Agents on Production ML Engineering in Real Repositories

cs.SE ·2026-10-16 ·Quoc Co Tran, Hitesh Laxmichand Patel, Diwakar Mahajan, Kshitij Bakliwal, Avi Sil +1 more
top venuebenchmark similarity 0.67
Abstract

AI agents are increasingly expected to modify and maintain machine-learning systems in production, yet existing benchmarks primarily evaluate either competition-style ML experimentation or generic repository issue resolution. We introduce MLOps-Bench, a repository-grounded benchmark for production ML engineering. Unlike prior ML-agent benchmarks, MLOps-Bench evaluates executable lifecycle changes inside real ML repositories, spanning data pipelines, feature engineering, model training, model serving, monitoring, CI/CD and governance, and end-to-end integration. Each task is grounded in repository evidence and includes a natural-language requirement, oracle patch, plausible buggy baseline, fail-to-pass tests, pass-to-pass tests, and provenance metadata. The benchmark contains 212 validated tasks from 31 public ML repositories, with 636 fail-to-pass tests and 856 pass-to-pass tests. We evaluate six agents and find that the best system solves 31.2% of tasks while passing 68.7% of tests, showing that current agents often make partial progress but still fail production-relevant behavioral requirements. MLOps-Bench shifts agent evaluation beyond model building and issue fixing toward repository-native MLOps work where correctness depends on lifecycle reasoning, behavioral robustness, and preservation of existing system behavior.

0.36

Characterizing and Predicting the Longest CI Builds in Machine Learning Projects

cs.SE ·2026-10-31 ·Yonghui Huang, Daniel Alencar da Costa, Mohamed Sami Rakha, Grant Dick
code/data similarity 0.66
Abstract

Continuous integration (CI) builds in machine learning (ML) projects tend to run longer than in other projects, yet little is known about the longest builds, which consume most CI time. This paper characterizes the longest builds in ML projects, analyzes how CI workflow properties are associated with them, and evaluates whether they can be predicted before execution. We study 269,930 GitHub Actions builds from 107 open-source ML projects. The top 18% longest builds account for 80.30% of total build time and fail in 39.09% of cases. Build duration differs significantly across ML project types. The framework and infrastructure ML projects have the longest upper tail and produce 98% of the longest builds, whereas projects that use existing ML libraries have the lowest median build duration. The longest builds are most strongly associated with the recent history of the same workflow, as the duration of its last build and the median duration of the project's last 20 builds show large effect sizes. The longest builds are more consistently associated with CI workflow structure than with configuration. The number of jobs, control flow complexity, coupling complexity, and maximum steps per job show medium effect sizes, whereas caching, matrix settings, and Docker builds show small or negligible effects. Framework-specific workflows have the largest percentage of builds among long builds. Platform-specific workflows produce the largest number of the longest builds. Finally, 47 features describing build history, workflow structure, and the ML project outperform a traditional baseline with all four classifiers. In a chronological evaluation, the best model (XGBoost) reaches an F1-score of 0.775, compared with 0.682 for the baseline, and an AUC-ROC of 0.955. These results show that build history and CI workflow structure help identify the longest builds in ML projects before they run.

0.34

Counterfactual Model Testing Under Distribution Shift

cs.AI cs.LG ·2026-10-24 ·Joran Leest, Ilias Gerostathopoulos, Claudia Raibulet, Patricia Lago
similarity 0.64
Abstract

Deployed machine learning (ML) models can degrade as the distribution of their input data shifts. Such shifts can result from changes in the population or from data integrity issues in the pipeline that materializes model features, two sources that operate through different mechanisms. Because labels are often absent or delayed, it is common practice to track shifts in the data instead, but a detected shift does not indicate how it affects the model. We formulate this problem as online model testing under distribution shift. Testing requires reproducing the shift on a controlled baseline with known labels. Since the same shift can come from either source, doing so requires knowing which mechanism changed. We propose a methodology that reproduces observed changes in both mechanisms to estimate current performance and the potential gain from repairing observed data integrity issues. Experiments on real-world datasets with simulated faults show that separating the two sources generally improves both estimates, supporting this formulation. A churn prediction case illustrates how these tests can support monitoring and repair prioritization

0.13

Reshaping the SDLC for Data- and AI-Centric Systems

cs.SE ·2026-08-18 ·Mamdouh Alenezi
similarity 0.62
Abstract

The traditional Software Development Lifecycle (SDLC) assumes that system behavior is determined primarily by source code, allowing correctness to be specified, implemented, and verified through code-centric practices. Data-intensive and AI-enabled systems challenge this assumption because their behavior emerges from the interaction of code, data, and learned models, while performance may degrade as real-world conditions drift from training data. This paper examines how integrating data engineering and software engineering practices, operationalized through DataOps, MLOps, and LLMOps, reshapes the SDLC for these systems. We make four contributions. First, we synthesize literature across software engineering, data management, machine learning systems, and human-centered computing into a phase-structured account of lifecycle transformation spanning requirements, architecture, development, testing, deployment, monitoring, governance, and organization. Second, we provide a lightweight formalization in which system behavior is defined over code, data, and model configurations; requirements become evaluation-led specifications with probabilistic acceptance regions; and promotion is controlled through statistically grounded validation gates. Third, we develop an adaptive five-layer lifecycle framework comprising artifact, contract, gate, control, and governance layers, positioning maintenance as a closed-loop control problem under configuration drift. Fourth, we propose a conceptual research model linking data engineering integration to measurable lifecycle outcomes and critically assess the evidence base. While the direction of transformation is increasingly established, its magnitude remains insufficiently quantified. We conclude with a research agenda for an empirically grounded, adaptive SDLC for data- and AI-centric systems.

0.30

A Hybrid Rule-Based and AI-Augmented Framework for Automatic Failure Recovery in DevOps Deployments

cs.SE cs.DC ·2026-09-21 ·Raju Chowdhury, Aniruddha Singh Gautam, Apurv Ghai
similarity 0.61
Abstract · This is the preprint version of a paper accepted at the ETNCC 2026 Conference. The final published version will appear in IEEE Xplore. This …

Automatic failure recovery is another difficult area of DevOps deployments due to the complexity of the system, high workload and the limitations of the traditional rules-based or manual approach. This study proposes a hybrid model integrating deterministic, rule-based recovery with the assistance of machine learning by fault prediction to automatically monitor failures, identify, classify and recover failures in real time. To test the architecture, a distributed log dataset of 100,000 records was used for models such as Decision Tree, Random Forest, Logistic Regression, LightGBM, Autoencoder with BiLSTM. The Decision Tree had a strong performance in defect identification with an F1 score of 80.8%, recall of 80.0%, precision of 81.6%, and accuracy of 89.4%. An impressive 83.3% success rate was achieved by the automated recovery measures, resulting in a 94.6% decrease in mean time to recovery (MTTR), a 95.4% reduction in downtime, and annual savings of 7,462. The combination of explainable rule-based logic and adaptive AI prediction in the framework means it is more resilient, operationally efficient, and scalable, and an effective, autonomous approach to DevOps environments evaluated under controlled experimental conditions.

0.35

SAGE-Loop: Reliable Closed-Loop LLM-Driven AutoML with Trial-and-Correction and Adaptive Ensembling

cs.LG ·2026-09-11 ·Junquan Gu, Shibo Cui, Xiangfeng Luo, Hang Yu
similarity 0.59
Abstract · 13 pages, 7 figures, 4 tables

Automated machine learning (AutoML) is reshaping data-driven science and industrial practice, and as large language models are introduced into AutoML, pipeline reliability becomes as important as automation efficiency. However, existing AutoML still struggles to realize instant feedback and adaptive optimization during execution, so once a run drifts into a suboptimal or failed state, it lacks a process-level correction mechanism. The fundamental pathology lies in its one-way pipeline: intermediate failures are typically terminated or bypassed, while fixed paradigms often strengthen model generation but leave ensemble decisions static, weakening both execution reliability and the controlled use of structural diversity. This indicates that LLM-driven AutoML needs a closed-loop ability for trial-correction-improvement together with evidence-based use of model diversity. To this end, we propose SAGE-Loop, a reliable closed-loop, self-adaptive, LLM-driven AutoML framework that performs multi-round generation and validation for trial-and-repair, and adaptively selects ensemble strategies in both supervised and unsupervised tasks, thereby unifying how to generate with how to use models. Across 20 public datasets, SAGE-Loop consistently improves performance and stability on classification, regression, and clustering tasks. Additional results further show its ability to recover from execution failures and maintain robust pipeline behavior.

0.17

Making Deployments Safe at Meta: Health Checks for Continuous Change-Safety

cs.SE cs.AI ·2026-08-20 ·Prakash KL, Anton Korenkov, Uttam Thakore, Christopher Hegre
top venue similarity 0.56
Abstract · 6 pages, 4 figures, Accepted to ISSRE 2026 Industry Track

Continuous deployment to large scale production systems creates a tension between release velocity and reliability. Every change is a potential reliability incident, yet every delay is a missed opportunity. This paper describes the deployment time health check infrastructure that Meta uses to mediate this tension across thousands of heterogeneous services. We summarize the architecture of this prevention based distributed system's service called Service Health Checker, explain how check authors compose templated metric queries, thresholds, and workflow predicates; and discuss how the system is integrated with tiered and phased rollouts so that regressions trigger automatic rollback. We then describe the operational problems that emerged at scale, such as noise, alert fatigue, drift, and uncovered regressions, and the program of measurement, tooling, and improved defaults we deployed to address them. We close with lessons learned from years of operating deployment health checks at Meta, and the directions we are exploring next, including AI assisted health check tuning. Index Terms: deployment safety, continuous deployment, monitoring, software reliability, release engineering, software reliability engineering, AIOps, anomaly detection

0.36

Evaluating Automated Testing on an Open-Source Web Application Using Cypress

cs.SE cs.ET ·2026-08-20 ·Quoc-Binh Nguyen, Truc-Ly Phan Nguyen, Ngoc Hong Tran, Dung Hai Dinh
publishedcode/data similarity 0.56
Abstract · Accepted by the 9th International Conference on Research in Engineering and Technology (RET 2026)

End-to-end automated testing is increasingly used in web software development to ensure system quality and shorten response times during development. However, the true effectiveness of automated testing depends on many factors including execution time, stability of test results, and maintainability of the test suite as the application continues to evolve. In this paper, we evaluate the effectiveness of end-to-end automated testing using the Cypress framework for an open-source web application. We deployed the experiment with 27 test cases. The test's effectiveness is measured by execution speed, reliability, and maintainability. The experimental results show that the Cypress-based end-to-end test suite has short and stable execution times. It suits frequent runs during software development. The majority of test cases achieved consistent results across multiple runs, while flakiness only occurred in a few tests which involve complex interactive functions. Furthermore, the study highlights the impact of element locator strategies and Page Object Model (POM) architecture on test suite maintainability, demonstrating that resilient data-cy attributes significantly reduce maintenance overhead when UI changes occur.

0.53