CS Paper Radar

Characterizing and Predicting the Longest CI Builds in Machine Learning Projects

Yonghui Huang, Daniel Alencar da Costa, Mohamed Sami Rakha, Grant Dick

cs.SE ·submitted 2026-10-31 06:06 UTC · v1
code/data

Abstract

Continuous integration (CI) builds in machine learning (ML) projects tend to run longer than in other projects, yet little is known about the longest builds, which consume most CI time. This paper characterizes the longest builds in ML projects, analyzes how CI workflow properties are associated with them, and evaluates whether they can be predicted before execution. We study 269,930 GitHub Actions builds from 107 open-source ML projects. The top 18% longest builds account for 80.30% of total build time and fail in 39.09% of cases. Build duration differs significantly across ML project types. The framework and infrastructure ML projects have the longest upper tail and produce 98% of the longest builds, whereas projects that use existing ML libraries have the lowest median build duration. The longest builds are most strongly associated with the recent history of the same workflow, as the duration of its last build and the median duration of the project's last 20 builds show large effect sizes. The longest builds are more consistently associated with CI workflow structure than with configuration. The number of jobs, control flow complexity, coupling complexity, and maximum steps per job show medium effect sizes, whereas caching, matrix settings, and Docker builds show small or negligible effects. Framework-specific workflows have the largest percentage of builds among long builds. Platform-specific workflows produce the largest number of the longest builds. Finally, 47 features describing build history, workflow structure, and the ML project outperform a traditional baseline with all four classifiers. In a chronological evaluation, the best model (XGBoost) reaches an F1-score of 0.775, compared with 0.682 for the baseline, and an AUC-ROC of 0.955. These results show that build history and CI workflow structure help identify the longest builds in ML projects before they run.

Closest papers

Nearest neighbours by meaning across the whole index.

CTML: A Comprehensive Framework for Continuous Testing in Machine Learning Operations

cs.SE ·2026-10-07 ·Bestoun S. Ahmed
similarity 0.66
Abstract

The increasing use of machine learning (ML) systems in production has highlighted some important weaknesses in traditional software testing methods. Unlike regular software, ML systems can fail in specific ways due to their reliance on data, including issues like data drift, concept drift, and performance decline. To address these issues, this paper introduces the Continuous Testing for Machine Learning (CTML) framework. This framework is designed specifically for MLOps environments and offers a thorough, automated solution. It establishes a standardized way to evaluate ML systems across four main areas: data quality, drift detection, model quality, and pipeline validation. The CTML framework consists of a modular setup that includes a Test Registry, Test Runner, Result Store, and Integration Layer. This design ensures it integrates smoothly with existing Continuous Integration/Continuous Delivery (CI/CD) pipelines and MLOps platforms. We put this framework to the test in a predictive maintenance case study, which showed how effective it is at identifying issues like data drift, performance degradation, and bottlenecks in the pipeline problems that traditional testing methods might overlook. The findings indicate that CTML successfully uncovers critical failures in the system across all testing dimensions, providing useful insights to help keep ML systems running smoothly in production. Overall, this work presents a new, integrated framework that connects academic research with industry best practices, pushing forward the standards in MLOps quality assurance.

0.24

Sustainability of Open-Source Machine Learning Robustness Assessment Tools: A Repository Mining Study

cs.SE ·2026-08-28 ·Joshua Owotogbe, Indika Kumara, Willem-Jan van den Heuvel, Damian Tamburri
code/data similarity 0.61
Abstract · 32 pages. Preprint

Robustness evaluation is essential for deploying machine-learning (ML) systems in real-world settings, where models may face adversarial perturbations, distribution shifts, and other operational stressors. Many open-source tools, including Adversarial Robustness Toolbox, Foolbox, and Robustness Gym, support robustness testing and evaluation. However, little is known about how these tools are maintained, publicly engaged with, and sustained over time, even though practitioners may rely on them to select evaluation dependencies, reproduce robustness assessments, and provide evidence for AI assurance. We present an empirical study of the open-source robustness tooling ecosystem. Starting from a curated seed set derived from prior work, we systematically searched GitHub and identified 28 robustness-tool repositories. We analyzed repository artifacts to characterize observable community engagement, maintenance activity, and project longevity using established software-engineering metrics. Our results show that engagement and maintenance are unevenly distributed, with sustained activity concentrated in a small subset of repositories. At the data collection date of January 21, 2026, five repositories were classified as active, 22 as inactive, and one as archived. These findings highlight the need to treat robustness tools as evolving software systems.

0.35

MLOps-Bench: Benchmarking AI Agents on Production ML Engineering in Real Repositories

cs.SE ·2026-10-16 ·Quoc Co Tran, Hitesh Laxmichand Patel, Diwakar Mahajan, Kshitij Bakliwal, Avi Sil +1 more
top venuebenchmark similarity 0.60
Abstract

AI agents are increasingly expected to modify and maintain machine-learning systems in production, yet existing benchmarks primarily evaluate either competition-style ML experimentation or generic repository issue resolution. We introduce MLOps-Bench, a repository-grounded benchmark for production ML engineering. Unlike prior ML-agent benchmarks, MLOps-Bench evaluates executable lifecycle changes inside real ML repositories, spanning data pipelines, feature engineering, model training, model serving, monitoring, CI/CD and governance, and end-to-end integration. Each task is grounded in repository evidence and includes a natural-language requirement, oracle patch, plausible buggy baseline, fail-to-pass tests, pass-to-pass tests, and provenance metadata. The benchmark contains 212 validated tasks from 31 public ML repositories, with 636 fail-to-pass tests and 856 pass-to-pass tests. We evaluate six agents and find that the best system solves 31.2% of tasks while passing 68.7% of tests, showing that current agents often make partial progress but still fail production-relevant behavioral requirements. MLOps-Bench shifts agent evaluation beyond model building and issue fixing toward repository-native MLOps work where correctness depends on lifecycle reasoning, behavioral robustness, and preservation of existing system behavior.

0.36

FLOWSTEP: A Framework for Learning to Predict GitHub Actions Workflow STEPs

cs.SE ·2026-10-26 ·Jiangnan Huang, Bin Lin, Yutaro Kashiwa
similarity 0.59
Abstract

Continuous Integration and Delivery (CI/CD) are fundamental to modern software development, yet configuring pipelines such as GitHub Actions (GHA) workflows remains complex and error-prone. Recent work has explored Transformer-based models for GHA workflow recommendation, but existing approaches typically model workflow steps as a single homogeneous generation task. This formulation overlooks a key distinction: reusable Actions are structured, ecosystem-level components, whereas project-specific Scripts encode local build, test, and deployment semantics. Consequently, current models struggle to capture the different forms of knowledge required for accurate workflow completion. In this work, we present FLOWSTEP, a framework for GHA workflow step completion that explicitly distinguishes between Action and Script generation. FLOWSTEP combines large language models (LLMs) with contrastive learning to better capture semantic relationships among workflow contexts, Actions, and Scripts. Evaluated on established benchmarks, FLOWSTEP improves end-to-end workflow step prediction accuracy from 38.23% to 51.09%. Beyond prediction accuracy, we further study whether LLM-as-a-judge can provide a more semantic evaluation of generated workflow steps. By comparing LLM-based assessments with human annotations, we find that the best-performing model configuration reaches substantial agreement with human ground truth (Krippendorff's alpha = 0.701), suggesting that LLM-as-a-judge can serve as a scalable complement to manual evaluation.

0.08

BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection

cs.AI cs.LGcs.MAcs.SE ·2026-11-18 ·John Wu, Rushi Qiang, Yuchen Zhuang, Guowang Li, Girish Narayanswamy +8 more
similarity 0.58
Abstract

Enabling AI systems to autonomously improve AI models requires agents capable of performing machine learning engineering (MLE). In practice, designing, training, and evaluating complex models demands modifying multi-file repositories to reproduce results, debug training runs, integrate datasets, and port frameworks. However, existing MLE benchmarks limit agents to single-file edits without full repository or shell access, and common repository-level software engineering benchmarks have rapidly saturated. To address this, we introduce the Code Complexity Index (CCI), a difficulty metric based on ten structural features spanning repository scale, dependency depth, and patch size. CCI strongly predicts average solve rates across eight public coding benchmarks ($R^2 = 0.951$), identifying patch size and complexity as top predictors. Guided by CCI, we develop BuggyMLE, a scalable framework where agents inject controlled mutations into working ML repositories, synthesizing over 6,600 bug-injected MLE repository variants and nearly 260K lines of injected code based on 222 source repositories. Using BuggyMLE, we make two contributions. First, we construct BuggyMLE-Bench-Lite, the first difficulty-graded, repository-level variant of MLE-Bench Lite; across 14 evaluated models, the top solve rate drops from 95% to 9% as difficulty is scaled from the easiest to the hardest tier. Second, we synthesize a training curriculum across three difficulty levels over 200 additional ML repositories, fine-tuning a Qwen3.5-9B model to rival frontier models on BuggyMLE-Bench-Lite. Our trained model transfers beyond ML, raising its SWE-bench Lite resolve rate from 24% to 33% and its LiveCodeBench v6 pass@1 from 69% to 71%.

0.08

ML-OPF-Bench: Benchmarking Machine Learning for Optimal Power Flow

eess.SY cs.LG ·2026-10-03 ·Xinyi Liu, Xuan He, Danny H. K. Tsang, Yize Chen
benchmarkcode/data similarity 0.57
Abstract · 25 pages, 9 figures, in submission. Code available at https://github.com/XinyiLiu04/ML-OPF-Bench; Datasets available at https://huggingface.…

Machine Learning (ML) methods promise a fast solution process for Optimal Power Flow (OPF). While inconsistent test cases, implementations, and evaluation metrics across existing studies make it challenging to determine which algorithmic advances are most critical for real-world deployment. To this end, we propose ML-OPF-Bench, a unified benchmark for AC- and DC-OPF that evaluates representative ML algorithms under a consistent pipeline, stress-tests them across system sizes, distribution shifts, and resource budgets, and ranks them with a multi-objective framework. We find that prediction accuracy alone is not a reliable indicator of operational feasibility. Under heavily loaded, congested conditions, even the strongest in-distribution performers lose their advantage, while feasibility is maintained largely by post-processing that enforces the target constraints rather than by the underlying pure ML predictor. Data scaling shows that prediction accuracy and constraint violations follow different trajectories, whereas compute scaling shows that returns diminish and that larger models do not consistently perform better. These results expose critical trade-offs among ML methods' speed, accuracy, and feasibility, and offer practical guidance for future ML-OPF design. We open-source the benchmark as an extensible Python package for integrating new learning-based OPF algorithms and evaluating them under the same standard as the existing baselines.

0.33

Reshaping the SDLC for Data- and AI-Centric Systems

cs.SE ·2026-08-18 ·Mamdouh Alenezi
similarity 0.57
Abstract

The traditional Software Development Lifecycle (SDLC) assumes that system behavior is determined primarily by source code, allowing correctness to be specified, implemented, and verified through code-centric practices. Data-intensive and AI-enabled systems challenge this assumption because their behavior emerges from the interaction of code, data, and learned models, while performance may degrade as real-world conditions drift from training data. This paper examines how integrating data engineering and software engineering practices, operationalized through DataOps, MLOps, and LLMOps, reshapes the SDLC for these systems. We make four contributions. First, we synthesize literature across software engineering, data management, machine learning systems, and human-centered computing into a phase-structured account of lifecycle transformation spanning requirements, architecture, development, testing, deployment, monitoring, governance, and organization. Second, we provide a lightweight formalization in which system behavior is defined over code, data, and model configurations; requirements become evaluation-led specifications with probabilistic acceptance regions; and promotion is controlled through statistically grounded validation gates. Third, we develop an adaptive five-layer lifecycle framework comprising artifact, contract, gate, control, and governance layers, positioning maintenance as a closed-loop control problem under configuration drift. Fourth, we propose a conceptual research model linking data engineering integration to measurable lifecycle outcomes and critically assess the evidence base. While the direction of transformation is increasingly established, its magnitude remains insufficiently quantified. We conclude with a research agenda for an empirically grounded, adaptive SDLC for data- and AI-centric systems.

0.30

About the Influence of Workflow Topology on Task Intensity Prediction through Graph Learning

cs.DC cs.LG ·2026-09-30 ·Max Otto, Haci Ismail Aslan, Joel Witzke, Jonathan Bader, Odej Kao
publishedbenchmark similarity 0.57
Abstract · Published at the IEEE 26th International Symposium on Cluster, Cloud and Internet Computing Workshops (CCGridW)

Efficient resource provisioning for large-scale workflows on cloud infrastructures is a critical performance engineering challenge. These workflows are often structured as directed acyclic graphs (DAGs), where under-provisioning can cause critical bottlenecks and over-provisioning leads to unnecessary costs. Accurate, task-level prediction of resource intensity (e.g., CPU load and memory usage) is essential for mitigating these issues. While task-level features are commonly used for prediction, the performance impact of the workflow's overall topological structure is often overlooked or assumed. The central question of our work is: To what extent does what part of the DAG topology influence task-level resource intensity, and what is the most effective way to model this influence? This paper presents a comprehensive benchmark to systematically quantify the impact of graph topology on task intensity prediction. We evaluate and compare a spectrum of modeling approaches. Our findings demonstrate that topology is a critical feature for accurate prediction. Models incorporating important topological information, even through simple handcrafted features, significantly outperform baseline models. We show that graph-native models provide the highest accuracy, achieving low mean absolute errors for both CPU and memory predictions, and can still be combined with simple topological features that they do not learn for better performance.

0.32