CS Paper Radar

A Hybrid Rule-Based and AI-Augmented Framework for Automatic Failure Recovery in DevOps Deployments

Raju Chowdhury, Aniruddha Singh Gautam, Apurv Ghai

cs.SE cs.DC ·submitted 2026-09-21 20:36 UTC · v1

Abstract

Automatic failure recovery is another difficult area of DevOps deployments due to the complexity of the system, high workload and the limitations of the traditional rules-based or manual approach. This study proposes a hybrid model integrating deterministic, rule-based recovery with the assistance of machine learning by fault prediction to automatically monitor failures, identify, classify and recover failures in real time. To test the architecture, a distributed log dataset of 100,000 records was used for models such as Decision Tree, Random Forest, Logistic Regression, LightGBM, Autoencoder with BiLSTM. The Decision Tree had a strong performance in defect identification with an F1 score of 80.8%, recall of 80.0%, precision of 81.6%, and accuracy of 89.4%. An impressive 83.3% success rate was achieved by the automated recovery measures, resulting in a 94.6% decrease in mean time to recovery (MTTR), a 95.4% reduction in downtime, and annual savings of 7,462. The combination of explainable rule-based logic and adaptive AI prediction in the framework means it is more resilient, operationally efficient, and scalable, and an effective, autonomous approach to DevOps environments evaluated under controlled experimental conditions.

Comment
This is the preprint version of a paper accepted at the ETNCC 2026 Conference. The final published version will appear in IEEE Xplore. This preprint does not include IEEE formatting or copyright. After publication, the accepted manuscript with IEEE copyright notice and DOI will be added

Closest papers

Nearest neighbours by meaning across the whole index.

Scalable and Data-Driven Decision Support in the Maintenance, Repair, and Overhaul Process

cs.AI cs.LG ·2026-09-27 ·Houkun Zhu, Helena Ebel, Dominik Scheinert, Florian Schmidt, Jens Altenkirch +1 more
published similarity 0.62
Abstract · Published in: 2022 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM)

Several businesses apply maintenance, repair, and overhaul (MRO) principles to the life-cycle of their existing products. In cases like casted gas turbine component Product Lifecycle Management (PLM), repairing components in frequent intervals can extend the lifetime expectation of the product, provide higher cost efficiency compared to newly produced components, and even improve the part design during the repair cycle. Another aspect of repair concerns sustainability, as products often contain rare materials. The emissions produced by the repair process are usually smaller than mining materials and casting new components. To optimize the repair process further, we propose the Smart Expert System (SES), which assists engineering experts with machine learning-based decision support throughout the repair process. We elaborate on its IT architecture and present machine learning models employed for representative MRO use cases. The SES is evaluated using actual industry data from a leading gas turbine company and demonstrably fulfills formulated requirements concerning the suitability of the overall decision support and the stability of the enclosing IT architecture.

0.42

Learning from Past Mistakes: AI-Assisted Analytics of Software Defects and Their Fixes for Future Defect Prevention - An Industrial Case Study

cs.SE ·2026-10-14 ·Vehid Goruslu
similarity 0.62
Abstract

Context. Despite advances in software testing and quality assurance, defects continue to recur in industrial software systems. Defect data are routinely collected in issue-tracking systems, but their use is largely confined to detection, reporting, and short-term resolution, leaving the knowledge embedded in historical defects and their fixes under-exploited for prevention and long-term improvement. Objective. This paper investigates how historical defect and defect-fix data can be analyzed, with AI assistance under human oversight, to identify the human-error origins of recurring defects and to feed that understanding back into prevention and QA improvement. Method. We developed an AI-assisted, human-in-the-loop analytics approach that classifies defects by failure type and by human-error origin across the SDLC, determines how each could have been prevented, and synthesizes engineer-focused capability-building insights. It was deployed and evaluated in a large software company through an industrial case study, using practitioner surveys and interviews, classification-reliability checks, and an effort-normalized analysis of defect density. Results. Practitioners and independent review judged the classifications accurate; the outputs were rated useful and actionable; and defect density decreased after the introduction of the feedback, a tentative association we interpret as consistent with, but not proof of, a positive effect, given the study's uncontrolled pre/post design. Conclusion. AI-assisted, human-governed analysis of defect-fix data can help shift organizations from defect detection toward prevention, continuous learning and improvement.

0.25

CTML: A Comprehensive Framework for Continuous Testing in Machine Learning Operations

cs.SE ·2026-10-07 ·Bestoun S. Ahmed
similarity 0.61
Abstract

The increasing use of machine learning (ML) systems in production has highlighted some important weaknesses in traditional software testing methods. Unlike regular software, ML systems can fail in specific ways due to their reliance on data, including issues like data drift, concept drift, and performance decline. To address these issues, this paper introduces the Continuous Testing for Machine Learning (CTML) framework. This framework is designed specifically for MLOps environments and offers a thorough, automated solution. It establishes a standardized way to evaluate ML systems across four main areas: data quality, drift detection, model quality, and pipeline validation. The CTML framework consists of a modular setup that includes a Test Registry, Test Runner, Result Store, and Integration Layer. This design ensures it integrates smoothly with existing Continuous Integration/Continuous Delivery (CI/CD) pipelines and MLOps platforms. We put this framework to the test in a predictive maintenance case study, which showed how effective it is at identifying issues like data drift, performance degradation, and bottlenecks in the pipeline problems that traditional testing methods might overlook. The findings indicate that CTML successfully uncovers critical failures in the system across all testing dimensions, providing useful insights to help keep ML systems running smoothly in production. Overall, this work presents a new, integrated framework that connects academic research with industry best practices, pushing forward the standards in MLOps quality assurance.

0.24

Compound AI System Reliability: A Failure Taxonomy and Resilience Pattern Catalog from 150 Production Incidents

cs.SE cs.AIcs.DCcs.LG ·2026-10-01 ·Rudrendu Kumar Paul, Sourav Nandy
publishedcode/data similarity 0.60
Abstract · Accepted at the AIWILD Workshop, ICML 2026. Camera-ready version

Deploying compound AI systems reliably and safely requires understanding failure modes that emerge at component boundaries, not within individual models. Cascading errors propagate across component boundaries, silent quality degradation evades standard monitoring, and coordination failures yield incorrect collective behavior from individually correct parts. We analyze 150 production incident reports from open-source compound AI projects and anonymized enterprise deployments to construct a taxonomy of 23 failure modes organized into five categories: retrieval failures, generation failures, tool failures, orchestration failures, and integration failures. For each category, we propose resilience patterns with measured effectiveness from controlled fault injection experiments. Circuit breakers reduce cascade propagation by 89%, output quality gates catch 73% of silent degradation before user impact, and component isolation reduces blast radius by 64%. Systems implementing three or more resilience patterns from our catalog reduce mean-time-to-recovery (MTTR) by 71% compared to unstructured monitoring baselines. We release the incident taxonomy and pattern catalog as a practitioner resource.

0.46

Representative Dataset Generation Framework for AI-based Failure Analysis during real-time Validation of Automotive Software Systems

cs.SE ·2026-08-10 ·Mohammad Abboush, Christoph Knieke, Andreas Rausch
published similarity 0.60
Abstract

Recently, thanks to its ability to extract knowledge from historical datasets, the data-driven approach has been widely used in various phases of the system development life cycle. During real-time system validation, remarkable achievements have been accomplished in developing an intelligent failure analysis based on historical data. However, despite its superiority over other conventional approaches, e.g., model-based and signal-based, the availability of representative datasets persists as a major challenge. Thus, for different engineering applications, new solutions to generate representative faulty data in different forms should be explored. Therefore, in this study, a novel approach based on Hardware-in-the-Loop (HIL) simulation and real-time Fault Injection (FI) method is proposed to generate and collect data samples under single and simultaneous faults for Machine Learning (ML) applications during system validation phases. The developed framework can generate not only sequential data, but also textual data including fault logs. The results show the applicability of the proposed framework in simulating and capturing the system behaviour under faults within the system components.

0.22

Discovery of fully efficient fault indicators along a data-based diagnosis process

cs.AI cs.LG ·2026-09-23 ·Igor Bezmaternykh, Louise Travé-Massuyès, Elodie Chanthery
publishedbenchmark similarity 0.58
Abstract · Submission accepted to IFAC WC 2026 (waiting for publication)

The integration of model-based and data-driven paradigms provides a powerful framework for fault diagnosis by combining the interpretability of analytical redundancy relations, i.e., input-output relations that are used as diagnosis indicators in model-based diagnosis, with the adaptability of learning techniques. DT4X is a recent diagnosis algorithm that uses symbolic regression to generate multivariate relations leveraging some properties of analytical redundancy relations and uses them as split functions in a decision tree. However, its symbolic regression procedure optimizes only the separation between two selected classes at each node, often fragmenting the remaining classes and degrading both interpretability and diagnosis performance. This paper introduces DT4X+, an enhanced version of DT4X that modifies the construction of training sets and the symbolic-regression loss so that expressions separate the target classes while preserving the coherence of non-target classes. The resulting relations become fully consistent with ARR properties and lead to more informative splits, improved robustness, and better performance on dynamic-system datasets. Experiments conducted on several benchmark systems demonstrate the benefits of this enhanced formulation.

0.45

Enterprise Process Control Tower for IT-Led Improvement

cs.CE ·2026-09-19 ·Shunmukha Sagar Puppala
similarity 0.58
Abstract · 10 pages

Enterprise integration traffic in large Enterprise Resource Planning (ERP) systems move through Electronic Data Interchange (EDI) and the Intermediate Document (IDoc), and the operational tools that surround that traffic remain fragmented. Status records are written for a technical audience, exception handling is performed one document at a time, and process health is read manually from long lists. This work proposes an enterprise process control tower that lets an Information Technology (IT) team observe, diagnose, and improve business workflows from a single operational layer. The design links process health observation, semantic translation of technical status text, machine-learning diagnosis of exceptions with model explainability, grouped recovery actions, and key performance indicators inside one governance framework. The proposed framework is evaluated on a synthetic IDoc event dataset of 84,000 records spanning twelve months, of which 15,120 error records are used to train a five-class remediation-routing model that maps each exception to a standard recovery action. A rule-based router that reproduces current production heuristics serves as the primary baseline. The proposed gradient-boosted diagnosis engine reaches a macro-averaged F1 score of 0.883 on the held-out test partition, against 0.742 for the rule-based router, while Shapely-based attributions expose the drivers of each routing decision to operators. Results are derived from synthetic data and are presented as an internal-consistency demonstration rather than real-world validation. The contribution is an integrated tool design that places diagnosis and governance on the same plane as observation.

0.28

Reliability Engineering for AI Systems: Challenges, Methods, and Directions

cs.AI stat.AP ·2026-09-28 ·Rong Pan, Yili Hong, Min Xie
benchmark similarity 0.57
Abstract

AI reliability concerns whether an AI system performs its intended function dependably over a stated period and under stated operating conditions, with clear evidence. As these systems become more autonomous, this function should include more than only a correct output. Retrieval, memory, tool use, permissions, human oversight, and interactions among systems must operate consistently and safely, and, for a generative AI, so must the reasoning process that produces the output. Average benchmark accuracy only measures system capability; it does not quantify this broader reliability claim. This paper adapts established reliability engineering methods, from failure definitions and operational envelopes to FMEA, accelerated testing, field monitoring, and reliability growth, to AI systems. A four-level diagnostic framework classifies failures as component, operational-loop, agentic-conduct, or network and governance failures. Test, evaluation, verification, and validation (TEVV), sequential monitoring, and FRACAS create and refresh evidence. The SMART framework provides statistical guidance for measurement, analysis, assessment, and test planning, and the NIST's AI risk management framework provides organizational guidance for governance, evaluation, monitoring, and mitigation. Three cases are discussed to illustrate these programs. It is shown that while the established reliability engineering principles can still provide a usable foundation, new measurements and safety guardrails are urgently needed as these systems are self-evolving at a fast pace.

0.14