CS Paper Radar

EvEMTBench: An Open Benchmark for Machine Learning in Power System Protection

Julian Oelhaf, Georg Kordowich, Christian Bergler, Andreas Maier, Johann Jäger, Siming Bayer

eess.SY cs.LG ·submitted 2026-09-23 14:05 UTC · v1
benchmarkcode/data

Abstract

Studies of machine-learning-based power system protection are difficult to compare because task definitions, measurement access, data partitions, metrics, and generalization conditions often differ. EvEMTBench addresses this gap with an open, executable, and versioned benchmark that fixes these evaluation choices while leaving model design open. Across four grids spanning 20-345 kV, it defines 12 protection and event-analysis functions instantiated as 24 scored tasks and supports structured evaluation across observability conditions, predefined distribution shifts, and zero-shot and fine-tuned cross-grid transfer. Committed partitions, leakage controls, and reproducible reporting provide a common basis for comparing future methods. A reference evaluation spanning trivial, conventional, feature-based, and deep-learning baselines shows that wider observability is not uniformly beneficial, shifted conditions can reveal failures not apparent in-distribution, and cross-grid transfer is substantially stronger for fault detection than for fault localization. Protection-relevant diagnostics identify failure modes not apparent from primary metrics alone. EvEMTBench therefore makes generalization in machine-learning-based protection an explicit and reproducible evaluation problem.

Comment
15 pages, 3 figures. Supplementary information included. Code: https://github.com/EvEMTBench/evemtbench-benchmark

Closest papers

Nearest neighbours by meaning across the whole index.

Protection-Oriented Assessment of Machine Learning-Based Fault Detection in Power Systems

cs.LG eess.SP ·2026-11-13 ·Julian Oelhaf, Georg Kordowich, Changhun Kim, Christian Bergler, Andreas Maier +2 more
publishedhot direction similarity 0.82
Abstract · 9 pages, 3 figures, 5 tables. Accepted for oral presentation at the 21st International Conference on Developments in Power System Protection…

Machine learning (ML)-based fault detectors are increasingly proposed for power system protection, yet aggregate metrics can obscure the consequences of false and missed trips. We present a protection-oriented assessment of three ML detectors and a sequence-current reference on PROTECT-90, comprising 9,022 simulated fault episodes on a 90 kV double-circuit line. The assessment combines five window conditions, causal window-end labelling, episode-grouped cross-validation, and operating-point selection under a fixed security budget. It reports security, dependability, detection timing, decision margins, and episode-level nuisance assertions. Gradient-boosted trees, a convolutional neural network, and logistic regression achieve AUC-ROC values above 0.999, yet at a security budget of $10^{-3}$ their developed-fault missed-trip rates range from 0.007% to 1.99%. Post-clearing false-trip rates of approximately 0.12-0.16% per window coexist with nuisance assertions in 11.3-19.1% of episodes. Among the analysed physical descriptors, missed trips are most consistently associated with higher fault resistance within the sampled range. An analytical current proxy suggests a fault-current gradient only partly reproduced by waveform-derived current (rank correlation 0.62). No single topology state or faulted phase consistently dominates within the sampled domain. Because PROTECT-90 contains fault episodes only, security is assessed on pre-fault and post-clearing windows rather than independent non-fault disturbances.

0.27

A Standardized Framework for Machine Learning in Power System Protection

cs.LG cs.AIeess.SP ·2026-08-20 ·Julian Oelhaf, Georg Kordowich, Paula Andrea Pérez-Toro, Christian Bergler, Johann Jäger +2 more
publishedbenchmarkcode/data similarity 0.77
Abstract · 32 pages, 4 figures, 26 tables. Code: https://github.com/julianoelhaf/protection-eval-framework. Dataset: PROTECT-90, doi:10.5281/zenodo.211…

Studies of machine-learning-based power-system protection increasingly report near-perfect scores, yet the meaning of those scores depends strongly on the evaluation setting. Protection task, physical scope, measurements, timing, targets, preprocessing, and validation often vary jointly and remain incompletely specified. This paper proposes a standardization-oriented framework that treats evaluation design as part of the scientific contribution. It defines seven required study dimensions: protection objective, physical scope, observability, timing and decision windows, targets and sample validity, validation protocol, and evaluation outputs. The framework is instantiated in a bounded case study on the public PROTECT-90 electromagnetic-transient benchmark, comprising 9022 simulated episodes from a 90 kV double-line topology, for onset-conditioned fault classification and localization. Under centralized sensing, simulation-metadata-aligned 20 ms windows, and episode-grouped validation, a multi-layer perceptron (MLP) achieved a five-fold mean macro-averaged F1 score of 0.991 +/- 0.001 for classification and a localization mean absolute error of 10.20 +/- 0.25% of line length (mean +/- std across episode-grouped folds). Extending the decision horizon to 50 ms preserved this task-dependent performance asymmetry, while reduced observability approximately doubled the MLP localization error but had little effect on classification. A synchronized two-ended conventional locator outperformed the learning locators under its richer clean information set, and measurement degradation showed that clean predictive performance did not determine robustness. The framework turns evaluation assumptions into explicit, reproducible evidence and provides a basis for more comparable, auditable evaluation and future certification-oriented assessment of machine-learning protection functions.

0.77

A simulation based dataset of faults and events for machine learning in power systems

eess.SY ·2026-08-20 ·Georg Kordowich, Jonathan Loebel, Julian Oelhaf, Andreas Maier, Siming Bayer +2 more
similarity 0.74
Abstract

The integration of inverter-based renewable energy sources into electric grids challenges conventional power system protection. Machine learning-based solutions can address these challenges by utilizing available data in modern smart grids. However, the lack of open datasets prevents reproducibility and fair comparisons between different approaches and their results, which hinders further progress. Therefore, this paper presents EvEMTBench, a synthetic dataset of faults and events generated using electromagnetic transient simulations. The physical plausibility of the power system simulation is ensured by validating the simulation parameters against established literature and providing comprehensive documentation of the simulation procedure. The dataset is designed for training, fine-tuning, and benchmarking machine learning models by providing synchronized point-on-wave voltage and current measurements at 9600 Hz across a diverse set of topologies and voltage levels. The inclusion of a wide range of fault and operating events allows the utilization of EvEMTBench for different tasks like incipient fault detection, fault localization, or event detection.

0.50

A Defense-Aware Explainable AI Framework for Feature-Level Vulnerability Analysis in Optimal Power Flow

cs.CR ·2026-10-28 ·S. M. Monzurul Hoque Chowdhury, Qinghua Li, Jianhui Wang
similarity 0.72
Abstract

Optimal power flow (OPF) is a foundational analysis in power system operation, and inaccuracies in OPF decision-making can result in significant operational and security consequences. With the increasing adoption of machine-learning-based OPF surrogate models for real-time and large-scale applications, power system operation has become more susceptible to cyber threats, including data manipulation and false data injection attacks. Explainable artificial intelligence (XAI), particularly Shapley value-based approaches, has been employed to quantify the influence of input features on OPF outputs. Features exhibiting high influence are commonly interpreted as vulnerable. However, this interpretation neglects the role of attack detection mechanisms in the system, which typically monitor input-to-input relationships to detect anomalous behavior. Consequently, highly influential features may be readily detectable and may not represent the most vulnerable attack surfaces. To address this limitation, we propose a defense-aware explainable AI framework that explicitly incorporates attack detectability alongside operational impact. The framework integrates an OPF surrogate model representing input-to-output dependencies with a vanilla autoencoder representing input-to-input dependencies. Shapley-based explainability is applied to both models to construct a feature-level vulnerability metric. Empirical evaluation on IEEE test systems demonstrates that the proposed framework more accurately identifies security-relevant vulnerabilities than conventional OPF-surrogate-model-only explainability approaches.

0.11

A Systematic Evaluation of Machine Learning Methods for Fault Detection and Line Identification in Electrical Power Grids

cs.LG eess.SP ·2026-09-15 ·Julian Oelhaf, Georg Kordowich, Paula Andrea Pérez-Toro, Tomás Arias-Vergara, Andreas Maier +2 more
publishedhighly cited similarity 0.71
Abstract · Accepted at ICASSP 2025. 5 pages, 4 figures. Published version: DOI 10.1109/ICASSP49660.2025.10890544

The integration of renewable energy sources into the electrical grid introduces complex challenges in fault detection and coordination of grid recovery mechanisms. Traditional relay protection systems, which operate based on static rules and predefined thresholds, are inadequate for addressing these challenges, particularly in detecting and isolating faults such as short circuits. Consequently, the conventional methodologies applied to electrical network protection frequently fail to achieve optimal performance in fault detection, especially in terms of adherence to safety standards and the selective limitation of damage. Recent research indicates that machine learning (ML)-based approaches can effectively tackle these issues; however, variations in grid configurations and analysis windows have impeded consistent comparative assessments. In this study, we assess the efficacy of various ML models in detecting electrical faults and pinpointing defective transmission lines within a 10 ms measurement interval - a critical time-frame for real-time operational viability, for the first time. The most effective model attained an F1 score of 0.991 +/- 0.018 and demonstrated a processing time of 0.342ms +/- 0.509ms.

0.52

Characterizing Cross-Relay Generalization Failure in Machine Learning-Based Power System Protection

cs.LG ·2026-10-20 ·Nithin Pradeep Nadayil Kizhakkethil, Julian Oelhaf, Christian Bergler, Andreas Maier, Siming Bayer
published similarity 0.70
Abstract · 6 pages, 3 figures. Accepted for presentation at the IEEE Power & Energy Student Summit (PESS 2026), Karlsruhe, Germany

Machine learning-based fault detectors are often evaluated at the relay locations used for training, although local voltage and current measurements vary across power networks. This work investigates cross-relay generalization for binary detection of faults anywhere in the network using local measurements, rather than line-selective tripping. Using 50 ms windows from a simulated 110 kV transmission network, we evaluate a multilayer perceptron across all 64 source target combinations of eight relays and five random seeds. A shared simulation-level split and source-only threshold selection provide a controlled held-out evaluation without target-specific calibration. Despite nearly unchanged mean F1-scores (0.730 within-relay versus 0.734 cross-relay), the mean false-positive rate (FPR) increases from 0.088 to 0.341, while the mean false-negative rate decreases from 0.399 to 0.318. The change is strongly directional: models trained at Bus 2 relays yield a mean FPR of 0.670 at Bus 1 and Bus 3 relays, compared with 0.096 for cross-relay transfers within the same bus group. Linear probing indicates that learned representations retain coarse bus-group information, while exploratory class-conditional latent distribution analysis shows no consistent monotonic association with transfer errors. These findings demonstrate that aggregate metrics can conceal location-structured changes in false-alarm behavior, highlighting the importance of evaluating error types separately when assessing cross-relay generalization.

0.23

Data-Optimized Contingency Screening: A Machine Learning Approach to Power System Security

cs.AI cs.LG ·2026-09-03 ·Joshua Salako, Folajimi Osikomaiya, Olakorede Olamiju
published similarity 0.68
Abstract · Accepted for presentation at the 3rd EIE International Conference on Computing, Networking, Robotics and Telecommunication (EIECon), 2025. T…

Ensuring the security of the power system is essential for stability and reliability, especially in the event of disruption. Effective classification of contingency in power systems enables proactive decision-making and mitigates large-scale breakdowns and failures. This study explores the use of machine learning algorithms to classify security levels of contingencies in power systems into safe, moderate or severe classes. For this approach, Newton-Raphson load flow method extracts system data from contingency scenarios, using Overall Performance Index (OPI) as safety measure. For data pre-processing, Synthetic Minority Over-Sampling Technique (SMOTE) and Principal Component Analysis (PCA) is used to address class imbalance and reduce dimensionality, respectively. K-Nearest Neighbours (KNN), Random Forest (RF) and Support Vector Machines (SVM) is trained and evaluated on datasets generated through N-k contingency scenarios for k equal 1, 2, and 3 on IEEE-14 and IEEE-30 bus systems using four hybrid pre-processing configurations: normalized, SMOTE-balanced, PCA-transformed, and a combined SMOTE PCA-transformed. Performance is assessed by precision, recall and F1 score, with priority given to the severe contingency classes. The RF achieved the highest F1 scores of 0.97 in IEEE-30 and 0.86 in IEEE-14, SVM benefits significantly from PCA and improves the accuracy of the classification, while KNN is best suited for SMOTE and PCA conversion. The findings show that PCA contributes more than SMOTE to the overall performance of the model. However, SMOTE improves recall but can introduce false positives and is therefore a compromise of accuracy. This study highlights machine learning as a scalable and powerful alternative to traditional contingency analysis, which improves the assessment of security in real time.

0.25

Severity-Aware Minimum Security Sets for Accelerated Preventive SCOPF Using Machine Learning

eess.SY ·2026-10-14 ·Mohammad Iman Alizadeh, Saeed Zeynali, Salah Ghamizi, Sylvain Kubler
similarity 0.67
Abstract

This paper proposes a severity-aware machine learning framework for reducing the computational burden of preventive AC security-constrained optimal power flow (PSCOPF). The central idea is to identify a compact minimum security set (MSS) containing only the non-dominated contingencies and associated post-contingency constraints that materially influence the preventive operating point while satisfying a prescribed severity tolerance. An offline procedure combines AC security assessment, severity-based ranking, non-dominated screening, and iterative reduced PSCOPF. After each reduced PSCOPF solution, exhaustive security assessment over all credible contingencies identifies additional contingency--constraint pairs whose residual severity exceeds the specified threshold. Severity is characterized using the normalized number, accumulated magnitude, maximum magnitude, and average magnitude of post-contingency violations. The resulting operating-point-dependent MSSs are used as labels for a supervised multi-label learning model that predicts the security elements required to construct the reduced PSCOPF directly from the pre-contingency state. The proposed framework therefore focuses the optimization on the contingencies and constraints that affect preventive control decisions, reducing problem size and computation time while explicitly controlling residual post-contingency severity. Results on the IEEE 60-bus and 118-bus systems, the latter having an approximately seven-times-larger PSCOPF formulation, demonstrate substantial computational reductions with limited impact on the preventive operating solution.

0.22