CS Paper Radar

Metamorphic testing of learned AC optimal power flow surrogates reveals representation dependence

Stephen Peng, Vineet Jagadeesan Nair

cs.LG cs.AIeess.SYmath.OC ·submitted 2026-11-10 04:33 UTC · v1
top venuecode/data

Abstract

Evaluating a learned AC optimal power flow (AC-OPF) surrogate on one grid serialization cannot show whether its accuracy depends on that encoding. We test this dependence with thirteen metamorphic relations, each specifying when a transformation is valid and how to compare its outputs. Our audit covers 17 model and pipeline configurations: two released trained checkpoints and 15 random-weight architecture or mechanism controls. Every configuration violates at least six relations, even at reporting tolerances ten times tighter or looser. For GridSFM-Open v1.1, source inspection and intermediate-tensor interventions point to an indexing bug and three noncanonical or numerically unstable preprocessing choices; a repaired checkpoint remains untested. In a separate case500 training experiment, augmentation lowers transformed-view error in three of four models and median discrepancy and constraint-violation P95 in all four. These benefits depend on the training protocol: some relations worsen, and every evaluated prediction remains AC infeasible. A GPU/CPU eigensolver mismatch accounts for most of the large raw-Laplacian-baseline gain. We recommend reporting representation consistency alongside accuracy, feasibility, and cost. These results establish neither deployment readiness nor universal invariance.

Comment
Accepted to the NeurIPS 2026 Workshops on (i) Trustworthy AI Evaluation, (ii) AI Foundations for Power Grids, (iii) Tackling Climate Change with Machine Learning, (iv) Sim2Science, and (v) Representations for the Physical Sciences

Closest papers

Nearest neighbours by meaning across the whole index.

Generalization Testing of Grid Foundation Models for Power Flow and Optimal Power Flow

eess.SY cs.LG ·2026-11-14 ·Ayushi Jolotia, Parikshit Pareek
similarity 0.70
Abstract

There is growing interest in foundation models (FMs) for power grids, particularly for power flow and optimal power flow. Their usefulness depends on generalization accuracy, which existing works test on their own benchmarks. We propose a three-axis framework to test their generalization limits: changed admittance matrices of training grids, load levels, and unseen grids. We test GENCO, LUMINA and GridSFM against the DC baseline, alongside their reported limitations, and discuss theoretical reasons for their failures. Experiments reveal that adding a line degrades all FMs more than removing one, and longer lines eventually make LUMINA and fine-tuned GENCO worse than DC. All three degrade beyond their training load range, where the loading-voltage curve steepens. On unseen grids, LUMINA degrades sharply even inside its training range, and GridSFM fails on the grid farthest from it. Thus, beyond training conditions, FMs behave more like meta-models requiring test-time fine-tuning data and compute.

0.24

Solver-Admitted Metamorphic Testing of Learned Network Optimization Surrogates

cs.AI cs.LG ·2026-11-06 ·Vineet Jagadeesan Nair, Stephen Peng, Arun Ravindran
similarity 0.68
Abstract

Equivalent encodings of an optimization problem can produce different learned predictions. Interpreting these differences requires accounting for numerical error and multiple optima. We construct metamorphic tests for optimization solver surrogates with explicit eligibility conditions, output alignment, and numerical thresholds. Reference-solver agreement and independent constraint checks determine which sampled pairs enter the audit. We study alternating-current optimal power flow (AC-OPF), the traveling salesman problem (TSP), the capacitated vehicle routing problem (CVRP), and maximum independent set (MIS). Across five grids, GridSFM-Open violates each of 13 relations on at least one grid, and 12 by more than twice the threshold. Randomly initialized copies of the grid models fail nearly the same relations, so the failures do not require training. Source patches and input canonicalization remove selected discrepancies, but no intervention establishes preserved accuracy and feasibility within the declared margin on every grid. In routing, Policy Optimization with Multiple Optima (POMO) has a single-pass rotation/reflection violation rate that rises from 12.7% to 89.5% between 20 and 100 cities. Graph-Based Diffusion Solvers for Combinatorial Optimization (DIFUSCO) violates the MIS disjoint-union relation on 35.6% of draws. Post hoc normalization within connected components removes these violations on the tested graphs. These findings support reporting metamorphic consistency alongside optimality, feasibility, and accuracy, and validating repair quality separately from consistency.

0.29

Benchmarking CPU, GPU, and ML Power-Flow Solvers from 14 to 70,000 Buses

cs.DC cs.SE ·2026-10-20 ·Benjamin Donnot, Sami Tazi, Eva Boguslawski, Nourredine Henka
benchmarkcode/data similarity 0.65
Abstract

Learned surrogates for power-flow (PF) computation are usually motivated by speed, compared against a single CPU-based baseline. We show this comparison is fragile: across 7 open-source CPU-based solvers evaluated on identical inputs, throughput varies by roughly 2.5-3 orders of magnitude, so the choice of baseline alone can flip the conclusion. We build a benchmark spanning 9 physics-based solvers: 7 CPU-based and 2 GPU-based, including a new batched GPU solver, gumsim2grid - and 3 published ML surrogates, evaluated under a shared, independently implemented KCL-mismatch check across 15 grids from 14 to 70,000 buses. On the PF task itself, a sufficiently parallelized CPU solver or gpusim2grid matches or exceeds the throughput of the ML surrogates we test while remaining orders of magnitude more accurate; we find no evidence that these GNN-style surrogates are yet competitive for PF specifically, though prior work reports genuine gains on related but more combinatorial tasks such as optimal power flow or power grid control. Beyond this finding, we hope the benchmark suite, code and reference evaluator released at https://github.com/Grid2op/gpusim2grid helps close a broader reproducibility gap in this space, where batched GPU-based PF solvers are rarely open-sourced or evaluated against a common set of baselines.

0.27

Scaling Laws for Physics-Aware ACOPF Surrogate Learning

cs.LG ·2026-09-14 ·Yijiang Li, Emon Dey, Stefano Fenu, Massimiliano Lupo Pasini, Teja Kuruganti +1 more
similarity 0.64
Abstract

Learning-based surrogates for AC optimal power flow (ACOPF) promise large speedups over classical solvers, but their operational value depends on physical feasibility as much as predictive accuracy. Physics-aware objectives such as the augmented Lagrangian (AL) improve constraint satisfaction at additional per-step cost, yet how this trade-off behaves with scale is uncharacterized. We sweep model and dataset sizes under MSE and AL training, and measure how violation changes with network size. Both objectives improve as power laws: MSE prediction loss depends more on model size than on data, while the AL composite of prediction loss and violation improves comparably along both axes. Violation grows markedly more slowly with network size under AL. On matched hardware and equal training samples, AL reduces violation about $19\times$ with $2.4\times$ the training time and negligible added memory. The training objective shapes not only where a surrogate lands but how its quality evolves with scale.

0.20

ML-OPF-Bench: Benchmarking Machine Learning for Optimal Power Flow

eess.SY cs.LG ·2026-10-03 ·Xinyi Liu, Xuan He, Danny H. K. Tsang, Yize Chen
benchmarkcode/data similarity 0.62
Abstract · 25 pages, 9 figures, in submission. Code available at https://github.com/XinyiLiu04/ML-OPF-Bench; Datasets available at https://huggingface.…

Machine Learning (ML) methods promise a fast solution process for Optimal Power Flow (OPF). While inconsistent test cases, implementations, and evaluation metrics across existing studies make it challenging to determine which algorithmic advances are most critical for real-world deployment. To this end, we propose ML-OPF-Bench, a unified benchmark for AC- and DC-OPF that evaluates representative ML algorithms under a consistent pipeline, stress-tests them across system sizes, distribution shifts, and resource budgets, and ranks them with a multi-objective framework. We find that prediction accuracy alone is not a reliable indicator of operational feasibility. Under heavily loaded, congested conditions, even the strongest in-distribution performers lose their advantage, while feasibility is maintained largely by post-processing that enforces the target constraints rather than by the underlying pure ML predictor. Data scaling shows that prediction accuracy and constraint violations follow different trajectories, whereas compute scaling shows that returns diminish and that larger models do not consistently perform better. These results expose critical trade-offs among ML methods' speed, accuracy, and feasibility, and offer practical guidance for future ML-OPF design. We open-source the benchmark as an extensible Python package for integrating new learning-based OPF algorithms and evaluating them under the same standard as the existing baselines.

0.33

KernelMutate: Mutation-Guided Validation of Generated GPU Kernels

cs.SE ·2026-11-11 ·Ziyuan Li, Zihe Song, Shuxiao Xie, Mingfang Ji, Xiaofang Qi +2 more
hot direction similarity 0.62
Abstract · 29 pages, including an appendix with supplementary evidence

Generated GPU implementations are commonly accepted after a few differential tests, although replacement obligations may remain unchecked. We present KernelMutate, an empirical study of validation reliability using controlled faults, released implementations, and replacement tests. A five-draw anchor accepts 548 of 838 CUDA candidates. The full suite finds native FP32 witnesses for 136 accepted candidates and dtype-only findings for another 362, mostly low-precision execution limitations without established publisher guarantees. Source-guided testing identifies the full suite's 692 candidates at 26.1% lower charged cost, but yields only dtype evidence for 16 accepted candidates with native witnesses in the full suite. Independently observed context and boundary findings cover the same candidate set. Task-held-out template matching reveals differences in ranking and observation coverage: global top-eight recommendations match more probes than class-conditioned lists (51 versus 48 of 75), while conditional analyses restrict the apparent advantage at smaller limits. These template-matching results are offline measurements, not deployed-suite gains. Independent controls and replacement cases expose numerical-policy, binding, and custom-execution boundaries. Validation therefore needs to report witness domains and observation coverage alongside candidate yield and cost.

0.41

Benchmarking a Grid Foundation Model for AC Optimal Power Flow: Feasibility and Warm-Start Results on Synthetic Texas and South Carolina Grids

eess.SY ·2026-11-17 ·Tam Kemabonta, Brighton Ombuki, Vladimir Abdelnour
similarity 0.62
Abstract · 5 pages, 3 figures, 2 tables. Submitted to the 2027 IEEE PES General Meeting

Learning-based surrogates for alternating-current optimal power flow (AC-OPF) are often judged by the gap between their objective value and that of a nonlinear solver, which does not show that a prediction satisfies power balance and network limits. This paper tests whether predictions from the pretrained foundation model GridSFM-Open 1.1.0, recently released by Microsoft, are feasible AC-OPF operating points and whether they are useful initial values for a nonlinear solver. The model was evaluated zero-shot and after fine-tuning on 10, 100, and 1,000 labeled scenarios (FT10, FT100, and FT1000) on 500 scenarios for each of two synthetic grids, ACTIVSg2000 (Texas) and ACTIVSg500 (South Carolina), with Ipopt, MATPOWER MIPS, and pandapower PIPS as solver references. The lowest Texas median cost gap, 0.44\% for FT100, coincided with thermal violations in 24.1\% of cases and a median maximum bus power mismatch of 89 MW and 197 Mvar. Texas zero-shot predictions had a median maximum mismatch of 648 MW and 1,641 Mvar, and every South Carolina case violated a thermal limit for every model variant. Seeding Ipopt with FT1000 predictions gave geometric-mean solve-time speedups of 1.55x on South Carolina and 1.12x on Texas (p = 0.06, not significant at the 5\% level), while zero-shot seeds slowed South Carolina solves (0.26x). A small objective gap therefore did not indicate feasibility: although median zero-shot inference took only 1.32 s on Texas and 0.157 s on South Carolina on a shared host, every prediction still required a nonlinear solver, and fine-tuned predictions gave modest, grid-dependent warm-start speedups while zero-shot predictions slowed convergence on one grid. GridSFM may therefore be useful for initializing a nonlinear solver, but obtaining feasible AC-OPF solutions still required the solver.

0.32

Faults That Fortify: CNN Adversarial Robustness via GPU Undervolting

cs.LG cs.ARcs.CR ·2026-08-20 ·Behnam Omidi, Ahmad Tahmasivand, Husam Alsyouri, Saba Al-Sayouri, Chongzhou Fang +2 more
similarity 0.61
Abstract · 6 pages, 4 figures, 1 table. Submitted to IEEE HOST 2027

Convolutional Neural Networks (CNNs) face a dual challenge: vulnerability to adversarial attacks and prohibitive training cost. Adversarial training is effective but expensive, a burden that grows as learning shifts to the energy-constrained edge. This paper addresses both through GPU undervolting during training. Reducing supply voltage introduces stochastic perturbations that act as implicit regularization, improving robustness while lowering power. We characterize undervolting-induced faults at the bit level, then train LeNet, VGG-6, and MobileNetV3 on MNIST and CIFAR-10 under two training regimes, standard and adversarial, each at nominal and undervolted voltage, and evaluate all models against adversarial attacks. In both regimes, the undervolted model consistently achieves higher adversarial accuracy than its nominal-voltage counterpart, showing that hardware-induced faults strengthen even adversarial training. Because dynamic power scales quadratically with supply voltage, these robustness gains arrive with substantial energy savings. GPU undervolting is therefore a readily deployable hardware-level defense requiring no algorithmic change, and opens a promising direction in which robustness and energy efficiency move together.

0.38