CS Paper Radar

Reinforcement learning on the EPRI AI-ccelerating Unit Commitment benchmark: diagnosing over-commitment and guiding MILP with learned policies

Maharani Devira Pramita, Aidan O'Sullivan

cs.LG stat.ML ·submitted 2026-10-18 21:31 UTC · v1
benchmark

Abstract

Reinforcement learning (RL) is increasingly proposed as a fast solver for unit commitment (UC), but standalone agents remain far more expensive than mixed-integer linear programming (MILP). In the first study of RL on the 51-generator, 72-hour benchmark of the Electric Power Research Institute (EPRI) AI-ccelerating Unit Commitment competition, we trace this gap to the agents' own exploration, close most of it, and show that the resulting policy can accelerate an exact solver. In every configuration tested, Proximal Policy Optimisation (PPO) and an action-branching Deep Q-Network (DQN) trained from scratch committed about twice the units of a thermal-only MILP optimum, at 39-64% higher cost, whereas a merit-order rule came within 1.6-7% (10-24% when unserved energy is priced). When each unit's decision is sampled independently, spare units insure against random de-commitments. Consistent with this, injecting exploration noise during fine-tuning enlarged a lean policy's fleet, a reward without shaping terms did not remove the over-commitment, and under their training noise the agents outperformed the rule subjected to comparable noise. Imitating the MILP solutions and fine-tuning conservatively brought PPO within 2.6% of the optimum, and these policies had the lowest penalised cost of any evaluated (about 5%). Fixing the fine-tuned policy's confident decisions made the MILP 3.8-8.5 times faster within 0.10-0.25% of the best solution, with a smaller worst case than an imitation-only policy fixing as many variables (1.1% against 8.9%). Because dispatch method and benchmark scope alone shift the reported optimality ratio by up to 3.6% and 9%, we propose a reporting protocol.

Closest papers

Nearest neighbours by meaning across the whole index.

Learning to Fix: Optimisation-Aware Machine Learning for Accelerated Unit Commitment

eess.SY ·2026-09-30 ·Benjamin Fritz, Andreas Makrides, Maryam Fetanat, Pierre Pinson
similarity 0.72
Abstract

Unit Commitment is a computationally demanding mixed-integer linear optimisation problem requiring many binary commitment decisions across a scheduling horizon. To reduce this computational burden, machine learning approaches can predict a subset of these decisions, thereby shrinking the search space explored by the optimisation solver. Existing confidence-based approaches determine which variables to fix based on user-defined probability thresholds, which are agnostic to the downstream optimisation problem. We address this limitation by introducing an optimisation-aware framework that yields generator-specific confidence thresholds based on the impact of fixing errors on Unit Commitment solution quality. This approach won first place in the 2025 EPRI AI-ccelerating Unit Commitment competition. The proposed framework achieves a mean optimality gap below 0.5% while also delivering an average speed-up of more than 20x. It therefore provides a general mechanism for translating probabilistic predictions into controlled reductions of mixed-integer search spaces, directly linking learning decisions to their downstream computational and economic consequences.

0.20

Feasibility-Aware Learning for Network-Constrained Unit Commitment

eess.SY ·2026-11-08 ·Hien Thanh Doan, Yeji Lim, Hongseok Kim
similarity 0.69
Abstract

Learning-based approaches to network-constrained unit commitment (NCUC) usually still rely on an optimization solver to repair or complete their predictions, which limits the speedup they can deliver. This paper proposes FALCON-UC, a feasibility-aware graph learning framework that predicts commitment and dispatch jointly and produces final schedules without solving a mixed-integer program at inference time. It combines an LSTM--GraphSAGE predictor over a fully connected generator graph, a differentiable correction that embeds commitment-dependent dispatch bounds, power balance, and PTDF-based branch flows in training, and deterministic commitment, dispatch, and ramp-coupled reserve repairs at inference. On the 2383-bus system, FALCON-UC achieves $99.09\%$ commitment accuracy and a $0.4350\%$ mean absolute cost gap with all evaluated constraints within tolerance at $0.0562$ s per instance, a $2.27\times10^{3}$ speedup over the solver. MTLR R-SCUC-FL is slightly more accurate but still needs an optimization solver to complete each schedule, which takes $262.2$ s per instance, $4.67\times10^{3}$ times the FALCON-UC runtime. A label-only baseline without repair violates the balance and branch limits on every 2383-bus test instance, and FALCON-UC also meets all evaluated constraint tolerances on a 1354-bus system.

0.26

Learning to Solve Two-Stage Stochastic Unit Commitment Problems with Quality Guarantees

eess.SY math.OC ·2026-09-16 ·Andrea Fusco, Andrea Lodi, Lavanya Marla
benchmark similarity 0.67
Abstract · 11 pages, 2 figures, 3 tables, 1 algorithm. Submitted to IEEE Transactions on Power Systems

Two-stage stochastic Mixed-Integer Linear Programs are a canonical modeling tool to optimize power system operations under uncertainty, yet their extensive-form counterparts scale linearly with the number of scenarios and quickly become computationally prohibitive under day-ahead time constraints. We propose an Input Convex Neural Network architecture to learn a convex surrogate of the second-stage value function, enabling fast first-stage optimization while preserving convexity by construction. We couple the surrogate with a Neural-Benders correction loop that refines the first-stage solution a posteriori, recovering the exact optimum whether the network overestimates or underestimates the recourse cost, and certifying solution quality independently of surrogate accuracy. We evaluate the method on IEEE Stochastic Unit Commitment benchmarks (case9-case118) under both continuous and integer recourse. The proposed approach achieves solutions with zero optimality gap on the $K$-scenario instance across all test cases and scenario dimensions $K \in \{10, 50, 100\}$, with speedups up to $214\times$ over the Extensive Form in the integer setting. Solve times are stable across scenario realizations and compatible with day-ahead time windows on the tested benchmark instances.

0.21

Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Games

cs.AI ·2026-10-08 ·Nick Leenders, Roy Lindelauf, Joost van Oijen, Boris Cule
similarity 0.65
Abstract

Deep reinforcement learning agents reach strong performance in real-time strategy games but can be brittle against opponents outside their training distribution. Separating strategic command selection from learned unit control allows different strategies to be selected for different opponents while reusing the same execution policy. This requires an executor that can follow different commands and measurable criteria for assessing whether it does so. We introduce a constrained command-conditioned Proximal Policy Optimization (PPO) policy, the executor, for MicroRTS, a real-time strategy environment. Discrete commands specify strategic objectives and behavioral requirements for economy, army composition, military posture, and worker policy over multiple environment steps; the executor determines the unit-level actions used to fulfill them. A Thompson-sampling bandit acts as the strategist, selecting command tuples from an estimate of the opponent's strategy built from in-game observations rather than opponent identity. In a controlled comparison with a flat PPO baseline trained with the same architecture, budget, curriculum and self-play league, the strategist-executor system wins significantly more often against three of the four strongest opponents on a training map, including the two strongest held-out ones (0.55 to 0.97 and 0.01 to 0.34), with no significant difference against the others.

0.25

Strategyproof Deep Reinforcement Learning for Online Resource Allocation

cs.MA cs.AIcs.GT ·2026-10-27 ·Fariba Dehghan, Sebastian Stein, Vahid Yazdanpanah, Stephanie Gauthier
theory similarity 0.63
Abstract · 15 pages, 2 figures

Online resource allocation faces two coupled challenges: (1) allocating scarce resources efficiently under uncertain future demand, and (2) ensuring that agents report task valuations and requirements truthfully (strategyproofness). Reinforcement Learning (RL) adapts allocation decisions through experience but does not necessarily ensure strategyproofness. Existing strategyproof RL uses a linear allocation rule, which limits the resource interactions it can represent. Neural mechanism design methods offer greater expressiveness, but provide only approximate incentive guarantees. Against this background, we introduce the first strategyproof deep RL mechanism for online resource allocation. It integrates three novel components: (i) a monotonic neural network that learns nonlinear interactions by construction (higher reported value cannot reduce acceptance, while a longer reported duration or larger resource requirement cannot increase it); (ii) a generalised critical payment: the lowest accepted value, with all other inputs fixed; and (iii) training by either RL or imitation of an existing rule. Together, monotonicity and the critical payment ensure strategyproofness and individual rationality, while the neural rule provides flexibility and imitation improves training efficiency. Empirically, on a four-resource stream with two partner pairs that a linear difference cannot represent, the neural rule achieves 97.8% of an offline welfare upper bound, compared with 93.8% for a linear learner and 89.0% for first-come, first-served, closing 65% and 80% of the remaining optimality gap, respectively. Moreover, imitation reaches 97.5% in only 261 seconds, about 89 times faster than RL.

0.17

A Universal Policy Wrapper With Guarantees

cs.LG eess.SYmath.DS ·2026-11-02 ·Georgiy Malaniya, Anton Bolychev, Pavel Osinenko
published similarity 0.63
Abstract · Accepted for IEEE Access (2026)

Many control tasks require driving the system into a prescribed goal set and subsequently keeping it there. The considered setting assumes that a working controller is available that provably reaches and maintains the goal, albeit with suboptimal performance. This controller is referred to as the fallback policy. Conversely, although state-of-the-art reinforcement-learning (RL) algorithms can discover policies with impressive empirical performance, the neural networks at their core act as opaque black boxes learned through data-driven exploration and, as such, provide no theoretical mechanism to guarantee that the system will ever reach---and stay within---the prescribed goal set. This article presents CALF-Wrapper, a method that combines the fallback policy with a pretrained RL base policy and uses the critic estimate associated with the base policy to select between them during deployment. The theoretical analysis establishes that CALF-Wrapper inherits the goal-reaching property of the fallback policy under the stated assumptions and provides probability bounds for the settling time and the maximum distance from the goal set over a prescribed time interval. The experimental study evaluates CALF-Wrapper in four simulated control environments of different complexity. Comparisons include the base policy, fallback policy, and constrained reinforcement-learning methods formulated for the same objective. Sensitivity and ablation studies provide direct hyperparameter recommendations. The experiments show how CALF-Wrapper can provide effective task performance while attaining the maximum possible observed goal-reaching rate. CALF-Wrapper requires no additional dynamics model, online learning, or constrained optimization during execution.

0.35

Envs-FORGE: Frontier-Optimized Reward-Grounded Environment Synthesis for Agent RL

cs.CL ·2026-08-14 ·Xiaojun Wu, Cehao Yang, Honghao Liu, Xueyuan Lin, Zhichao Shi +5 more
code/datahighly cited similarity 0.63
Abstract · 19 pages, 5 figures

Reinforcement learning (RL) for terminal agents needs executable training environments with reliable rewards and useful difficulty. Fixed recipes such as few-shot, Self-Instruct, and Evol-Instruct apply the same prompting policy to every seed, even when the current policy would benefit from a harder, easier, or simply different task. We present Envs-FORGE, a prompting policy that converts verifier rewards into per-seed environment-synthesis actions. Envs-FORGE estimates seed pass rates, scores six projection--direction actions around a target learning frontier, and solves a per-seed mixed-integer linear program (MILP) to choose the action that conditions generation. The selected action drives synchronized rewriting of the instruction, fixtures, oracle solution, tests, and Docker environment; only gold-verified bundles enter RL training. The indexed MILP form also supports optional soft skill coverage for portfolio planning. On Qwen 3.5 35B, Envs-FORGE improves Pass@1 over Base by 9.2 percentage points on tb-core (40.0% to 49.2%) and 6.4 points on tb-2.0 (23.0% to 29.4%), exceeding the strongest fixed-recipe baseline by 2.4 and 2.1 points. It reaches 77.1% on SWE-bench Verified versus 73.4% for Base, and improves tb-core by 6.8--9.2 points across the evaluated 4B--35B models. All synthesis methods export 100 verified environments and use 2.27M--2.88M synthesis tokens, placing the comparison at the same downstream training-set size and the same operational scale. The source code is available at https://github.com/DataArcTech/DataArc-SynData-Toolkit/.

0.68

Learning to Modulate, Not to Cycle: Soft Actor---Critic Recovers Inverter-Style Heat-Pump Control

cs.LG cs.AI ·2026-08-10 ·Faizan Ahmed, Aniket Dixit, James Brusey
published similarity 0.62
Abstract · 9 pages, 4 figures, Accepted at UKCI 2026, Will be published in Springer

On--off cycling is the main cause of compressor wear in residential heat pumps, yet reinforcement learning (RL) controllers for buildings typically optimise only energy cost and thermal comfort, ignoring how much the learned policy cycles. We add a levelised compressor-wear term to the control reward and study how the resulting behaviour depends on the RL algorithm. Training Soft Actor---Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest hydronic heat pump case, we find that SAC learns a continuous modulation policy that keeps the compressor permanently engaged---the operating principle of an inverter-driven heat pump---achieving zero start-ups per day, whereas PPO collapses to bang-bang control that cycles more than the baseline. On the BOPTEST emulator the SAC policy cuts thermal discomfort by up to 90.7% for an 11.5% cost increase, while eliminating all baseline cycling.

0.25