CS Paper Radar

Every minute counts: Wind farm power control during imbalance settlement using distributional soft actor-critic

Max Bruninx, Seyed Soroush Karimi Madahi, Timothy Verstraeten, Jan Decuyper, Chris Develder, Ann Nowé, Jan Helsen

eess.SY ·submitted 2026-10-19 09:16 UTC · v1

Abstract

Wind farms can provide flexibility by curtailing their output during periods of excess generation in the power grid. This can be done through implicit balancing, where the power setpoints of the farm are adapted to anticipate the imbalance price. By doing so, the wind farm helps reduce the resulting system imbalance while also lowering its own costs. However, implicit balancing is challenging since the imbalance prices are unknown in advance and hard to predict. Furthermore, the control strategy of the wind farm can affect the imbalance price formation. To address both challenges, we develop a control policy with 1-minute setpoints using distributional soft actor-critic. Evaluated on real data of the Belgian grid and an offshore wind farm, we show that our method outperforms a price threshold controller, often used in industry, by 88% and performs on par with soft actor-critic. Furthermore, unlike soft actor-critic, our approach can learn a risk-averse policy by optimizing a risk measure, such as conditional value at risk. The risk-averse policy was found to better account for its own impact on the price, showing a reduction in overshoots, while achieving a similar degree of imbalance cost savings. Last, we find that using 1-minute instead of 15-minute setpoints further reduces imbalance costs for all evaluated control policies.

Closest papers

Nearest neighbours by meaning across the whole index.

Value-oriented policy distillation for interpretable wind farm power control

cs.AI eess.SY ·2026-11-05 ·Max Bruninx, Timothy Verstraeten, Jan Decuyper, Ann Nowé, Jan Helsen
similarity 0.70
Abstract

Wind farm power control can help the system maintain balance and avoid blackouts. In Europe, this can be achieved through implicit balancing, where a wind farm adjusts its power setpoint in anticipation of the imbalance price using information published by the Transmission System Operator. Thereby, the wind farm helps restore system balance while also reducing its own imbalance costs resulting from forecast errors. However, implicit balancing is a challenging problem since the settlement price is unknown in advance and can be impacted by the pursued control strategy. Reinforcement learning methods could help to solve this problem, but lack the interpretability needed to be deployed on real wind farms. Therefore, we propose a policy distillation framework using explainable boosting machines (EBMs), a variant of generalized additive models, to obtain a transparent control policy. To increase the value captured by the EBM model, we introduce a value-oriented perspective to policy distillation inspired by research in the field of energy forecasting. Furthermore, we suggest to include expert knowledge during model design, such that the control strategy is also justifiable to stakeholders. Using real data of the Belgian grid and an offshore wind farm, we find that our value-oriented and expert-based controller can reduce imbalance costs by 82%, while outperforming a price threshold controller as well as a policy distillation baseline using decision trees.

0.21

Distributionally Robust Chance-Constrained Reinforcement Learning for Home Energy Management under Electric Vehicle Uncertainty

eess.SY ·2026-10-18 ·Meng Yuan, Ye Wang, Yi Mei, Ramesh Rayudu, Alan Brent +2 more
similarity 0.62
Abstract

The electrification of residential energy systems and adoption of electric vehicles (EVs) create demand-side flexibility. However, random driving behaviour makes it difficult to ensure EV readiness at departure and maintain thermal comfort. To address this, we formulate residential scheduling as a distributionally robust (DR) chance-constrained Markov decision process, whose 2-Wasserstein ambiguity set guards against tail risk and distribution shift in EV behaviour. The intractable DR chance constraint is then replaced by a sufficient tightened expectation constraint under the nominal distribution. The resulting problem is solved by a Lagrangian soft actor--critic with distributional cost critics, and an online boundary guard enforces EV reachability and thermal comfort within the actuator limits at every step. Five controllers are compared over one year in Wellington, New Zealand, under a degraded forecast. The proposed controller is the cheapest controller that meets the EV target on every day and both chance constraints under forecast error and random driving behaviour. Compared with the rule-based baseline, the only other controller to do so, it lowers the annual cost by 8.6% and the comfort violation hours by 69%. The ablation study confirms that the boundary guard ensures pointwise feasibility within the actuator limits, and the DR chance constraint reduces tail risk.

0.16

DER Allocation without Load Prediction via Reinforcement Learning

eess.SY ·2026-08-17 ·Abed AlRahman Al Makdah, Aravind Ramana, Shaofeng Zou, Oliver Kosut, Lalitha Sankar
published similarity 0.60
Abstract · 5 pages. Presented at the 2026 IEEE Power & Energy Society General Meeting (PES GM)

The growing variability of renewable generation increases the need for fast and flexible grid-balancing mechanisms. Existing frameworks for distributed energy resource aggregations (DERAs) rely on short-term forecasts of net demand, making their performance highly sensitive to prediction errors. In this paper we present a forecast-free reinforcement learning (RL) framework for DERA allocation that learns optimal policies directly from operational data. We model the DERA dynamics as a deterministic linear system and the exogenous net load as a feature-based linear Markov process, capturing short-range temporal dependencies without explicit forecasting. We derive a closed-form expression for the optimal policy, which is learned through a least-squares value iteration (LSVI) algorithm using data collected across episodes. The proposed framework preserves the interpretability and constraint satisfaction of DER model while adapting to stochastic demand variations through data-driven updates. Numerical experiments on real California Independent System Operator (CAISO) net-demand data demonstrate that the learned controller achieves high tracking accuracy and stable regulation across heterogeneous DER aggregators without requiring any demand prediction.

0.37

From Market Signals to Maintenance Decisions: Grid-Informed Maintenance Optimization in Offshore Wind Farms

eess.SY math.OC ·2026-10-20 ·Keyi Wang, Robert Mieth, Ahmed Aziz Ezzat
similarity 0.58
Abstract

The reliable and cost-effective integration of large-scale offshore wind farms into power systems is largely influenced by maintenance operations. To date, power producers primarily rely on farm-level information (e.g., health signals, power generation, weather conditions) for maintenance planning of wind turbine fleets. We propose to consider additional grid-level information in maintenance operations, such as generation curtailment and market pricing. Periods of high generation curtailment or negative electricity pricing can be viewed as opportunities for preventive maintenance, since assets are fully or partially shut down due to grid-imposed constraints. We first analytically establish that access to market information has non-negative economic value for maintenance scheduling under a stylized setting, providing a theoretical foundation for treating market and grid information as a maintenance opportunity. We then formulate a grid-informed opportunistic maintenance scheduling model as a receding-horizon mixed integer linear program that jointly leverages both farm- and grid-level signals under realistic conditions to determine a maintenance plan for offshore wind turbines. Considering a 600-MW offshore wind farm in the United States, our proposed method reduces annual maintenance costs by up to 10%, while enhancing asset availability and resource utilization, relative to advanced maintenance models that solely rely on farm-level information.

0.40

Offline Reinforcement Learning for Wind Farm Control: A Wind Tunnel Study under Dynamic Wind Directions

cs.LG ·2026-09-11 ·Yuhan Su, Hongyang Dong, Simone Tamaro, Filippo Campagnolo, Carlo L. Bottasso +1 more
benchmark similarity 0.58
Abstract

This paper addresses the wind farm power maximization problem in the presence of wind direction changes. Specifically, a model-free Modified Twin Delayed Deep Deterministic Policy Gradient with Behavior Cloning (MTD3-BC) algorithm is proposed to tackle this task through yaw control under varying wind direction conditions. MTD3-BC is an offline reinforcement learning (RL) algorithm that aims to infer good behavior from only a precollected offline dataset. Additionally, to ensure smooth and moderate yaw adjustments, a new action consistency term is introduced into the policy optimization objective. Unlike online RL methods, MTD3-BC does not require extensive interactions with a wind farm simulator during training, significantly reducing computational costs and training time. A wind tunnel experiment is conducted to validate the effectiveness of the algorithm under varying wind directions. The results demonstrate that MTD3-BC successfully mitigates wake effects, delivering farm-level power gains of approximately 10\% over the baseline greedy strategy and performance on par with a data-calibrated model-based wake-steering benchmark, while requiring no wake model and only a small fraction of the training cost of online RL. To our knowledge, this is the first time an offline RL wind farm control policy has been validated and demonstrated experimentally.

0.30

Safely Mitigating Grid Imbalance: A Hybrid Distributional Wide-Margin Recourse Approach

eess.SY ·2026-11-17 ·Shoaib Hussain, Mostafa Farrokhabadi, Hamidreza Zareipour
similarity 0.58
Abstract · 17 pages, 10 figures, 3 tables

This paper proposes the Multi-Agent Distributional Wide-Margin Policy Gradient (MADWMPG) framework: a dual-policy hybrid imitation and safe reinforcement learning (IL-SRL) method designed for safely mitigating grid imbalance in multiphase active distribution networks with distributed soft open points (SOPs). The framework comprises a wide-margin constraint barrier recourse (WCBR) module that seeks the maximal separation of constraint-compliant and unsafe actions and learns to proactively suppress exploratory violations. The framework addresses operational volatility using a novel adaptation of noncrossing quantile regression (NQR) distributional critics that track the complete distribution of policy returns. The set of behavior and recourse policies integrates an instantaneous physics-based and differentiable action projection mechanism that generates SOP constraint-compliant actions without relying on active learning. Policy safety is made responsive using an enhancement of conventional experience replay, and the safety performance of the framework is supported by probabilistic bounds derived in the paper. The proposed framework was trained and tested on a modified multiphase IEEE 123-bus test feeder. Results demonstrate its superiority to conventional reinforcement learning and SRL methods and show that it is highly competitive against the nonlinear AC optimal power flow.

0.28

Comparative Evaluation of MILP, MPC, and Reinforcement Learning for Commercial Battery Dispatch Under Time-of-Use Tariffs

eess.SY ·2026-09-13 ·Hafiz Majid Hussain, Wajiha Samar, Lurian Klein, Pedro Nardelli
benchmark similarity 0.57
Abstract

Battery energy storage systems (BESS) paired with rooftop photovoltaics (PVs) can deliver measurable cost savings under time-of-use (TOU) electricity tariffs; however, the relative performance of model-based and model-free dispatch strategies remains insufficiently benchmarked on full-year, real-world commercial datasets. This paper presents a full-year (2023) comparative evaluation of three BESS dispatch approaches using data from a commercial PV installation operating under a TOU tariff. The examined strategies include: (i) a mixed-integer linear programming (MILP) formulation with perfect foresight, providing an oracle performance benchmark under the assumed model; (ii) a model predictive control (MPC) scheme based on a day-ahead persistence forecast, representing a low-complexity deployable approach; and (iii) a soft actor-critic (SAC) deep reinforcement learning agent trained under causal information constraints. The MILP benchmark achieves an annual cost reduction of 24.6\% relative to a no-storage baseline. The persistence-based MPC approach recovers 99.2\% of this benchmark using only prior-day data. In contrast, the evaluated SAC agent yields an annual cost higher than the no-storage baseline. This outcome is analyzed in the context of known challenges in reinforcement learning for energy systems, including limited observability and reward design. Overall, the results indicate that, for the studied dataset and tariff structure, persistence-based MPC captures nearly all achievable economic benefits under practical deployment constraints, whereas the considered RL configuration does not yield competitive performance under the same information limitations.

0.21

Control-Oriented Scenario Tree Construction through Reinforcement Learning

cs.AI cs.LGeess.SY ·2026-08-10 ·Fabio Pavirani, Bert Claessens, Pierre Pinson, Chris Develder
similarity 0.57
Abstract

Multistage stochastic model predictive control (MPC) handles uncertainty by optimizing over a scenario tree, a finite branching approximation of future outcomes constructed from sampled forecasts. To build such a tree, conventional methods focus on matching the underlying probability distribution---e.g., via Wasserstein-based scenario reduction---but improved distributional accuracy does not necessarily yield better control performance. We propose a control-oriented approach that learns scenario tree construction directly from its impact on downstream decisions. Fixing the tree topology, we formulate tree construction as a sequential assignment of sampled scenarios to leaves. This assignment is parameterized by an attention-based policy over the scenario set and trained using reinforcement learning, with closed-loop control profit as the objective. Training is stabilized by an asymmetric critic that leverages realized future trajectories. We evaluate the method on a risk-averse battery arbitrage problem. Across a range of forecast set sizes, the learned construction consistently achieves the highest profit, outperforming classical forward and backward reduction methods and certainty-equivalent (single-trajectory forecast) control. The learned policy also exhibits greater robustness on challenging instances, consistently demonstrating better tail-risk characteristics. Analysis of the resulting trees indicates that our method constructs compact, selectively branching structures that capture high-impact events while keeping most trajectories nearly deterministic. These findings highlight that the value of a scenario tree depends critically on the decisions it supports, and provide an effective framework to train scenario tree constructors merely based on the closed-loop control optimization signal.

0.04