CS Paper Radar

Converter-Grid Interaction Stability Guaranteed Safe Deep Reinforcement Learning for Energy Storage Systems in Grid Frequency Support

Fei Liu, Mengfan Zhang, Zhipeng Li, Frede Blaabjerg, Qianwen Xu

eess.SY ·submitted 2026-09-15 08:22 UTC · v1

Abstract

The growing integration of converter interfaced renewable energy resources (RESs) intensifies stability challenges. Energy storage system (ESS) can provide fast and flexible frequency support to mitigate frequency deviations. However, the interface converter of ESS may encounter converter-grid interaction stability issues. This paper proposes a converter-grid interaction stability guaranteed safe DRL (CIS-DRL) method for ESS integrated power systems to achieve frequency regulation. We first obtain a double DNN-based stability region to identify the guaranteed converter-grid interaction stability. Next, a novel converter-grid interaction stability Safe-TD3 (CIS-STD3) algorithm is designed that integrates a stability feasibility projection layer to map unsafe actions into stable action set before execution, enforcing converter-grid interaction stability as a hard constraint throughout learning process. The proposed approach enables ESS for grid frequency support with 100% converter-grid interaction stability without violations. Experimental results show that the proposed CIS-DRL method achieves improved frequency regulation performance while preventing unstable operating points, demonstrating its practical applicability for real time ESS frequency support.

Comment
10 pages, 13 figures. Submitted to IEEE Transactions on Smart Grid

Closest papers

Nearest neighbours by meaning across the whole index.

Physics-Informed Graph-Based Safe RL with Constraint-Specific Critics for Community Battery Coordination in Distribution Network

cs.AI eess.SYmath.OC ·2026-10-24 ·Chenghao Huang, Yue Yang, Hao Wang
top venue similarity 0.65
Abstract

High penetrations of distributed energy resources (DERs) and fast-varying electric vehicle (EV) charging intensify voltage and line-limit violations in distribution networks (DNs), making safe, low-cost operation increasingly challenging. This paper proposes a physics-informed safe deep reinforcement learning (DRL) framework for coordinating community battery energy storage systems to mitigate constraint violations while minimizing operating cost. To represent nonlocal spatiotemporal interactions, we develop a dual-level physics-informed graph representation in which line impedance and capacity shape spatial information propagation, while power-balance, voltage-profile, and flow-reconstruction losses regularize the latent state toward control-relevant physical consistency. Building on this representation, we develop Lagrangian Dedicated-Critic PPO (LDCPPO), which implements constraint-specific value decomposition: independent reward and cost critics preserve heterogeneous voltage- and line-loading-related learning signals and combine their advantages only during the Lagrangian policy update. Case studies on DNs with high rooftop photovoltaic penetration and EV charging demonstrate that the proposed method reduces the key voltage-violation degree by up to 80.5% compared with the tested baselines, while also achieving the lowest voltage-violation count and line-overload degree with competitive operating cost.

0.31

Stability-Constrained Imitation Learning for Performance-Guaranteed Neural Feedback Frequency Control in Microgrids

eess.SY ·2026-11-17 ·Sajid Kamal, Yichen Zhang, Feng Qiu, Tianqiao Zhao
similarity 0.62
Abstract · 10 pages

Guaranteeing performance for frequency control is challenging within traditional control paradigms, as it demands precise disturbance estimation and real-time control signal computation within a timeframe of seconds. This paper addresses this challenge by proposing a stability-certified imitation learning framework for frequency regulation in low-inertia microgrids. Based on the frequency response model of the microgrid, an offline numerical optimal control is employed to generate near-optimal pairs of state and control dataset, which are subsequently used to train a deep neural network (DNN) feedback controller. To ensure the closed-loop stability, the DNN controller is trained with a set of Lyapunov-based stability constraints derived using absolute stability theory. Such an imitation learning paradigm can significantly improve the training efficiency, while the resulting DNN controller acts as an end-to-end function that simultaneously handles disturbance estimation and control optimization. The proposed framework is validated on a modified IEEE 33-node diesel-wind microgrid. The DNN controllers are deployed in wind turbine generators (WTGs) to provide decentralized, intelligent supplementary active power support to always maintain frequency deviations within predefined safety bounds under different disturbances while preserving closed-loop stability.

0.34

LLMs are Few-Shot Decision-Makers: Generalized Context-Aware Microgrid Frequency Control through Prompt Decision Transformer

eess.SY cs.AI ·2026-08-22 ·Xu Yang, Chenhui Lin, Haotian Liu, Kaihang Deng, Yunhe Li +1 more
similarity 0.61
Abstract

The rapid evolution of energy structures has positioned microgrids as pivotal components of next-generation power systems, offering enhanced resilience and renewable energy integration. However, the inherent low inertia, complex dynamics, and poor model conditions of microgrids necessitate advanced data-driven frequency control strategies. Although reinforcement learning (RL) has demonstrated certain potential and advantages, existing RL methods often struggle with generalization across diverse microgrid configurations and lack adaptability to unseen environments, particularly when explicit system parameters are unavailable. To address these challenges, in this paper, we introduce a novel prompt decision transformer (Prompt-DT) architecture for microgrid frequency control. Unlike traditional approaches that rely on hard-to-obtain environmental characteristic parameters, the proposed method leverages few-shot expert historical trajectories as prompts to guide autonomous perception and adaptive decision-making. In addition, we propose a context-aware training and execution mechanism utilizing self-supervised contrastive learning to enhance environment recognition and prompt utilization efficiency. In addition, a physics-informed prompt design technique that filters prompts based on cumulative reward and frequency volatility is proposed, ensuring high-quality physical guidance during online execution. Finally, to ensure generalization in unseen environments with limited data, we develop a lightweight finetuning approach that achieves performance comparable to full-parameter finetuning with minimal adjustments.

0.15

Battery Energy Storage Sizing in PV Ramp-Rate Smoothing using Constraint-Aware Reinforcement Learning-Based Controller

eess.SY ·2026-11-13 ·Arash Omidi, Tanmay Mishra, Mads R. Almassalkhi
benchmark similarity 0.61
Abstract

This paper presents a constraint-aware reinforcement learning (CARL) controller for battery energy storage system (BESS) sizing and photovoltaic (PV) ramp-rate smoothing. Rapid PV power fluctuations caused by cloud transitions can introduce operational and stability challenges in power systems. To mitigate these impacts, grid codes impose ramp-rate limits (RRL) on PV power injected to the grid. BESSs can support such grid codes by absorbing or supplying power during rapid variations. However, the extent to which BESSs can reduce PV-induced variability depends on (i) the BESS power and energy capacities; and (ii) the adopted controller. Thus, BESS sizing and controller design are coupled. Conventional moving-average control can increase battery usage due to filter-induced lag, while ramp-rate control is reactive and does not account for the long-term impact of actions on battery state of charge and future ramp events. Optimization-based methods can reduce BESS requirements by anticipating future PV variations, but their performance depends on accurate PV forecasts. To address these limitations, the CARL controller learns a causal state-feedback policy from historical PV data without requiring forecasts during deployment. Ramp-rate, converter-power, and state-of-charge constraints are incorporated into a causal feasibility interval that guarantees feasible actions by definition. A full-horizon input-convex neural network (ICNN)-based optimization is developed as a noncausal benchmark for the minimum achievable BESS requirements. Evaluation using three real-world PV datasets shows that, at a 10\%/min ramp-rate limit, CARL reduces peak BESS power by 8.7--10.4\% and energy capacity by 47.5--62.6\% relative to conventional ramp-rate control while maintaining the grid-codes compliance.

0.20

Safe Deep Reinforcement Learning for Energy-Efficient HVAC Control in Multi-Zone Residential Buildings

eess.SY ·2026-08-18 ·Oussama Ziadi, Abdelilah Rochd, Samir Idrissi Kaitouni, Mohamed Oualid Mghazli, Adnane Saoud
published similarity 0.59
Abstract · 6 pages, 5 figures. Accepted to IEEE Conference on Control Technology and Applications (CCTA) 2026

HVAC systems represent a major share of building energy consumption. Traditional control strategies are limited in coordinating energy-comfort tradeoffs across multiple zones simultaneously. Reinforcement learning (RL) offers adaptive, data-driven control that optimizes performance over time. However, deploying learned neural network controllers in safety-critical building systems remains challenging due to lack of formal safety guarantees. We propose a safety-certified deep RL framework for multi-zone residential HVAC control. Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) agents are trained in an EnergyPlus/Sinergym simulation to minimize energy consumption while maintaining thermal comfort. Post-training safety certification is performed on the PPO policy using Lipschitz-based forward invariance analysis, building on existing tools for the computation of Lipschitz constants for neural networks, to guarantee constraint satisfaction. Both agents are evaluated over an annual simulation cycle in an eight-zone variable refrigerant flow (VRF) testbed. The PPO agent achieves 67\% comfort violation reduction compared to rule-based control, while the SAC agent achieves 27.6\% energy savings. The PPO policy satisfies formal safety certification with a margin of $2.003^\circ$C. These results demonstrate the feasibility of combining reinforcement learning with post-training safety verification for multi-zone building control.

0.20

LLM-Based Hierarchical Coordinated Control with Continuation-Aware Policy Learning

cs.AI ·2026-08-15 ·Changhong He, Jinda Gao, Xinkuan Liu, Le Zhang, Xizi Luo +1 more
similarity 0.59
Abstract · 30 pages, 5 figures

Coordinating multiple interacting units in complex engineering systems is challenging when system interactions are difficult to model, operational information is heterogeneous, and low-level actions must satisfy strict constraints. We propose an LLM-based hierarchical framework in which the LLM coordinates interacting units based on heterogeneous operational context, while task-specific controllers or optimizers generate executable and constraint-aware actions. We further introduce Continuation-Aware GRPO to capture the consequences of coordination decisions over subsequent control intervals. Rather than judging a decision only by its immediate outcome, the method also evaluates how the system evolves afterward under the current policy. We validate the framework on multi-ramp traffic control and virtual power plant (VPP) energy management, using simplified system models for training and more realistic simulators for evaluation. Across both tasks, the proposed method consistently outperforms direct task-specific control and optimization, end-to-end reinforcement learning, rule-based and RL-based hierarchical coordination, and prompting-only LLM coordinators, demonstrating the value of heterogeneous-context reasoning, hierarchical execution, and continuation-aware policy learning.

0.10

Robust Scenario-Based Data-Enabled Predictive Control of a Battery Energy Storage System: An Experimental Study

eess.SY math.OC ·2026-10-06 ·Sebastian Zieglmeier, Nikolas Recke, Mathias Hudoba de Badyn
similarity 0.58
Abstract · 8 pages, 5 figures

Battery energy storage systems must respect strict state-of-charge (SOC) limits to prevent overcharge and deep discharge, a task complicated by measurement noise and costly-to-model nonlinear dynamics. Data-enabled predictive control (DeePC) resolves the costly modeling issue by predicting future behavior directly from data. However, its regularization robustifies only the prediction and not the constraints. Scenario-based DeePC (Scenario-DeePC) extends DeePC with the scenario approach, building constraint robustness fully data-driven from observed prediction errors rather than an assumed disturbance distribution. This paper presents the first real-world deployment of Scenario-DeePC, on a grid-connected battery system at the NEST research facility, whose SOC estimate exhibits abrupt, irregular recalibration jumps in addition to ordinary noise. Compared to standard DeePC, Scenario-DeePC achieves comparable tracking performance with substantially fewer constraint violations. Its adaptive scenario buffer further tightens constraint handling automatically, improving robustness to unpredictable recalibration events.

0.17

LLM-Enhanced Deep Reinforcement Learning with Human-in-the-Loop Calibration for Real-Time Optimal Power Flow

eess.SY cs.AI ·2026-11-11 ·Jiayang Tu, Bozhen Jiang, Bojun Du, Yunxiao Ling, Ershun Du +1 more
similarity 0.58
Abstract

Real-time optimal power flow (RT-OPF) requires fast and reliable decision-making under time-varying operating conditions. However, many practical operating preferences and safety related requirements are difficult to encode using conventional numerical rewards alone, which limits the effectiveness and controllability of deep reinforcement learning (DRL) for RT-OPF. To address this issue, this paper proposes an LLM-enhanced DRL framework with human-in-the-loop (HITL) for RT-OPF. The proposed method augments the physics-based reward with sparse language evaluative feedback during offline training, so that semantic operational requirements can be incorporated into policy learning. To improve reliability in RT-OPF, a selective HITL calibration mechanism is further introduced to inspect and correct questionable LLM judgments before they continuously shape the learning process. Case studies on the IEEE 30-bus and IEEE 118-bus systems show that LLM feedback improves policy safety over vanilla TD3, while further HITL calibration achieves the best overall performance and more stable training behavior. These results demonstrate the effectiveness of combining language guidance and expert calibration for RT-OPF.

0.14