CS Paper Radar

Strategic Investment Decision Making for Value Creation in Energy Transition: A Reinforcement Learning Approach

Yasaman Cheraghi, Reidar B. Bratvold, Aojie Hong, Ressi B. Muhammad, Sergey Alyaev

cs.CE cs.LGeess.SY ·submitted 2026-10-07 18:29 UTC · v1
benchmark

Abstract

The global challenge of climate change has driven significant steps to reduce CO2 emissions, guided by international agreements like the Paris Agreement of 2015. Acting too slowly could result in future losses and reputational damage, while moving too quickly could jeopardize shareholder value due to the marginal profitability or potential losses due to technology immaturity of many renewable projects. To navigate this complex transition, energy companies must adopt Sequential Decision Making (SDM) strategies to maximize value creation from decision flexibility under uncertainties. To support this, we developed a custom simulation environment to model the dynamic energy landscape up to 2050. Building on this, we designed a multi-criteria SDM framework that explores various decision strategies related to different portfolios for allocating funds across three sectors: oil & gas, renewables, and CO2 reduction. It aims to maximize value during the transition while accounting for uncertainties in productions, energy prices, and costs. This framework has three objectives: maximizing profit, minimizing CO2 social costs, and enhancing competitive advantage in the renewable energy sector. This research evaluates the use of Reinforcement Learning (RL) to identify optimal investment policies within the defined SDM framework. The agent's sequential decisions shape a virtual dynamic environment by influencing key variables such as oil and gas production, renewable energy output, CO2 emissions, and revenues. Through repeated interaction, the RL algorithm explores the state space and learns an optimal policy under uncertainty. We benchmark the RL strategy against a set of manually defined baseline policies and find it consistently outperforms them in adaptability and long-term value creation.

Closest papers

Nearest neighbours by meaning across the whole index.

Reinforcement Learning for Sequential Solar PV Policy Design under Uncertainty: An Agent-Based Approach

cs.AI ·2026-09-04 ·Iias Faiud, Jonaid Shianifar, Michael Schukat, Karl Mason
published similarity 0.66
Abstract

Designing effective and fiscally sustainable policies for solar photovoltaic (PV) adoption requires balancing adoption gains against public expenditure under uncertainty and heterogeneous decision-making. This study formulates PV policy design as a sequential decision problem and integrates reinforcement learning (RL) with a stochastic agent-based model (ABM) that simulates yearly solar PV adoption under uncertainty. A policymaker agent selects annual incentives, including capital grants, subsidised loan rates, and feed-in tariffs, over a 16-year horizon. Adoption--cost trade-offs are explored by varying policy preferences within a scalarised reward framework. Policies are learned using PPO, SAC, and TD3 and evaluated under stochastic simulation. The results show that this approach produces a clear trade-off structure: the highest-adoption policy (TD3, $w_{\text{cost}}=0.5$) achieves approximately 4,145 adopters at a cost of EUR 41.73 million, while the lowest-cost policy (PPO, $w_{\text{cost}}=2.0$) reduces expenditure to EUR 7.27 million with 2,682 adopters. The balanced policy (PPO, $w_{\text{cost}}=1.6$) achieves 3,495 adopters at a cost of EUR 22.47 million. Across algorithms, consistent trade-off patterns are observed, indicating robustness of the adoption--cost relationship. Compared with static baseline policies, the RL framework explores a broader range of policy configurations. These findings demonstrate the potential of RL as a flexible tool for adaptive policy design under uncertainty.

0.35

Eliciting ESG Preferences for Reinforcement Learning-Based Portfolio Optimization

cs.CE q-fin.PMcs.LG ·2026-09-02 ·Giovanni Dispoto, Marcello Restelli, Carmine Ventre
similarity 0.60
Abstract

Modern portfolio management increasingly demands a balance between traditional risk-adjusted returns and strict Environmental, Social, and Governance (ESG) mandates. Current Reinforcement Learning (RL) approaches typically optimize for a single ESG provider, neglecting the significant divergence in rating methodologies across the industry and the unintuitive nature of manually weighting conflicting objectives. This paper addresses these limitations by formulating ESG-aware portfolio optimization as a Multi-Objective Reinforcement Learning (MORL) problem that simultaneously incorporates ratings from three distinct ESG agencies. To bridge the gap between high-dimensional algorithmic trade-offs and human decision-making, we integrate a Preference Elicitation framework using Gaussian Processes. This system enables practitioners to align the RL policy behavior with their preferences through intuitive pairwise comparisons of candidate portfolios based on their Sharpe ratios and aggregate ESG scores. We systematically evaluate our framework by employing prompt-conditioned LLMs as synthetic preference oracles under varied regional conditions and assess how the methodology adapts to heterogeneous investment behaviors and sustainability preferences. The experiments demonstrate feasibility in a controlled synthetic setting and characterize sensitivity to oracle and prompt design.

0.12

Online Learning-Based Adaptive Hybrid Benders Decomposition for Risk-Averse Optimal Sizing

eess.SY ·2026-09-22 ·Saif Ahmad, Seifeddine Benelghali, Hafiz Ahmed
similarity 0.60
Abstract

Optimal sizing problem (OSP) for battery energy storage system (BESS) under uncertain inputs is often formulated as a two-stage stochastic program (2SP), which typically introduces computational and RAM (memory) bottlenecks for large scenario sets. Benders Decomposition (BD) is a popular approach for tackling this problem, but it suffers from slow convergence to the exact solution. To address this problem, we propose an accelerated online learning-based adaptive hybrid BD algorithm for a risk-averse 2SP formulation. The proposed method avoids getting stuck in the infeasible region during early iterations by explicitly embedding a carefully selected scenario in the master problem, while a tail-relevant scenario selector based on online learning helps to avoid solving the entire scenario set at every iteration. The OSP is formulated to select a behind-the-meter BESS in a multi-site energy community with existing renewables. The use case explores an interesting middle-ground between deterministic acceleration methods and training-heavy ML models, showcasing the potential of ML-assisted decision-making. Compared with vanilla BD, OLAH-BD reduces subproblem evaluations and total wall time by up to 80\% under the same tolerance settings.

0.19

Robust Procurement Planning for Community Choice Aggregators: Managing Market Price Uncertainty

eess.SY ·2026-10-20 ·Manar Jaradat, James Himelic, Neha Patankar
code/data similarity 0.59
Abstract

Community Choice Aggregators (CCAs) play an important role in the energy transition by procuring renewable electricity for local communities while balancing affordability, resource adequacy, and clean-energy requirements. In California, these long-term procurement decisions are coordinated through integrated resource planning (IRP). However, conventional IRP frameworks typically represent wholesale electricity prices deterministically, even though price uncertainty can expose CCAs to substantial financial risk by affecting the cost of market imports and, particularly, the value of export opportunities. This paper develops a robust optimization framework for CCA resource procurement that hedges against uncertainty in wholesale import and export prices. We extend the open-source GenX capacity expansion model with a budgeted robust optimization formulation that protects procurement decisions against adverse price realizations while remaining computationally tractable for large-scale, multistage planning studies. Using data from a California CCA, we show that market price uncertainty can materially change optimal procurement strategies. Rather than aggressively expanding renewable capacity based on expected export revenues, robust portfolios reduce dependence on uncertain market revenues, leading to more conservative renewable investments and less reliance on market exports. We demonstrate that moderate uncertainty budgets provide most of the risk-reduction benefits of robust planning at limited additional cost, whereas higher robustness levels yield diminishing benefits as portfolio conservatism increases. These findings provide guidance for selecting uncertainty budgets in IRP studies and demonstrate how robust optimization can support affordable and resilient procurement decisions as CCAs contribute to the transition toward low-carbon electricity systems.

0.40

RL-PaO: Prediction as Action in Decision Making under Uncertainty

cs.LG ·2026-09-29 ·Jiahui Feng, Dafang Zhao, Zheng Chen, Zhengmao Li, Lingwei Zhu
similarity 0.59
Abstract

Decision-making under uncertainty often relies on predicted parameters, yet accurate prediction does not necessarily lead to good operational decisions. Aligning prediction with downstream optimization requires learning from the consequences of the decisions those predictions induce. We introduce RL-PaO, a reinforcement learning framework that integrates system formulation, optimization, and decision execution into a single environment. This yields a Markov decision process in which prediction is regarded as action: it shifts the environment to produce subsequent context and reward that explicitly aligns prediction error with realized cost, and learning the optimal policy does not require differentiating through the black-box solver. We evaluate RL-PaO on day-ahead energy scheduling using real historical data. On the test year, RL-PaO achieves the lowest annual cost among the non-oracle baselines, achieving on average $10\%$ cost reduction. Moreover, RL-PaO is capable of further analyses to provide strong interpretability both from the policy evolution perspective and the cost-accuracy trade-off.

0.24

Comparative Evaluation of MILP, MPC, and Reinforcement Learning for Commercial Battery Dispatch Under Time-of-Use Tariffs

eess.SY ·2026-09-13 ·Hafiz Majid Hussain, Wajiha Samar, Lurian Klein, Pedro Nardelli
benchmark similarity 0.58
Abstract

Battery energy storage systems (BESS) paired with rooftop photovoltaics (PVs) can deliver measurable cost savings under time-of-use (TOU) electricity tariffs; however, the relative performance of model-based and model-free dispatch strategies remains insufficiently benchmarked on full-year, real-world commercial datasets. This paper presents a full-year (2023) comparative evaluation of three BESS dispatch approaches using data from a commercial PV installation operating under a TOU tariff. The examined strategies include: (i) a mixed-integer linear programming (MILP) formulation with perfect foresight, providing an oracle performance benchmark under the assumed model; (ii) a model predictive control (MPC) scheme based on a day-ahead persistence forecast, representing a low-complexity deployable approach; and (iii) a soft actor-critic (SAC) deep reinforcement learning agent trained under causal information constraints. The MILP benchmark achieves an annual cost reduction of 24.6\% relative to a no-storage baseline. The persistence-based MPC approach recovers 99.2\% of this benchmark using only prior-day data. In contrast, the evaluated SAC agent yields an annual cost higher than the no-storage baseline. This outcome is analyzed in the context of known challenges in reinforcement learning for energy systems, including limited observability and reward design. Overall, the results indicate that, for the studied dataset and tariff structure, persistence-based MPC captures nearly all achievable economic benefits under practical deployment constraints, whereas the considered RL configuration does not yield competitive performance under the same information limitations.

0.21

A Stochastic Mean-CVaR Framework for BESS Multi-Market Bidding Strategies

eess.SY ·2026-09-16 ·Younes Zahraoui, Jun Cao, Samir Kouro, Pedro Rodriguez Cortes
published similarity 0.58
Abstract · The paper has been accepted in IECON26 without any comments

Battery Energy Storage Systems (BESS) operators face significant challenges when participating in multiple electricity markets due to the complex coupling of price volatility and stochastic reserve activation. Traditional deterministic dispatch models neglect the "tail risks" associated with extreme market realizations, potentially leading to technical infeasibility or severe economic losses. This paper proposes a risk-aware stochastic optimization framework for the co-optimization of BESS participation in the Day-Ahead (DA) energy market and the manual Frequency Restoration Reserve (mFRR) market. The model explicitly captures multi-dimensional uncertainties by utilizing non-parametric Kernel Density Estimation (KDE) to generate joint price scenarios that preserve the empirical characteristics of European balancing markets. A two-stage stochastic programming approach is employed.To manage the financial exposure to high-impact price events, the Conditional Value-at-Risk (CVaR) metric is integrated into a Mean-CVaR objective function. This allows decision-makers to tune their risk-aversion levels and identify an efficient frontier between profitability and robustness. Simulation results demonstrate that the proposed joint CVaR approach significantly enhances revenue stability compared to deterministic benchmarks.

0.30

Learning Control Policies from Heterogeneous Multi-Horizon Time Series in Battery Energy Management Systems

eess.SY ·2026-09-18 ·Sheng Yin, Vivek Teja Tanjavooru, Holger Hesse, Christoph Goebel
published similarity 0.58
Abstract

This paper introduces Representation-to-Decision (R2D), an end-to-end imitation learning framework that maps heterogeneous multi-horizon time-series inputs directly to battery control decisions through modular Temporal Feature Extractors (TFEs) and a shared latent representation, without an explicit load and PV forecasting step. While accurate forecasting improves prediction quality, optimal control performance remains unguaranteed in prediction-then-optimization pipelines; standard Reinforcement Learning (RL) lacks the long-horizon temporal awareness due to short-window observations, even when forecast signals are available as additional inputs. R2D offers a different perspective: rather than forecasting first and deciding second, it learns to decide directly from raw temporal inputs, with control optimality anchored by an aging-aware Mixed-Integer Linear Programming (MILP) expert through Behavior Cloning (BC). Benchmarked against six controllers on a high-fidelity electro-thermal battery simulation across five industrial sites, R2D achieves 62--77\% of the global clairvoyant optimum, outperforms the tested Model Predictive Control (MPC) and RL benchmarks, and yields battery degradation nearly identical to its MILP teacher across all five sites. Comprehensive ablation studies over temporal backbone, model size, expert formulation, aging-cost weighting, and horizon configuration confirm that an LSTM encoder with a 15-minute single-step control horizon provides the most robust and deployment-ready configuration, and cross-site and single-factor out-of-distribution tests show that generalization to unseen profiles is profile-dependent, with policies trained on high-activity sites transferring most reliably.

0.26