CS Paper Radar

Bidirectional Multimodal Fusion of Sky Images and Time-Series for Solar Forecasting with Large Language Models

Ken Chen, Maneesha Perera, Wei Wang, Sachith Seneviratne, Hansani Weeratunge, Saman Halgamuge

cs.LG ·submitted 2026-09-10 06:26 UTC · v1

Abstract

Short-term photovoltaic (PV) power and global horizontal irradiance (GHI) forecasts are essential for effective dispatch, reserve scheduling, and grid operations. At these forecasting horizons, errors are predominantly driven by cloud induced ramps: relying solely on historical numerical data may struggle to anticipate an incoming cloud, making ground-based sky images a crucial complementary physical signal. Furthermore, forecast performance is highly sensitive to location and local observing conditions, creating a strong need for site-specific data that are often scarce. Recently, large language models (LLMs) have demonstrated competitive performance and high data efficiency in time-series forecasting. Despite their success, existing LLM-based forecasting methods remain predominantly unimodal, relying primarily on historical numerical time-series data. Effectively incorporating sky imagery into an LLM-based forecasting framework remains under-explored and an open challenge. In this paper, we propose SolCloudLLM, an LLM-based multimodal forecasting framework. SolCloudLLM aligns sky-image patches with time-series patches and fuses their corresponding representations through bidirectional multimodal fusion, yielding a unified representation that is subsequently mapped into the embedding space of an LLM. Extensive experiments on the SIRTA and SKIPP'D datasets demonstrate that SolCloudLLM consistently outperforms the best baseline methods in MSE across all forecasting horizons, achieving a maximum relative MSE reduction of 25.4%. Stratified analysis further indicates that the benefits of multimodal fusion are concentrated primarily under cloudy conditions. Notably, SolCloudLLM achieves the best performance in nearly all few-shot settings, whereas other deep learning baselines experience substantial performance degradation and are frequently outperformed by the non-learning physical method.

Closest papers

Nearest neighbours by meaning across the whole index.

GateFormer: Forecast-Space Decomposition for Intra-Hour Solar Irradiance Prediction from Sky Images

eess.IV cs.AIcs.CVcs.LG ·2026-10-22 ·David Hamlyn, Sunny Chaudhary, Jonathon Hare, Tasmiat Rahman
publishedbenchmark similarity 0.76
Abstract

Sub-hourly dispatch, balancing, and storage control increasingly depend on forecasts that anticipate rapid variability in solar generation. Accurate sky-image-based irradiance forecasting requires a combination of visual cloud dynamics with recent irradiance state while adapting to different short-term irradiance-change regimes. To address this, we introduce GateFormer, a GHI-conditioned dual-decoder ViT for intra-hour global horizontal irradiance (GHI) forecasting through forecast-space decomposition. GateFormer embeds observed GHI into the visual token stream, allowing interaction with sky-image patch features through self-attention. From this shared representation, a Trajectory Head predicts future GHI sequence, while a Delta Head predicts final-horizon change from the latest observed GHI. The two final-horizon forecasts are adaptively weighted in prediction space, with magnitude- and direction-supervision targeting the remaining error. On the NREL-TSI benchmark, GateFormer attains 125.01 W/m^2 RMSE at +30 min, improving over a state-of-the-art transformer baseline by 9.6% under paired-timestamp evaluation. Across NREL-TSI, Chilbolton, and SIRTA, GateFormer achieves the lowest final-horizon RMSE among the evaluated baselines, with forecast skill of 22.97-28.86% relative to smart persistence. Component ablations show adaptive weighting outperforms both individual decoders and fixed averaging, distinguishing GateFormer from conventional forecast combination, while the supervised correction further reduces RMSE by 1.68 W/m^2 and MAE by 4.86 W/m^2. Ramp-regime diagnostics show larger gains during operationally important positive ramps, reducing RMSE by 26.27 W/m^2. These results identify forecast-space decomposition as an alternative forecasting formulation, with potential relevance to other multimodal short-horizon forecasting tasks and deployment using only sky images and measured GHI.

0.26

Vision Transformers for Estimating Irradiance Using Data Scarce Sky Images

eess.IV cs.AIcs.CVcs.LG ·2026-10-22 ·David Hamlyn, Sunny Chaudhary, Tasmiat Rahman
publishedhighly cited similarity 0.75
Abstract

Accurate estimation of diffuse horizontal irradiance (DHI) is critical for optimising photovoltaic system performance and energy forecasting yet remains challenging in regions lacking comprehensive ground-based instrumentation. Recent advancements using Vision Transformers (ViTs) trained on extensive sky image datasets have shown promise in replacing costly irradiance measurement equipment, but the scarcity of long-term, high-quality sky imagery significantly restricts practical implementation. Addressing this critical gap, this study proposes a novel dual-framework approach designed for data-scarce scenarios. First, calculated atmospheric parameters, including extraterrestrial irradiance and cyclic time encodings, are integrated to represent sky conditions without utilising any instrumentation. Next, a sequential pipeline initially predicts synthetic global horizontal irradiance (GHI) and uses it as a feature, to refine DHI estimation. Finally, a dual-parallel architecture simultaneously processes raw and overlay-enhanced fisheye sky images. Overlays are generated through unsupervised, physics-informed cloud segmentation to highlight dynamic sky features. Empirical validation is performed using data from the Chilbolton Observatory, chosen for its temperate climate and frequent cloud variability. To simulate data-scarce conditions, models are trained on a single month (e.g., January) and evaluated across a temporally disjoint, full-year test set. Under this setup, the sequential and dual-parallel frameworks achieve RMSE values within 2-3 W/m^2 and 1-6 W/m^2, respectively, of a state-of-the-art ViT trained on the complete dataset. By combining physics-informed modelling with unsupervised segmentation, the proposed method provides a scalable and cost-effective solution for DHI estimation, advancing solar resource assessment in data-constrained environments.

0.56

Constructed regional photovoltaic generation fields improve ultra-short-term distributed solar forecasting

cs.AI cs.CE ·2026-10-12 ·Dayin Chen, Xiaodan Shi, Xin Xu, Zhongyue Su, Shibo Zhu +3 more
similarity 0.71
Abstract

Ultra-short-term forecasting of densely distributed photovoltaic (PV) systems remains challenging because external meteorological and satellite products may not adequately resolve rapid local variations, while heterogeneous PV systems exhibit different generation responses under similar atmospheric conditions. Meanwhile, dense PV fleets provide spatially distributed measurements that reveal regional generation dynamics. Here, we reformulate fleet-level PV forecasting from modeling discrete station time series to predicting explicit, spatially structured regional PV generation fields. We develop SatPV-Field, a regional-to-local forecasting framework that transforms distributed PV measurements into image-like generation fields and models their spatiotemporal evolution. By combining historical PV fields with satellite-derived future shortwave radiation guidance and meteorological forecasts, the framework predicts future regional generation states, which subsequently provide location-specific guidance for individual-station forecasting. Experiments involving 496 distributed PV systems, including 446 source stations and 50 held-out target stations, demonstrate that these regional fields can be effectively forecast and used to improve station-level predictions. The proposed approach reduces station-level MAE by approximately 10.0% relative to the best-performing single-station temporal baseline and by 2.5% relative to a graph-based regional forecasting approach using the same source fleet. These findings establish an explicit spatial-field modeling paradigm that connects regional generation dynamics with individual-site forecasting.

0.11

SolarBench: A global solar energy nowcasting benchmark

cs.CV cs.LG ·2026-09-05 ·Yuhao Nie, Stephen Campbell, Quentin Paletta, Liwenbo Zhang, Tao Jing +16 more
benchmark similarity 0.71
Abstract

As the share of solar power grows, nowcasting weather-driven solar variability becomes critical for reliable energy system operation. State-of-the-art approaches increasingly apply deep learning to sky camera and geostationary satellite observations, but fragmented datasets and inconsistent evaluation make it difficult to determine whether reported improvements generalize across climates, cloud regimes, and photovoltaic (PV) systems. Here we introduce SolarBench, an open global benchmark for image-based solar nowcasting. SolarBench harmonizes more than six million sky and satellite images from 11 diverse sites spanning a decade, together with irradiance or PV output and auxiliary atmospheric data. An accompanying toolbox supports reproducible data access, processing, model development, and evaluation. Using SolarBench, we benchmark representative models and reveal a gap between average forecasting accuracy and the ability to capture rapid solar fluctuations. We further quantify predictability across cloud regimes and demonstrate data-efficient adaptation to new PV systems. SolarBench provides an extensible foundation for fair comparison and methodological innovation in solar nowcasting.

0.17

IMPACT: Inverter-Informed Multimodal Learning for Photovoltaic Generation Forecasting

cs.LG cs.AI ·2026-10-29 ·Anushka Katiyar, Cong Feng, Jing Wang, Tristan Liu, Simon Julien +1 more
similarity 0.70
Abstract

This work develops a physics-informed multimodal deep learning framework for 10- to 110-minute-ahead probabilistic photovoltaic power forecasting. The forecasting model is built on a four-branch architecture that separately encodes plant-level temporal history and inverter-level spatial cloud dynamics through a graph attention encoder with a learned adjacency, known-future solar geometry, and the current meteorological state, fused through gated cross-attention. The geometry branch adds two forward-looking physical signals, spatial cloud heterogeneity and inverter ramp-down activity, each empirically weighted toward the horizon where it is most predictive. The decoder anchors its prediction to a smart persistence baseline, learning a horizon-conditioned correction, and the resulting quantile forecasts are calibrated online through a kernel-weighted adaptive conformal calibration step. The proposed model outperforms all 14 baselines, achieving a forecast skill score (FSS) of 0.2950, 0.099 above the next-best model. A branch-composition ablation shows that every branch contributes positively and the model retains substantial skill in challenging overcast and variable sky conditions (FSS 0.198 and 0.255).

0.10

Foundation-Style Pretraining for Scalable and Transferable Distributed PV Forecasting

cs.LG ·2026-10-20 ·Dayin Chen, Xiaodan Shi, Xin Xu, Zhongyue Su, Shibo Zhu +4 more
similarity 0.67
Abstract

The rapid growth of distributed photovoltaic (PV) generation is increasing the demand for accurate forecasting across large numbers of geographically dispersed stations. Existing approaches are commonly developed for individual sites or specific forecasting horizons, requiring repeated training and maintenance as new stations are integrated and limiting direct reuse at previously unseen sites. To address this scalability gap, this paper presents a domain-specific foundation-style pretraining framework that learns a shared forecasting model from heterogeneous multi-site and multi-horizon PV data. NWP-assisted forecasting is formulated as a sequence-completion problem that unifies historical PV power, meteorological observations, future numerical weather prediction (NWP) forecasts, and time encodings. Three pretraining objectives are designed and systematically evaluated, including forecast-aligned reconstruction, generic reconstruction, and mixed reconstruction. The mixed objective combines the operational structure of PV forecasting with random sequence reconstruction and provides the most balanced performance across forecasting accuracy, degraded NWP conditions, and downstream adaptation. Experiments on 479 distributed PV stations use 340 stations for model development and 139 spatially held-out stations for zero-shot evaluation. Across horizons from 6 hours to 7 days, the multi-site pretrained MR model outperforms representative station-specific and shared baselines, as well as generalist time-series models, including a recent large-scale pretrained foundation model. The pretrained model can also be directly deployed at new PV stations and subsequently adapted to region-specific operational objectives when local data become available, providing a scalable forecasting paradigm for large distributed PV fleets.

0.18

From Nowcasting to Forecasting: Adapting a Reanalysis-Trained

cs.LG physics.ao-ph ·2026-09-03 ·Mikko Partio, Leila Hieta, Ossi Laine
similarity 0.67
Abstract

Accurate cloud-cover forecasts are important for temperature prediction, radiation forecasting, and solar-power operations. Short-range forecasting methods can preserve observed cloud placement during the first forecast hours, but their skill decreases when cloud fields evolve through formation, dissipation and deformation. Longer lead times require accounting for atmospheric evolution, but operational numerical weather prediction (NWP) forecasts may not accurately represent the satellite-observed cloud state at initialization. We develop CloudCast v2, a machine-learning model for 12-hour cloud-cover forecasting from observation-based initial conditions. The model is first trained on the Copernicus European Regional Reanalysis (Ridal2024) to learn cloud-evolution dynamics, and is then adapted to satellite-derived cloud fields using conditional flow matching (Lipman2023), a generative method that transforms noise into cloud-cover forecasts conditioned on the observed initial cloud fields and NWP inputs. CloudCast v2 reduces mean absolute error by 10% relative to its predecessor, CloudCast v1 (Partio2025), over the 1-12 h range. It also overtakes CloudCast v1 in fractions skill score, a neighborhood-based measure of spatial agreement, after approximately 3-6 h, depending on the cloudiness category. These results show that observation-initialized machine-learning forecasts can extend beyond the usual 1-3-hour nowcasting range while retaining spatial detail from satellite cloud fields.

0.24

Horizon-specific Expert Fusion for Photovoltaic Power Forecasting

cs.AI ·2026-09-14 ·Xu Yuqing, Zhou Liguo, Sun Ze, Yu Lei, Jiang Mingming
similarity 0.66
Abstract

Short-term photovoltaic power forecasting requires models to represent regular solar cycles and weather-driven fluctuations whose importance changes with the forecast horizon. This study develops a hierarchical ensemble that combines temporal neural models, historical analogs, state climatology, and gradient-boosted trees. Solar geometry and numerical weather forecasts describe the expected generation conditions, while horizon-specific convex weights combine complementary predictions. A separate calibration step uses available historical forecast errors to account for recent bias. The framework is evaluated on public PVDAQ data at 15--240-minute horizons and on three GEFCom2014 solar zones at hourly horizons up to four hours. On PVDAQ, the ensemble achieves a daylight capacity-normalized mean absolute error of 4.315%, reducing error by 4.11% relative to full-feature LightGBM and by 6.03% relative to fine-tuned Chronos-2 under identical calibration. Expert-removal experiments identify redundancy within the ensemble. Across three training seeds on GEFCom2014, learned fusion improves upon equal weighting but performs comparably to LightGBM. The results support horizon-specific combination as a useful forecasting strategy while showing that its advantage over strong individual models depends on the dataset and evaluation period.

0.12