CS Paper Radar

Probabilistic Machine Learning for Short-Term Atmospheric Seeing Forecasting

Mary-Joe Medlej, Rahul Srinivasan, Simon Prunet, Aziz Ziad, Christophe Giordano

cs.LG astro-ph.IM ·submitted 2026-10-21 12:35 UTC · v1
publishedbenchmark

Abstract

Accurate short-term forecasting of atmospheric seeing is of significant interest for astronomical observations and free-space optical communication systems, where rapid variations in optical turbulence can affect system performance and operational planning. In this work, we benchmark machine learning models for predicting atmospheric seeing up to two hours in advance, with a particular focus on probabilistic forecasting and uncertainty quantification. We compare two probabilistic approaches, Gaussian Processes (GPs) and a normalizing-flow- based model (FloTS), against deterministic recurrent neural network (RNN) and long short-term memory (LSTM) baselines. All models are trained and evaluated on seeing measurements from the Maunakea summit. We assess the models using both deterministic metrics, including the root mean square error and the Pearson correlation coefficient, and probabilistic metrics that evaluate the quality and calibration of predictive uncertainty estimates. The results show that deep learning approaches outperform standard recurrent networks for short-term forecasting, with LSTM and FloTS achieving the best overall predictive accuracy. While Gaussian Processes provide a useful probabilistic baseline, their predictive distributions are limited by the Gaussian assumption. FloTS combines competitive forecasting accuracy with flexible uncertainty quantification and produces the most informative probabilistic forecasts.

Comment
9 pages, 2 figures, Proceedings of the SPIE 2026 Astronomical Telescopes + Instrumentation, Copehagen, July 2026
Journal ref
Proc. of SPIE Vol. 14152, 141520J (2026)
DOI
10.1117/12.3103367

Closest papers

Nearest neighbours by meaning across the whole index.

Gaussian Process Decorrelation for Spatiotemporal Deep Learning-Based Snow Water Equivalent Prediction

cs.LG stat.ME ·2026-08-27 ·Colin Fenster, Adrienne Marshall, Soutir Bandyopadhyay, Daniel McKenzie
similarity 0.69
Abstract

In the Western United States, snowmelt is essential to the agricultural industry in addition to being a key source of municipal drinking water. Consequently, accurate snowpack forecasting is critical for water policy and management. Automated Snow Telemetry (SNOTEL) stations provide accurate daily measurements of snow water equivalent (SWE) that exhibit strong correlations in space and in time. We tackle the problem of predicting future SWE values across the SNOTEL network. Specifically, we use a Gaussian Process-based linear transformation to remove spatial correlations before training a long short-term memory (LSTM) neural network on the decorrelated SWE data. This approach allows the LSTM to learn a clean temporal signal at each station. We show that this separation of spatial and temporal components yields better predictive success than multiple baseline models. Furthermore, we incorporate conformal prediction to quantify uncertainty in the resulting SWE forecasts, providing a distribution-free approach to illustrate a potential framework for establishing predictive intervals for spatiotemporal data. Together, accurate point forecasts and distribution-free uncertainty quantification provide a framework for SWE accumulation forecasting on subseasonal scales or projecting SWE with future data while motivating and supporting future work in predicting a large-scale, spatiotemporally complete SWE map.

0.19

Two-stage residual learning for calibrated uncertainty in solar irradiance forecasts

cs.LG eess.SY ·2026-11-05 ·RR Flanagan, E Kasha, PW Talbot, AS Epiny, MR Deinert
similarity 0.66
Abstract

Accurate forecasting of renewable generation requires not only point predictions but also reliable estimates of the uncertainty attached to them, particularly at extended horizons. Machine learning models are increasingly used to predict solar output, but they typically report a single value and capture the associated uncertainty poorly. The standard remedy is to train a model to predict the parameters of an assumed probability distribution directly. Here we develop an alternative two-stage method, motivated by synthetic solar signal generation, in which the forecast error is modeled separately from the expected irradiance. A first long short-term memory (LSTM) network predicts the expected global horizontal irradiance, and a second network models the distribution of the forecast error, which is added back to the mean to give both the forecast and its uncertainty. Using historical weather data from the National Solar Radiation Database and a probability integral transform evaluation, we show that explicitly modeling the error brings the predicted standard deviation markedly closer to the historical value, reducing the deviation between the two by 34.6 percent for a standalone LSTM and by 56.5 percent when the LSTM is coupled to Gaussian process regression, the most reliable of the configurations tested. Because the method models the total forecast error directly, it captures the full predictive uncertainty that matters for decisions, rather than only the model contribution that dropout-based approaches estimate.

0.07

SDECast: Probabilistic Weather Forecasting in Continuous Time with Neural SDEs

cs.LG stat.MLphysics.ao-ph ·2026-10-02 ·Maria Marchenko, Martin Andrae, Fredrik Lindsten, Christian A. Naesseth
top venue similarity 0.64
Abstract · Accepted to *AI for Stochastic Dynamics* & *Sim2Science* workshops at NeurIPS 2026

Existing machine learning weather forecasting models typically generate forecasts through autoregressive rollouts at a fixed temporal resolution. While highly efficient for long-range prediction, this formulation can suffer from severe error accumulation when used with shorter time steps and does not explicitly encode the locality and temporal continuity of atmospheric dynamics. To address these limitations, we introduce **SDECast**, a Neural Stochastic Differential Equation (SDE) framework for continuous-time probabilistic weather forecasting. SDECast extends SDE Matching to learn stochastic dynamics directly in physical space, without requiring repeated SDE simulation during training. On a simulated geophysical flow, we show that SDECast recovers meaningful drift dynamics and faithfully reproduces the underlying continuous-time behavior. We then demonstrate its scalability to global weather forecasting at hourly resolution, where SDECast produces skillful probabilistic forecasts for lead times of up to five days.

0.36

Climate Temporal Temperature Prediction via an Adaptive Stacked Bidirectional Residual Mamba Network

cs.LG ·2026-11-12 ·Yongxiang Lei
similarity 0.63
Abstract

Accurate temperatures forecasting relies on traditional meteorological parameters that are essential for monitoring weather informatics and guiding forecasting efforts. This study presents Deep DualMamba, a novel deep learning framework for temporal temperature prediction, built upon an enhanced variant of the recently proposed Mamba network. Unlike conventional Transformer-based architectures, which dominate foundation models but suffer from computational inefficiencies on long sequences, Deep DualMamba leverages selective structured state space models (SSMs) to achieve both scalability and performance. We identify a core limitation of prior subquadratic-time models-namely, their inability to perform long-term and robust forecasting and introduce two key innovations. First, we propose an adaptive bidirectional Mamba as encoders to learn complex and efficient features. Second, we develop a hardware-aware, parallel recurrent algorithm to compensate for the loss of convolutional efficiency, ensuring practical viability at scale. Last, the Deep DualMamba's approximation ability and convergence is theoretical analyzed. These design choices culminate in a simplified, attention- and MLP-free neural architecture that exhibits linear complexity with sequence length and significantly improved inference throughput. To demonstrate its applicability in climate informatics, we implement a full forecasting pipeline using historical meteorological data from UK monitoring stations.

0.15

PROSWIN: Probabilistic Solar Wind Speed Forecasting Using Deep Distributional Regression From Solar Images

cs.LG astro-ph.SRphysics.data-anphysics.space-ph ·2026-09-22 ·Daniel Collin, Yuri Shprits, Luca Chiarabini, Stefan J. Hofmeister, Nadja Klein +1 more
published similarity 0.63
Abstract

Accurately predicting fast solar wind conditions is challenging, as uncertainties are large and unquantified by traditional single-value prediction models. In particular, the risks of high-speed solar wind streams (HSSs), which can cause damage to technological infrastructure, cannot be reliably assessed without probabilistic forecasts. We present PROSWIN, a probabilistic machine learning model that forecasts the hourly solar wind speed (SWS) at Earth with a four-day lead time. The approach combines solar images and magnetograms using a deep neural network coupled to a distributional regression algorithm. Because standard error metrics underweight the relevance of HSS peaks, we further introduce the prediction score, a model-selection metric that jointly rewards timeline and HSS peak accuracy. On 14 years of data, our forecast achieves very well-calibrated uncertainties (<1% average deviation). Using the continuous ranked probability score (CRPS), a metric that assesses distributional accuracy, we obtain a timeline CRPS of 41.0 km/s, an HSS peak CRPS of 45.3 km/s, and a prediction score of 42.3 km/s. We find that the 171 Å channel is an important complement to the typically used 193 Å and 211 Å channels and that the prediction score for model selection improves the applicability of the model. Compared to selected models from the literature, ours is the only one that is accurate for both timeline and HSS peak values, rather than trading one off against the other. These results support the advantages of probabilistic over single-value solar wind models. The introduced methods are also transferable to other forecasting problems.

0.57

Skillful Data-Driven Subseasonal Soil Moisture Forecasting: Prospects and Limits for Flash Drought Prediction

cs.LG physics.ao-ph ·2026-10-05 ·Noelia Otero, Atahan Özer, Miguel-Ángel Fernández-Torres, Jackie Ma
published similarity 0.62
Abstract · 27 pages, 8 figures, 5 tables. Accepted for publication in npj Hydrosphere. Supplementary information available with the published version

Despite substantial progress in short-to-medium-range weather forecasting, predicting high-impact events such as flash droughts remains a key challenge for both early warning operations and physically-based subseasonal-to-seasonal (S2S) prediction systems. Here we demonstrate that, for S2S soil-moisture forecasting over Europe, forecast skill depends as much on how the prediction problem is formulated as on the forecasting model itself. Using a Vision Transformer-based architecture with dual-pathway temporal and spatial attention, we show that residual learning is essential to outperform persistence. This advantage is realized only when forecasting root-zone soil moisture in physical units rather than standardized anomalies, revealing that the target representation itself constrains predictability. A probabilistic extension via quantile-head fine-tuning further provides well-calibrated predictive distributions. Benchmarked against deep-learning and operational ECMWF S2S baselines over 2021-2022, our model achieves the highest deterministic and probabilistic skill at all lead times and reliably detects anomalously dry root-zone states (below the 20th percentile). Yet flash drought onset, defined by multi-pentad intensification criteria, remains a fundamental challenge shared across all current S2S systems. These findings advance data-driven S2S soil-moisture forecasting while highlighting the remaining challenge of predicting rapid drought development.

0.19

Multi-site Learning for Forecasting Day-Ahead Wind-Power Production for Onshore Wind Farms in Great Britain with Deep Neural Networks

cs.LG ·2026-10-18 ·James Kean, Aidan O'Sullivan
similarity 0.61
Abstract

Accurate day-ahead wind-power production forecasting is increasingly important for energy system operations, market activity and balancing Great Britain's grid as wind generation grows. In this study, we train multi-site deep neural networks (DNNs) using pooled historical production and reanalysis weather data from 88 onshore wind Balancing Mechanism Units (BMUs). Forecasts are produced at 09:00 UK time for the following midnight-to-midnight delivery day, matching NESO's operational day-ahead forecasting schedule. We use reanalysis weather as future covariates during training and numerical weather prediction (NWP) forecasts during inference, extending training beyond the available historical NWP archive. Our multi-site recurrent neural network outperforms site-specific RNNs and a multi-site XGBoost model, while processing the full future-weather trajectory for the forecast horizon further improves accuracy relative to hour-aligned approaches. Our best model, an RNN with a bidirectional weather encoder (BiRNN) for NWP trajectory processing, reduces MW mean absolute error and root mean squared error relative to NESO by 17.94% and 12.51%, respectively, across 638,246 matched BMU-hour observations under conditions approximating unconstrained wind farm operations. In 40 leave-one-out trials, the BiRNN achieves lower MAE than NESO at 32 excluded sites, indicating transferable forecasting ability despite reduced accuracy when sites are omitted from training. Our capacity-estimation, data-cleaning and reanalysis-based training approach provides a basis for scaling NWP-aware multi-site forecasting to additional wind farms, longer production histories, and increased forecast accuracy beyond that seen in this study.

0.08

Uncertainty-Aware End-to-End AI Weather Forecasting: Disentangling Observation and Model Contributions

cs.LG physics.ao-ph ·2026-08-31 ·Rodrigo Almeida, Noelia Otero, Jost Arndt, Simon Baur, Wojciech Samek +1 more
similarity 0.61
Abstract · 25 pages, 16 figures

End-to-end weather forecasting systems produce skillful global gridded and station forecasts directly from raw Earth observations, replacing the numerical weather prediction pipeline, including data assimilation, at a fraction of its cost. These systems are deterministic and issue no uncertainty. Here we render the Aardvark Weather model probabilistic by attaching one stochastic mechanism to each component: learned, input-dependent noise at the observation encoder, aimed at the uncertainty inherited from the observing system, and Monte Carlo dropout in the processor, aimed at the uncertainty of the learned dynamics. The resulting nested ensemble attributes forecast spread to the two sources through a law-of-total-variance decomposition, cross-checked by withholding observation streams. Probabilistic finetuning significantly improves the mean forecast, by 11% on average across variables and lead times. The ensemble is close to calibrated against ERA5 through the medium range, is at least as accurate as the deterministic model at weather stations while beating it in relative economic value at every lead time, and remains considerably behind the operational ECMWF IFS ensemble. The encoder branch behaves as observation-driven uncertainty. Component-attributed uncertainty makes end-to-end forecasts more transparent, a step toward observation-driven digital twins of the atmosphere.

0.22