CS Paper Radar

Do LLM-derived text features improve electricity price forecasts beyond document activity? Evidence from Australia's National Electricity Market

Fanxing Ning, Jianguo Zhu

eess.SY ·submitted 2026-10-23 09:06 UTC · v1
code/data

Abstract

Large language models (LLMs) can turn market notices and news into features for electricity price forecasting. Such features also record when documents arrive and whether their extraction failed, so a gain measured against a model without text can reflect document activity as well as content. We add the counts of the same documents and of their failed extractions to a no-text LightGBM model and measure each LLM representation against this activity reference, which needs no additional LLM requests. With gpt-6-luna as the extractor, we forecast Australian electricity prices from operator notices (5-30 minutes ahead) and WattClarity news (day-ahead) after the extractor's documented knowledge cutoff, averaging forecasts over three training seeds. For news, the counts alone lower mean absolute error (MAE) by 3.0%, most of the 3.6% reduction that the better LLM representation achieves relative to the no-text model; its increment beyond the counts is not significant. For notices, two of the three LLM representations lower MAE beyond the counts, by 0.14-0.21% and 0.45-0.52%, whether the counts are cumulative or cover recent windows, whereas the third shows no significant reduction. Time-shifted placebos give mixed evidence on whether these gains require text aligned with the target. Benchmarks using the operator's pre-dispatch forecasts (notices) or price history alone (news) are more accurate than all these text models. Gains against the no-text model and the activity reference differ in sign at individual seeds and, for notices with cumulative counts, in seed-averaged forecasts. Text gains should be reported against both references.

Comment
27 pages, 4 figures, 12 tables. Code and data: https://github.com/ningfanxing-sys/nem-llm-text-activity

Closest papers

Nearest neighbours by meaning across the whole index.

Large Language Models for Forecasting of Electricity Price Spikes

eess.SY cs.AI ·2026-11-03 ·Liang Lin, Jianguo Zhu, Gang Lei, Allen Wang, Xin Lu
similarity 0.81
Abstract · 31 pages, 7 figures

Can market notices read by a large language model (LLM) improve short-horizon electricity price spike forecasts? We study five regions of Australia's National Electricity Market. A local LLM extracted features once per notice, only notices visible at the forecast origin were used, and a gradient-boosting model produced onset probabilities. In an out-of-sample development back-test (July 2022 to July 2024), notice features did not lower the 60-minute Brier score of a numeric model including the market operator's next-hour forecasts. At 24 hours they did, but rule-based features did so almost equally and attribution to notice content was not established. A locally frozen protocol, not publicly deposited, governed a retrospective single-use holdout from 15 August to 17 October 2026. At 60 minutes, the hypothesis that LLM features lower the numeric model's Brier score was not established (H1; relative Brier skill -0.09%), so the planned comparison with rule-based features was not tested (H2). At 24 hours, the hypothesis that LLM features lower the score against a text-free control was not established (H3; +0.37%, 95% CI [-9.74, 1.39], inconclusive). By the frozen wording rule the non-significant 24-hour improvement comes mainly from recalibration and cannot be attributed to notice content, although post hoc the decomposition could not separate recalibration from resolution. The 24-hour base rate was 1.56% on the holdout against 14.0% in development. With 5 onsets on 3 days and 24-hour positives in 2 of 22 three-day blocks, the holdout neither established nor excluded a 24-hour effect.

0.24

TRACE: A Reproducible Benchmark for Electricity Price Forecasting with Official Operational Text

cs.LG stat.ML ·2026-09-30 ·Xinyi Yi, Zhangdie Yuan, Ioannis Lestas
publishedbenchmark similarity 0.66
Abstract · Accepted to the NeurIPS 2026 Workshop on Foundation Models for Time Series (FMTS)

Electricity price forecasting (EPF) supports scheduling, bidding, and risk management in electricity markets, yet existing benchmarks focus mainly on numerical inputs, leaving the forecasting value of forecast-time textual context insufficiently evaluated. We introduce TRACE, a reproducible benchmark of 7,300 zone--day instances pairing prices from five zones in a major U.S. market with official operational text available at the forecast cutoff. TRACE reconstructs official operational text at each cutoff, preventing post-cutoff information leakage. We evaluate TRACE for semantic alignment and forecasting value. Semantic assessments align with central movement and both tail risks in ground-truth prices, most consistently for upper-tail price risk. Forecasting value is reflected in a median 7.4\% reduction in upper-tail pinball loss across time-series foundation models. A controlled cross-day text-mismatch ablation reverses the gains, falling below the no-text baseline.

0.47

Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts

cs.LG ·2026-09-27 ·Cheng Peng, Ruixi Luo, Zhi Chen, Wei Tang
similarity 0.61
Abstract

Large language models (LLMs) are increasingly used to forecast real-world events, but access to the strongest individual forecaster may be costly or otherwise constrained. We study weak-to-strong forecast aggregation: can individually weaker LLM forecasters be aggregated to outperform a stronger forecaster? Using ForecastBench (Karger et al., 2025), we evaluate 70 LLM forecasters across 16 comparison groups, each with more than 1,000 shared subquestions, yielding 1,121 weaker-model pairs. Within each group, we identify the strongest individual by test Brier score and evaluate aggregates composed exclusively of weaker forecasters, with aggregation weights learned on separate training data. We find substantial evidence of weak-to-strong improvement. Learned linear pooling identifies a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16 groups. We also find that these improvements do not rely on having a near-best constituent and are generally accompanied by good calibration. Additional analyses show that adding more models does not consistently improve performance, and competitive weaker-model aggregates also remain available under practical constraints.

0.41

StatPrompt-LLM: A Lightweight Statistical Refinement for Foundation-Style Time-Series Forecasting

cs.LG ·2026-11-08 ·Aniket Sinha, Veronica W. Tsou
similarity 0.61
Abstract · 10 pages, 1 figure, 7 tables

Time-series forecasting has recently seen a paradigm shift with the introduction of foundation-style models that adapt large pretrained language architectures to sequential data. However, effectively exposing fundamental temporal structures, such as trend, scale, and temporal dependence, to these models remains an open challenge, particularly when the available observations are limited. In this paper, we conduct an exploratory empirical study across different generations of forecasting architectures and propose StatPrompt-LLM, a lightweight statistical refinement of Time-LLM. StatPrompt-LLM augments the existing Time-LLM prompting mechanism with dataset-level statistical information and introduces a zero-initialised autoregressive bypass as a complementary linear forecasting pathway. We evaluate our framework on electricity and meteorological time-series benchmarks, including ETTh1, ETTh2, and Weather, under full-prefix and reduced-prefix configurations. Under the 10% reduced-prefix configuration, StatPrompt-LLM achieved lower in-sample prediction errors than Time-LLM, with unweighted average relative reductions across datasets of 40.2% in MSE and 21.4% in MAE. Full-prefix results were more dataset-dependent. Ablation analyses revealed relatively modest and dataset-dependent changes when selected additional statistical descriptor groups were masked within the combined architecture. Computational profiling indicated additional model parameters and runtime overhead relative to Time-LLM.

0.29

TimeOmni-o1: Bidirectionally Coupling Time Series Foundation Models and LLMs for Multimodal Reasoning and Forecasting

cs.CL ·2026-10-14 ·Ziyang Zhang, Shenyi Li, Wen Wu, Chao Zhang
code/data similarity 0.60
Abstract · Code: https://github.com/OpenTSLab/TimeOmni-o1 Project page: https://windowso.github.io/TimeOmni-o1-web/

Time series from real-world systems are often accompanied by text, such as a report on an event that affects them. Two common tasks on these series are understanding, e.g., answering questions about a series, and forecasting its future values. Both tasks require reasoning over the series and the text together. Moreover, understanding outputs text while forecasting outputs a time series, so serving both requires a model that reads and generates both modalities. Large language models (LLMs) read time series as digit tokens and handle long series poorly. Time series foundation models (TSFMs) forecast well from large-scale pretraining but typically neither read nor generate text. Time series language models add a time series modality to the LLM but forecast on the LLM side, leaving the TSFM's pretrained forecasting ability unused. We present TimeOmni-o1, which couples a pretrained TSFM and an LLM bidirectionally. In one direction, the TSFM encodes the series in compressed form for the LLM, which reasons over it together with the text. In the other, the LLM's reasoning is fed back into the TSFM, which produces the forecast conditioned on it. One model architecture thus reads and generates both series and text, reasons in text, and draws on the TSFM's pretrained forecasting ability. To train it, we build MMTR, a multimodal time series reasoning dataset with annotated reasoning traces, covering both tasks across 12 domains with 500k+ samples. On MMTR, TimeOmni-o1 outperforms existing models on both understanding and forecasting across most domains, as well as the LLM and the TSFM inside it when used individually. The gains carry over to three open benchmarks, where it outperforms the dedicated time series models.

0.35

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications

cs.AI ·2026-08-24 ·Xiaogang Xu, Jiaqi Tang, Jianmin Chen, Yingying Yan, Zhenchao Tang +10 more
benchmark similarity 0.60
Abstract

Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tools, and iterative prediction. We investigate LLM-based forecasting agents, meaning systems in which a language model contributes to a scored prediction about a future or currently unobserved target. We organize architectures into three groups. Standalone LLM workflows operate on encoded time series or event context. Tool- and retrieval-augmented agents incorporate external evidence. Hybrid systems pair LLMs with statistical or foundation models. We then review training methods and evaluation protocols. We examine negative as well as positive evidence, including sensitivity to small input perturbations, ablations in which the LLM component does not improve accuracy, and benchmark gains that may reflect contamination instead of temporal reasoning. We cover applications in finance, weather, health, energy, and operations, and we summarize the benchmarks and datasets used for evaluation. The evidence indicates that measurement is a central limitation. Future work requires calibration under distribution shift, contamination-resistant live evaluation, explicit reporting of cost and accuracy together, and methods for handling feedback between deployed forecasts and the outcomes being forecast.

0.41

Can Large Language Models Forecast What Researchers Study Next?

cs.CL ·2026-09-01 ·Fenghai Li, Zihan Tang, Haofei Yu, Yining Zhao, Jiaxuan You
top venuebenchmark similarity 0.60
Abstract · 31 pages, 4 figures. Accepted to EMNLP 2026

Large language models increasingly generate research ideas, yet judging their novelty or feasibility at generation time does not establish whether they anticipate subsequent work. We introduce IdeaForecastBench to evaluate research idea forecasting. Given a community's literature up to a cutoff, a system produces up to five ranked ideas, which are evaluated against later papers. The benchmark comprises 624 rolling episodes across 52 topics, with a fixed retrieve-then-judge protocol and separately reported results from two judges. We compare five history-compression strategies across GPT-4.1, Qwen2.5-7B/14B, and Qwen3.5-9B, together with a learned Mode-Decomposition Forecaster (MDF). Under the primary GPT-4.1-mini judge, Summary improves on Direct in Hit@5 and Precision@5 across all four backbones. Qwen2.5 scores above GPT-4.1, whereas Qwen3.5 scores below it. An outcome-blind assessment finds that Qwen2.5 produces broader forecasts, but does not identify how much breadth contributes to its advantage. Threshold and judge diagnostics further clarify the limits of interpreting realization as precise anticipation. IdeaForecastBench provides a common task for studying which research ideas a community subsequently pursues and how reliably this outcome can be measured.

0.64

Reading or Remembering? Measuring Temporal Contamination in Language Models

cs.LG cs.CL ·2026-11-16 ·Arunkumar V, Gangadharan G. R., S. Senthilkumar
similarity 0.59
Abstract · 17 pages

Language models are now used to read central-bank text for economic forecasting, but most were trained on text written after the documents they read. A model reading a 2021 FOMC statement may already know that inflation rose in 2022, so part of its apparent skill can be memory of the outcome, a temporal form of test-set contamination. We measure this memory on FOMC statements and minutes from 2000 to 2026 by comparing modern open models with chronologically consistent models, which are trained only on text from before a fixed date. Asked about past GDP outcomes given only a date, models that cannot know the answer still reach an AUC of up to 0.64 from priors on dates and eras, so chance is the wrong baseline. Against these memory-free models on the same questions, most modern models recall which years had strong growth. Membership-inference scores sort documents by era and document type rather than by whether a model saw them, as exact counts in an open pretraining corpus show. In real-time nowcasts, every lookahead premium we estimate on US GDP is smaller than the forecast comparison can detect at macroeconomic sample sizes, and none on GDP or inflation survives correction for multiple testing. Lookahead audits should therefore measure memory directly against memory-free models on the same items.

0.28