CS Paper Radar

When Do Language Models Add Value? Relative Competence in Hybrid Event Forecasting

Aditi Tiwari, Aashrith Bandaru, Heng Ji

cs.AI ·submitted 2026-09-10 18:28 UTC · v2

Abstract

When a forecasting system already has a prediction from a market, crowd, or statistical model, when should it use a language model? Standalone accuracy cannot answer this question because a less accurate model may still improve the existing forecast. We study this distinction through relative competence, separating a model's replacement advantage from its complementary value. Under Brier loss (squared error between predicted probabilities and binary outcomes), we characterize when model disagreement can correct external forecast errors and derive the population gain from regime-specific affine pooling. We introduce a competence gate that estimates source weights from resolved outcomes, shrinks regime-specific estimates toward a global solution, and chronologically recalibrates the pooled forecast. Across 4,974 resolved binary questions and four language models, the best standalone model has 24.94% higher Brier loss than the calibrated external forecast, yet the gate improves that reference by 3.35% (95% CI, 1.80 to 4.90%). Model contributions vary across forecasting regimes, with larger corrections in thinner markets and forecasting crowds and smaller corrections in higher-participation settings. The gate also shows no statistically detectable performance difference from more flexible contextual aggregators. These results demonstrate that standalone superiority is not necessary for complementary predictive value and that language models should be evaluated relative to the forecasts already available.

Comment
18 pages, including references and appendices

Closest papers

Nearest neighbours by meaning across the whole index.

Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts

cs.LG ·2026-09-27 ·Cheng Peng, Ruixi Luo, Zhi Chen, Wei Tang
similarity 0.79
Abstract

Large language models (LLMs) are increasingly used to forecast real-world events, but access to the strongest individual forecaster may be costly or otherwise constrained. We study weak-to-strong forecast aggregation: can individually weaker LLM forecasters be aggregated to outperform a stronger forecaster? Using ForecastBench (Karger et al., 2025), we evaluate 70 LLM forecasters across 16 comparison groups, each with more than 1,000 shared subquestions, yielding 1,121 weaker-model pairs. Within each group, we identify the strongest individual by test Brier score and evaluate aggregates composed exclusively of weaker forecasters, with aggregation weights learned on separate training data. We find substantial evidence of weak-to-strong improvement. Learned linear pooling identifies a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16 groups. We also find that these improvements do not rely on having a near-best constituent and are generally accompanied by good calibration. Additional analyses show that adding more models does not consistently improve performance, and competitive weaker-model aggregates also remain available under practical constraints.

0.40

Logically Consistent Accurate Forecasts

cs.LG cs.AI ·2026-11-13 ·Kate Yuan, Rohan Alur, Manish Raghavan, Devavrat Shah
publishedposition similarity 0.69
Abstract · NeurIPS 2026 Workshop on Foundation Models for Temporal Systems (FMTS)

Agentic forecasting systems based on large language models (LLMs) are now competitive with expert human forecasters, but these systems still suffer from a critical limitation: the inability to produce forecasts that are logically consistent. For example, forecasts for related events often fail to satisfy the axioms of probability, which in turn implies that related forecasts cannot simultaneously be correct and undermines their utility for decision makers. Enforcing consistency is not trivial -- naive approaches (e.g., projection onto the set of consistent forecasts) can degrade accuracy. We argue that the key to overcoming the apparent tradeoff between consistency and accuracy lies in the reasoning which supports each forecast. We finetune a supervisory language model that leverages this reasoning to jointly reconcile inconsistencies while improving the accuracy of related forecasts. On ForecastBench, our method improves the Brier score of a retrieval-augmented GPT-5.1 forecaster by 9.3% while reducing measures of structural inconsistency by 82-90%. We interrogate our method via a comprehensive series of baselines and ablations, and find that no alternative method matches our gains in consistency and accuracy simultaneously.

0.17

Can LLMs Predict the Future? A Brier Score Analysis of Prediction Markets

cs.AI ·2026-09-26 ·Yuanbo Li, Zekun Li, Xiaoyan cong
benchmark similarity 0.68
Abstract

We study whether model upgrades improve probability estimates for prediction-market questions. Our Resolved Market Forecasting (RMF) benchmark contains 3,000 resolved binary questions across nine domains, on which we evaluate six Claude and Qwen model variants using a question-only, zero-shot protocol. We assess Brier scores relative to an empirical base-rate predictor, examine their Murphy decomposition, and compare models through paired differences, with results stratified by event category and timing relative to training cutoffs. In the reported post-cutoff stratum, the four Claude models achieve Brier scores of 0.183-0.192, improving on the base-rate reference by 0.024-0.033. Qwen 32B does not significantly outperform that reference, although its paired Brier is 0.024 lower than that of the 7B checkpoint. The evaluated Claude version and tier upgrades yield no significant improvement. Within-model differences across event categories exceed the observed differences among Claude variants. These results show why forecasting scores should be interpreted alongside simple probability baselines and question composition: under this protocol, newer versions or higher model tiers do not consistently produce more accurate probabilities.

0.28

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

cs.AI cs.LG ·2026-09-24 ·Liqin Ye, Haorui Wang, Fardin Ahmed, Rongzhi Zhang, Yuan He +6 more
similarity 0.67
Abstract

Forecasting requires large language model (LLM) agents to gather evolving evidence, revise their beliefs, and make calibrated predictions over time. Yet evaluating and improving these capabilities is challenging because real-world events unfold continuously and outcomes may take months to resolve. We introduce Forecast-Dojo, a replayable environment that reconstructs resolved prediction-market events as sequences of dated forecasting steps, enabling reproducible evaluation, longitudinal analysis, and training from agent interactions. It comprises 1,568 Polymarket events split temporally into training and evaluation sets, alongside 18.8 million dated news articles. Across 12 models, research tools consistently improve forecasting performance, and greater capture of new evidence is associated with larger improvements over time. Nevertheless, all models trail historical market forecasts, while carrying a belief notebook across dates reduces research costs without consistently improving forecast quality. We supervise fine-tuned Qwen3-30B-A3B-Thinking-2507 on collected trajectories from Qwen3-235B-A22B-Thinking-2507 to demonstrate the environment's training utility. The student's accuracy is improved from 34.8% to 42.8% on the evaluation set.

0.38

Can Language Models Learn to Forecast Stock Prices

cs.CL ·2026-09-29 ·Jiacheng Guo, Suozhi Huang, Shuzhen Li, Yunlong Gao, Zerui Cheng +9 more
benchmark similarity 0.67
Abstract · 18 pages, 4 figures

Post-training has been shown to significantly improve language models' performance on tasks with verifiable outcomes, including mathematical reasoning, software engineering, and computer use. However, whether the same approach can improve forecasting in financial markets is much less clear. Compared with tasks with verifiable outcomes, not only are realized returns noisy, but even what constitutes a relevant information set for making effective predictions is not obvious a priori: the model must decide which observations to gather and then commit to a numerical judgment before the outcome is known. We study this question in a chronological stock-price sandbox, where a language model gathers price, volume, relative-performance, and market-context evidence and predicts a future return. We post-train Qwen3-4B with supervised fine-tuning (SFT) on tool-use demonstrations, then proximal policy optimization (PPO) with a terminal reward given by the forecast score against the realized return. The resulting AURA-4B more than doubles the starting direction--magnitude score, from 20.94 to 43.31, and is comparable to frontier language models on this benchmark. Conditional magnitude agreement rises from 33.3 to 66.2, while directional accuracy changes from 62.9 to 65.4. SFT expands tool use, and PPO further increases the share of ranking and market-context queries. These results show that post-training can substantially improve financial forecasting performance, together with changes in how the model investigates the market, on this outcome-selected benchmark.

0.27

Approximate Recall, Approximate Forecasts: Recall as a Diagnostic for LLM Forecasting Errors

cs.AI cs.CEcs.LG ·2026-11-07 ·Prashanth Bhaskara, Shubhaankar Gupta, Seojoon Yeon
published similarity 0.67
Abstract · 16 pages, 7 figures (including appendix). Accepted as a Spotlight at the ICML 2026 Workshop on Forecasting as a New Frontier of Intelligence

Large Language Models (LLMs) exhibit approximate recall: they retain coarse associations about past events while losing fidelity on specific details. Past research on LLMs has quantified different models' forecasting ability in real-world events using prediction markets, but has not studied how losing said fidelity impacts their forecasting abilities. For example: while an LLM correctly recalls that the answer to: ''Did Anna Sawai win the Best Drama Actress in the 76th Emmy Awards?'' is ''Yes,'' it incorrectly states the event date as September 22 rather than September 15. In this paper, we introduce a novel metric called the Recall Fidelity Score to study how imprecision in internalized knowledge propagates to forecasting capability on future events. On resolved historical markets, we find that higher local recall fidelity around semantically similar past events predicts lower forecasting error on resolved markets, and the same signal predicts smaller distance from open markets prices.

0.23

CivWorldBench: Evaluating Language World Models in Structured Multi-Agent Sandbox Worlds

cs.CL ·2026-10-12 ·Ang Li, Haofei Yu, Zhengwei Tao, Liang Chen, Kuan Li +5 more
benchmark similarity 0.66
Abstract

The growing use of language models for forecasting motivates a broader evaluation of their world-model capabilities. Such evaluation must go beyond predictive accuracy to assess hidden-state inference and intervention prediction. Real-world forecasting benchmarks offer limited access to hidden-state ground truth and controlled experiments. We introduce CivWorldBench, a benchmark built on a multi-agent strategy game engine with recorded internal states, controlled observations, and paired intervention rollouts. Its 1,900 decision points from 475 worlds support three task levels: association forecasting, belief inference, and intervention prediction. Reference predictors support attributable evaluation of capability gaps, with an expert harness and empirical ceiling estimates contextualizing scores. Stored trajectories make intermediate predictions verifiable, while agent-play experiments test whether world-model capabilities support decision-making. The main panel comprises eleven frontier language models. Composite benchmark scores correlate with real-world forecasting leaderboards on matched model panels, more closely than with a general-capability index, supporting its use as a proxy for real-world forecasting capability. Strong association forecasts do not consistently extend to belief inference or intervention prediction, and effect magnitudes remain poorly calibrated despite informative direction predictions. Process verification reveals errors missed by final-answer evaluation, and structured reasoning improves intervention prediction for the strongest model in the process study. Agent-play experiments identify useful assisted plans and motivate evaluating decision-relevant capabilities beyond association forecasting. These findings identify hidden-state inference, intermediate-state prediction, and effect calibration as priorities for world-model training.

0.21

Evaluating Token Probabilities as Confidence Estimators in Human Behavior Simulations: A Case Study in Luxembourgish Next Like Prediction

cs.CL cs.AI ·2026-11-05 ·Nils Schwager, Simon Münker, Alistair Plum, Achim Rettinger
position similarity 0.66
Abstract · 12 pages, 1 figure, 2 tables. Accepted to 11th Workshop on Natural User-generated Text (W-NUT) at EMNLP 2026, Budapest, Hungary

As large language models (LLMs) are increasingly used to simulate human behavior, their operational validity is typically assessed through accuracy alone, implicitly treating human behavior as deterministic. We argue that on noisy digital trace data, a model's performance depends not only on predicting the correct outcome but on its calibration. We present a case study testing whether the token probabilities of open-weight LLMs can serve as confidence estimates in a Next Like Prediction task: predicting how a user of the Luxembourgish news platform RTL.lu rates a comment, conditioned on their prior interactions. Evaluating Ministral 3 (3B, 8B), we find that, with in-context learning only, the tested models are overconfident and are outperformed on accuracy and calibration by a simple statistical baseline. Accuracy and calibration do not move together: scaling from 3B to 8B improves accuracy while degrading calibration, and longer behavioral histories raise accuracy while confidence declines. Task-specific supervised fine-tuning acts as an effective calibration intervention, reducing Expected Calibration Error from 0.2 to 0.006 while improving accuracy from 0.536 to 0.758. We conclude that confidence estimation should be evaluated alongside accuracy to improve validity when using LLM-based agents.

0.22