CS Paper Radar

Beyond Local Accuracy: Benchmarking the Rollout Reliability of PDE Foundation Models

Jaewan Park, Seid Koric, Giri Krishnan, Sudhakar Pamidighantam, Surresh Marru

cs.LG ·submitted 2026-10-22 21:32 UTC · v1

Abstract

PDE foundation models enable data-efficient adaptation but are deployed autoregressively, so short-horizon accuracy need not imply reliable simulation. We ask whether the short-horizon advantage of a foundation-model recipe persists when its predictions are repeatedly fed back as inputs for a long horizon. We define error amplification as G=E(H_f)/E(H_s), which measures full-horizon error relative to short-horizon error. We evaluate seven models on seven two-dimensional time-dependent PDE datasets: scratch-trained FNO and DeepONet; fine-tuned DPOT, GPhyT, Poseidon, and Walrus; and frozen in-context VICON. We report VRMSE, relative L_2, and relative MAE. Six datasets provide paired short- and full-horizon scores, while CE-RM contributes short-rollout sweeps only. At least one fine-tuned foundation model outperforms both scratch baselines in 26 of 30 short-rollout dataset--budget cells under every metric. Nevertheless, full-horizon VRMSE exceeds short-horizon VRMSE in 140 of 144 paired trained configurations, with a median error amplification of $2.04$. The count below VRMSE one falls from 104 to 53, and 12 full rollouts exceed ten although none does at the short horizon. Walrus achieves the lowest full-horizon VRMSE in 16 of 24 cells, yet its median error amplification is $2.70$. Relative L_2 and relative MAE show the same qualitative pattern. These results show that favorable short-horizon rankings can coexist with substantial error amplification. PDE foundation-model benchmarks should therefore report horizon-resolved absolute errors, amplification statistics, and failure incidence rather than a single local forecast score.

Closest papers

Nearest neighbours by meaning across the whole index.

Frozen in Time: How a Physics Foundation Model Forgets Under Sim-to-Real Adaptation

cs.LG physics.flu-dyn ·2026-10-29 ·Haaris Mian, Miles Cranmer, Alessandro Favero, Payel Mukhopadhyay
publishedcode/data similarity 0.66
Abstract · Accepted at the Sim2Science workshop, NeurIPS 2026. Code: https://github.com/haarisamian/walrus-forgetting-paper

Scientific foundation models (SciFMs) have achieved remarkable performance on a wide range of PDEs, including the ability to efficiently adapt to unseen data distributions with additional finetuning. As with all neural architectures, finetuning such models comes with the risk of catastrophic forgetting, whereby adapting to new tasks degrades prior knowledge of the model. We study catastrophic forgetting in Walrus-1.3B, a SciFM for continuum dynamics, over the course of a two-stage sim-to-real adaptation: first finetuning on simulated flow around a cylinder and then adapting to experimental data of the same system. We find that finetuning on the simulations rapidly degrades rollout performance on the original pretraining datasets, with one-step errors increasing up to 35x after just 20 epochs. Subsequent adaptation to real data further erodes performance on the simulated task. We characterize the potential drivers of forgetting, such as dataset dependence and task representation similarity. We then show that replaying as little as 1% simulated data during real-data finetuning recovers most of the lost simulation performance while retaining effective adaptation to the experimental dynamics. Our results identify catastrophic forgetting as a practical obstacle to sim-to-real adaptation of SciFMs and show that lightweight replay strategies significantly improve retention.

0.34

Distillation of Foundation Models for Time-dependent PDEs

cs.LG ·2026-08-12 ·Daniel Musekamp, Boshra Ariguib, Andrei Manolache, Mathias Niepert
similarity 0.66
Abstract

Foundation models for time-dependent partial differential equations (PDEs) are trained on large and diverse collections of physical systems and can generalize effectively to new downstream tasks. After fine-tuning on only a few trajectories from a target domain, they can achieve strong accuracy in low-data regimes. However, these models are typically large and computationally intensive, limiting their usefulness as fast surrogates for numerical solvers. We propose Teacher Rollout Extension (TREX), a knowledge distillation framework that transfers the predictive capability of a pretrained foundation model into a compact and efficient student. Starting from a fine-tuned teacher, TREX augments limited downstream data by generating long synthetic trajectories through teacher rollouts, optionally with periodic noise injection. This procedure samples from the teacher-induced rollout distribution without requiring explicit knowledge of the initial-condition distribution, while exposing the student to long-horizon states and local recovery behavior around states encountered during autoregressive prediction. The student can further incorporate task-specific inductive biases, such as equivariance, that the teacher does not necessarily enforce. We evaluate TREX on multiple PDE benchmarks. The resulting students can match or surpass the teacher's accuracy while reducing the number of parameters by several orders of magnitude and achieving more than an order-of-magnitude speedup in inference.

0.34

CasEm: A Cascade Architecture for Long-Horizon Neural Emulation

cs.LG ·2026-09-28 ·Zhaoyi Li, Jingtao Ding, Shihua Li
similarity 0.66
Abstract · 39 pages

Autoregressive neural emulators can drift or diverge over long rollouts despite accurate short-term predictions. We introduce Cascaded Emulation (CasEm), a one-way rollout architecture that augments an existing full-state backbone with an independently evolving model of physically specified aggregates. Its forecasts guide corrections to full-state predictions, without feedback from the backbone to the aggregate model. Effective guidance requires aggregates that cover substantial backbone error, remain accurately predictable, and support useful full-state corrections. We derive a finite-horizon error bound that clarifies these three factors and use empirical diagnostics to guide subsystem selection. Across four ODE/PDE benchmarks, CasEm reduces long-horizon rollout errors across diverse backbones and suppresses the trend toward error divergence in both diffusion tasks using Fourier neural operator backbones. In global climate emulation, CasEm with a regional total-water subsystem reduces 10-year full-state time-mean error by 66.6% and 46.3% for frozen ACE and Spherical DYffusion backbones, respectively, while adding less than 3% to inference time.

0.34

Into the ORBIT for Time Series: Training Regimes for Foundation Models

cs.LG cs.AI ·2026-08-13 ·Hongjie Xia, Yiding Liu, Yifan Hu, Peiyuan Liu, Zewei Dong
highly cited similarity 0.65
Abstract

Time series foundation models (TSFMs) have advanced primarily through architectural innovation, while training regimes for large-scale heterogeneous corpora remain under-explored. As a result, pre-training distributions are often poorly controlled with respect to domain imbalance, context requirements, prediction horizons, and missingness. We introduce ORBIT (Omni-Range Bootstrap Incremental Training), a training paradigm that makes this distribution explicit and controllable. ORBIT combines Bootstrap Multi-Level Sampling, which controls dataset exposure and samples records, target variables, context windows, and prediction horizons, with Omni-Range Incremental Training, which varies context lengths and prediction horizons throughout a single training stage. Under ORBIT, we train Falcon-2.0, a simple univariate encoder-only Transformer with missingness-aware triple-channel patch tokenization and parallel patch prediction. We further introduce Rank-Guided Cross-Depth Alignment, a training objective that uses late-layer representations as stop-gradient teachers for shallow layers without additional inference cost. Evaluations on GIFT-Eval and fev-bench demonstrate strong zero-shot forecasting performance across diverse domains and frequencies.

0.42

Beyond Compression: Training Latent Representations for Stable Long-Horizon Rollout in Neural Surrogate Solvers

cs.LG cond-mat.mtrl-scics.CE ·2026-09-24 ·Andreas E. Robertson, Ashley T. Lenau, John D. Shimanek, Benjamin A. Jasperson, Vivek Oommen +3 more
similarity 0.64
Abstract

Latent neural surrogate solvers, or latent dynamics models, accelerate simulations of time-dependent physical systems by evolving a compressed latent space rather than resolving full-resolution fields directly. In principle this reduces computational cost and simplifies learning, but in practice errors often accumulate rapidly during long autoregressive rollouts, limiting predictive utility. We show that this instability does not stem from the latent representation itself, but arises when it is trained solely for reconstruction, producing representations poorly suited to long-horizon forecasting. We systematically evaluate training-level interventions that align latent representations with long-horizon rollout: Koopman operator learning and Hamming noise injection during autoencoder training to improve compression, together with noise injection and multi-step rollout fine-tuning to improve dynamics. Interventions that improve long-horizon rollout stability often degrade conventional training metrics, including reconstruction and one-step prediction accuracy. Collectively, these interventions reduce long-rollout error by approximately 40\% and match or exceed the accuracy of full-resolution models on two physics benchmarks, while requiring 2 orders of magnitude fewer floating point operations and half the GPU memory. Applied to mesoscale crystal-plasticity simulations of high-cycle fatigue, the resulting surrogate achieves stable extrapolation over horizons orders of magnitude beyond those observed during training. More broadly, these results show that neural compression should be designed not merely to reduce dimensionality, but to restructure the solution space for stable dynamical evolution, a key requirement for reliable, efficient neural surrogates in scientific applications.

0.24

Stabilizing Autoregressive PDE Foundation Model Rollouts with Event-Triggered Context Healing

cs.CE ·2026-09-26 ·Jaewan Park, Jay Phil Yoo, Kazuma Kobayashi, Seid Koric, Syed Bahauddin Alam +3 more
similarity 0.64
Abstract

Autoregressive PDE foundation models enable fast full-field forecasting but accumulate errors as predicted states are recursively reused. We introduce Event-Triggered Context Healing (ETCH), which compares forecasts with measurements from 12 cylinder-surface pressure taps. When their discrepancy exceeds a threshold, pressure-conditioned hourglass diffusion transformers reconstruct the velocity--pressure context for subsequent predictions. On 43 RealPDEBench Cylinder trajectories, ETCH reduces velocity and pressure errors from 48.05 and 63.71% to 6.29 and 7.25% for DPOT, and from 74.97 and 76.91% to 5.16 and 6.84% for Poseidon-T, using the same reconstructors and threshold. For DPOT, correcting 14.56% of windows maintains errors close to reconstructing every window and supports average real-time forecasting and correction on an A100 GPU. These results show that sparse physical feedback stabilizes long autoregressive rollouts across PDE foundation models while reducing reconstruction cost.

0.27

PDE-OBS: Controlled Evaluation Across Observation Patterns

cs.LG physics.comp-ph ·2026-09-29 ·Ruichen Xu, Siyao Wang, Fang Wan, Jiacheng Qiu, Wenhan Gao +5 more
benchmarkcode/datahot direction similarity 0.62
Abstract · 57 pages, including supplementary material. Code: https://github.com/ru1ch3n/PDE-OBS. Author affiliation updated

Physical-field reconstruction and forecasting depend on both measurement density and spatial layout, yet evaluation under a single observation pattern does not characterize performance when that pattern changes. We introduce PDE-OBS, an integrated benchmarking platform spanning numerical data generation, model training, and inference and evaluation under varying observation conditions. It combines 560,000 fields and trajectories from seven partial differential equation families with configurable observation operators and seven adapted baseline methods for stationary reconstruction and short-horizon forecasting. Separating observation construction from physical records allows users to specify parameterized patterns and deterministic mixtures for training and testing while preserving prediction targets and data splits. The evaluation protocol uses references trained for each test pattern to compare models on identical test observations and targets, alongside equal-count groups for spatial-layout comparisons. On a 14,000-record subset, we evaluate 441 trained models under nine test patterns, yielding 3,969 evaluations. Mean cross-pattern error exceeds mean matched-pattern error in all 49 PDE-method pairs, and this finding persists in a configuration-matched subset of 117 models. Denser test observations do not consistently reduce error for a fixed model. Mixed-pattern training on five completed pairs reduces large single-pattern transfer errors, although destination-trained references usually remain more accurate. Together, the benchmark and findings support systematic evaluation of observation-pattern sensitivity and provide a reusable workflow for developing methods under changing measurement conditions. Code: https://github.com/ru1ch3n/PDE-OBS.

0.53

NeuroHorizon: Long-Horizon Forecasting of Neural Population Activity with Hierarchical Memory

cs.LG q-bio.NC ·2026-11-07 ·Peng Jiang, Xiaoxuan Jia
top venue similarity 0.62
Abstract · Accepted at NeurIPS 2026 (Main Track). 25 pages, including references, appendices, and the NeurIPS paper checklist

Long-horizon forecasting of neural population activity tests whether a model has captured dynamics that remain predictive well beyond the observed history. Prediction quality typically declines as the horizon grows, yet neither short-horizon accuracy nor teacher-forced evaluation, which supplies the true activity preceding each predicted step, reveals how well models sustain predictions from an initial history alone. We compare direct multi-step prediction and autoregressive rollout on spiking recordings across horizons and introduce NeuroHorizon, which pairs event-based spike encoding with a causal decoder for population firing rates. Its hierarchical prediction memory keeps recent predictions at full temporal resolution and compresses earlier ones into segment summaries, giving the decoder both local detail and longer-range context. Under teacher-forced training this memory holds ground-truth activity, whereas during rollout it holds the model's own predictions. To reduce this mismatch, we train with scheduled sampling, progressively exposing the decoder to its own predictions. From 500 ms of macaque motor-cortex activity, NeuroHorizon achieves the highest forward-prediction score among the evaluated autonomous models at all four horizons from 100 to 1000 ms, and its per-bin predictions remain above a mean-firing-rate baseline across nearly the full 1 s window. Complementary experiments on macaque sensorimotor and mouse multi-region recordings show that performance generally increases with model and data size, alongside improved prediction after cross-subject pretraining and fine-tuning. Together, these results clarify how forecasting performance changes across horizons and offer an effective approach to long-horizon forecasting of neural population activity.

0.32