CS Paper Radar

Marginal Calibration Masks Generator Differences in Held-Out Evaluation of Multivariate Time Series, with an Application to ERA5 Weather Sequences

Mustafa Özaytaç, Özge Karadağ Ataş

cs.LG ·submitted 2026-09-27 16:59 UTC · v2

Abstract

Generative models for multivariate time series are routinely evaluated with pooled distributional metrics computed after marginal calibration. We show that, under a held-out protocol, the calibrated Kolmogorov-Smirnov (KS) distance differs from a real-versus-real floor by at most the distance of the calibrated sample to the training block, so that it carries little information about the generator. We audit a three-discriminator GAN with evolutionary weight adaptation (StatD2GAN) against re-implemented RCGAN and TimeGAN baselines on ERA5 weather sequences under a pre-specified protocol: the final two calendar years are held out behind a 168 hour embargo, calibration is fitted on the training block only, and the decision rules were frozen in advance. Evidence comes from 25 matched (location, seed) pairs across five locations, tested with Holm-corrected Wilcoxon signed-rank tests. First, isotonic calibration drives KS to a median of 1.019 times a per-location noise-and-shift floor, and neither baseline is distinguishable from StatD2GAN on it. Second, the sorted-representation discriminator is the only component whose removal significantly degrades cross-variable dependence (Kendall tau MAE +0.116, Holm p = 0.002); a rank-transformed variant also degrades it, consistent with quantile supervision rather than copula matching. Third, calibration raises physical constraint violations; projection removes them, but the largest marginal change (0.0110, Lhasa) exceeds the pre-specified tolerance in three locations. Fourth, the re-implemented TimeGAN is significantly better than StatD2GAN on Kendall tau and ACF MAE, RCGAN is worse on Kendall tau MAE and constraint violations, and the pre-specified collapse criterion was not met. We recommend floor-referenced marginal evaluation, matched-pair testing, and dependence and sequence-level metrics.

Comment
Version 2: supersedes v1, whose results are retracted. Code and weights: doi:10.5281/zenodo.22996771 and doi:10.5281/zenodo.23480462. Derived from the first author's Master's thesis. 37 pages

Closest papers

Nearest neighbours by meaning across the whole index.

When Does Adversarial Refinement Help? A Negative Result and Open Problem in Adapting R3GAN to Time Series Imputation

cs.LG ·2026-09-19 ·Yufeng He
publishedpositioncode/data similarity 0.63
Abstract · 4 pages, 1 figure, 2 tables. Accepted at the 12th Workshop on Mining and Learning from Time Series (MiLeTS 2026), held with KDD 2026. This a…

Diffusion models and transformers have supplanted GANs for multivariate time series imputation, largely on grounds of GAN training instability. R3GAN (NeurIPS 2024) removes that instability via regularized relativistic losses with provable convergence, raising a natural question: do stable, modern GANs revive adversarial imputation? We adapt R3GAN to 1D temporal data with a coarse-to-fine refinement framework and a frequency-domain discriminator, and audit 14 saved configurations across 3 datasets. Because these are heterogeneous single runs, the evidence is descriptive rather than a matched causal ablation. We report a negative result. All five saved mean/zero-start configurations improve by 48.4-70.2%. Among eight eligible non-legacy linear-start configurations, the mean change is -0.7% (range -3.0% to +1.1%); a separate -21.9% legacy logging anomaly is retained for provenance but excluded from that aggregate. In a saved Weather comparison, standalone R3GAN-1D underperforms BRITS by 5.8x. Crucially, we argue the common explanation (that GANs optimize distributional rather than point-wise objectives) cannot be the whole story, since diffusion models also optimize distributional objectives yet achieve state-of-the-art imputation. Our saved reconstruction-weight sweep is consistent with the adversarial signal being inert or harmful, but cannot identify its causal contribution; a matched discriminator-removed ablation is the key next experiment. We frame the precise reason a learned discriminator fails to provide useful refinement gradients (where a learned diffusion denoiser succeeds) as an open problem, and offer practical guidance on when adversarial refinement is worthwhile.

0.46

On the Reliability of Generative Augmentation: A Wasserstein-Based Theoretical and Empirical Study

cs.LG stat.ML ·2026-09-01 ·Chathurika S Abeykoon, Mathias Nthiani Muia, Mallory Goldstein
similarity 0.59
Abstract · 19 pages, 3 figures, appendix file

Generative data augmentation is widely used to mitigate class imbalance, yet its theoretical effect on downstream generalization remains poorly understood. In this work, we develop a statistical framework for conditional generative augmentation and analyze its impact on classification risk. We formalize augmentation as a distribution-mixing process and show that the resulting risk distortion is controlled by both the augmentation strength and the class-conditional Wasserstein discrepancy between real and generated distributions. We further derive a capacity-dependent generalization bound based on Rademacher complexity, revealing an explicit trade-off between hypothesis complexity, augmentation intensity, and generative fidelity. Empirically, we evaluate the framework on binary and multiclass imbalanced classification tasks using Conditional GAN and Conditional WGAN-GP augmentation. Across datasets, CWGAN-GP consistently achieves lower Wasserstein discrepancies than CGAN, indicating improved distributional fidelity. However, improved fidelity does not necessarily translate into superior classification performance, with classical oversampling methods often remaining competitive. These findings support the central theoretical prediction that augmentation reliability is governed by distributional approximation error rather than predictive performance alone. Overall, this work establishes generative augmentation as a distributional perturbation process whose reliability can be quantified through Wasserstein-based measures and supported by finite-sample generalization guarantees. The proposed framework provides a principled foundation for evaluating synthetic data quality beyond classification accuracy alone.

0.35

Estimating Spectrum-Free Information Bounds for Generative Surrogates Beyond Conditional Collapse

cs.LG hep-phhep-ex ·2026-11-05 ·Chaitanya Dumbre, Sourabh Dube
benchmark similarity 0.55
Abstract · 20 pages, 7 figures

Conditional generative surrogates in high-energy physics, such as models that predict reconstructed jets from partons, are often evaluated on $1\text{D}$ marginal divergences among other metrics. Taken alone these metrics can reward conditional collapse. On a CMS simulation benchmark, a sampler that ignores the conditioner beats a trained Mixture Density Network (MDN) on all four target marginals under both KL and Wasserstein-1, and cannot be stably separated from a Conditional Flow Matching (CFM) model on two of the four. We instead score the models with the Continuous Ranked Probability Skill Score (CRPSS) from forecast verification, taking the exact marginal as the reference. We show how global skill depends on the evaluation spectrum, and we also report it within bins of a conditioning variable, where that dependence largely goes away. We then introduce a data-driven $k$-Nearest-Neighbor ($k$-NN) extrapolation that estimates the best CRPS the conditioning information allows. Both models agree with this estimate on all four targets, which suggests that current architectures have reached the information available in the conditioner, at least as far as these per-target scores can tell.

0.24

Deciding When to Switch: E-Processes for Adaptive Minimax Training for Generative Adversarial Nets

cs.LG stat.ML ·2026-08-10 ·Hyunjoo Kim, Sicheng Wu, Agastya Venkatraman, Guang Lin, Sehwan Kim
benchmarktheory similarity 0.55
Abstract

Modern data science increasingly gives rise to hypothesis-testing problems that are not naturally formulated in terms of parameters within prespecified statistical models. One important example is the dynamic evaluation of optimization algorithms, where decisions must be made during training about whether further updates remain beneficial or the algorithm should switch to a different phase. This issue is particularly relevant in stochastic min-max optimization. Generative adversarial networks (GANs) provide a canonical example, as their training requires repeated decisions about when to switch between discriminator and generator updates, yet existing methods typically rely on fixed update ratios or heuristic criteria. We formulate this switching problem as sequential hypothesis testing and develop an e-process-based adaptive training procedure. During discriminator updates, one e-process tests the null that the discriminator-induced separation between the empirical data distribution and the generator law remains below a target level. During generator updates, with the discriminator fixed, a second e-process tests the reverse null that this separation remains above a refresh level. Conditional on the observed training sample, we prove that fresh empirical indices and latent draws yield conditional e-values that can be accumulated into e-processes, providing anytime-valid Type I error control under adaptive model updates and data-dependent switching. Across multimodal synthetic distributions and image benchmark datasets, the proposed method matches or outperforms the best fixed-ratio baselines under several widely used GAN objectives.

0.04

EmbeddGAN: A Novel GAN Framework Using an Embedding Network and Gini Distance Correlation

cs.LG ·2026-09-18 ·MaTais Caldwell, Yixin Chen, Xin Dang, Charles Walter
similarity 0.53
Abstract · 18 Pages, 14 Figures, Submitted to IEEE Access, Awaiting Revisions

Generative Adversarial Networks (GANs) have demonstrated strong performance in generating high-quality synthetic data. However, they are limited by no formal guarantees regarding convergence and the effectiveness of the learning process. In practice, this leads to training instability, mode collapse, and sensitivity to hyperparameters. To address this, we propose EmbeddGAN, a novel adversarial training framework based on a dependence-based objective. Instead of relying on a discriminator that classifies samples as real or fake, EmbeddGAN introduces an embedding network that learns a representation in which statistical dependence between samples and their real/fake labels is maximized, while the generator is trained to minimize this dependence. This objective is implemented using the Gini distance correlation (gCor), which equals zero if and only if the embeddings are statistically independent of the real/fake label. Minimizing this objective therefore encourages real and generated samples to become statistically indistinguishable in the learned embedding space. The embedding network projects both real and generated data into a shared low-dimensional space, where distributional discrepancies can be measured directly through pairwise distances. We adopt a minimax training strategy: the embedding network maximizes the Gini distance correlation (maximizing dependence), while the generator minimizes it (minimizing dependence). Experiments on the MNIST, CIFAR-10, and CelebA datasets demonstrate that EmbeddGAN achieves competitive performance relative to established baselines while exhibiting notably stable training dynamics on the evaluated datasets.

0.13

Forecasting Voltage Time Series under Anomalies Using a WGAN Model With Lipschitz Control and Progressive Kernel Critic Strategy

cs.LG ·2026-10-16 ·Najmieh Sadat Safarabadi
benchmark similarity 0.53
Abstract

In time-series forecasting tasks, the use of recursive forecasting on high-dimensional data substantially increases the complexity of the prediction process. The main challenge is that long-term dynamics becomes harder to model, since the sequences show greater temporal variability. To counteract this limitation and improve the modeling of long-term dynamics, and more importantly, address high-dimensional recursive forecasting, this paper uses a Wasserstein GAN equipped with a progressive Lipschitz penalty and an adaptive kernel. This model is capable of taking a window of the data stream and forecasting the next values, subsequently reconstructing the time-series stream and dynamically adapting its kernel to each new window. The proposed model is a deep learning framework based on a Wasserstein GAN (WGAN-CLP) with an adaptive kernel to model alterations in streaming voltage time series. To establish the methodological foundation, we review the WGAN-LP formulation another variant model in this class and extend its framework through an adaptive kernel critic model. We then evaluate this extended model using power ratios, correlation coefficient (CC), and relative root mean squared error (RRMSE). The experimental results demonstrate that the proposed model outperforms benchmark methods in both accuracy and robustness. The evaluation shows promising performance of the WGAN-CLP with an adaptive kernel framework in forecasting sequence alterations across the voltage and energy time-series datasets.

0.26

Benchmarking Generative Models for Near-Surface Data Assimilation on Real Station Observations

cs.LG cs.AI ·2026-09-30 ·Ruizhe Huang, Qidong Yang, Jonathan Giezendanner, Sherrie Wang
benchmark similarity 0.53
Abstract

Weather reanalysis products rely on computationally intensive numerical weather predictions followed by data assimilation that corrects the forecast toward observations. Recent advances in deep generative models offer a cheaper alternative that shifts much of this cost from inference to offline training. However, existing generative approaches have been evaluated on synthetic observations or under different datasets and evaluation schemes, making it unclear which design choices improve real-world data assimilation. We present the first controlled benchmark of generative data assimilation for single-time near-surface analysis from real weather station observations. Using 11,849 NOAA MADIS stations across the contiguous United States and four near-surface variables, we hold the dataset, observation operator, and deep learning architecture fixed, and measure spatial generalization at held-out stations. The benchmark compares the major design choices proposed for generative data assimilation, including diffusion versus flow matching, pixel versus latent-space formulations, and multiple inference-time conditioning strategies, against a classical 3D-Var baseline. The benchmark reveals three conclusions. First, the best generative methods outperform 3D-Var (35.7% vs. 33.3% RMSE improvement over ERA5), although 3D-Var receives the ERA5 field at the analysis time as its background and the generative methods receive none. Second, full-gradient guidance consistently outperforms stop-gradient and initial-noise optimization. Third, other choices provide little measurable benefit: diffusion and flow matching perform nearly identically under matched conditions, and latent-space variable mixing does not help. Both advantages widen when stations are sparse. Together, these results identify which components of generative data assimilation improve spatial generalization in near-surface analysis.

0.25

Safe Domain Adaptation for Physics: Overcoming Nuisances, Label Shifts, and Simulation Priors

cs.LG astro-ph.IMphysics.data-an ·2026-08-18 ·Ivan Kharuk
benchmark similarity 0.52
Abstract

Domain adaptation is widely used to make neural networks trained on simulations applicable to experimental data. Its premise is that the two domains differ only in nuisances, and that the quantity of interest is distributed identically in both. In physics neither assumption holds: simulations can be wrong about the physics, and the distribution of the target quantity - an energy spectrum, a redshift distribution - is often the measurement itself. We study the consequences of such mismatches on a toy air-shower benchmark in which a detector-response nuisance, a physical simulation shift, and an energy-spectrum shift can be switched on separately or together. Standard adversarial adaptation handles the conditional shifts, but once the two spectra differ it aligns them, replacing an uncontrolled bias by one anchored on the simulation prior. We present adaptive domain adaptation, which reweights the simulated events so as to focus domain adaptation on the genuine physical mismatch alone. Since the predicted spectrum depends on model training configuration, we provide a label-free model selection rule for selecting the near-the-best operation point.

0.18