CS Paper Radar

Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales

Marius P Linhard, Maurizio Filippone

cs.LG stat.ML ·submitted 2026-10-07 11:45 UTC · v1

Abstract

Bayesian neural networks need not be fully stochastic to be universal conditional density approximators, but it remains open which parameters should be stochastic. We learn this split by applying deep weight factorization to the prior scales, which are the standard deviations of the parameter priors, while fitting the functional prior to a Gaussian process with a maximum mean discrepancy objective. A parameter whose prior scale falls below a cutoff becomes deterministic and is optimized during inference, so the regularizer sparsifies stochasticity rather than capacity. We give a certificate for universal conditional density approximation that is checkable in linear time, together with a minimal repair when it fails. We further show that the common hybrid scheme of sampling some parameters and optimizing the others is stochastic approximation for a type-II maximum a posteriori objective, and that coupled step sizes can leave a tracking error that does not vanish as the step size shrinks. On a bimodal target, the learned split stays close to an unconstrained reference across all budgets and is insensitive to the cutoff, while random masks that distribute the same prior scales across layers are worse by up to two orders of magnitude. On UCI benchmarks, our method performs on par with a fully stochastic network while keeping about half of its parameters deterministic.

Closest papers

Nearest neighbours by meaning across the whole index.

Mitigating Representation Gaps in Amortized Bayesian Inference with Auxiliary Supervision

cs.LG stat.MLstat.COstat.ME ·2026-09-30 ·Hans Olischläger, Svenja Jedhoff, Šimon Kucharský, Aayush Mishra, Stefan T. Radev +1 more
similarity 0.68
Abstract

Casting Bayesian inference as a neural network optimization problem targeting an amortized posterior is attractive, as it extends to otherwise intractable statistical models and offers near instantaneous inference for new datasets after prepaying the training cost. Although theory guarantees faithfulness under ideal convergence, practical amortized inference still requires iterating over architectures and optimization choices and ultimately ``satisficing'' under finite simulation, compute, and time budgets. Even the best-performing solution may thus retain avoidable representation gaps that typically require problem-specific fixes. Here, we propose a generic alternative which improves training dynamics with auxiliary guidance losses applied to internal representations. Specifically, we show how such guidance leads to faster convergence when training data is abundant and to better performance when it is scarce. We formalize representation gaps as getting stuck in a local optimum at the information bottleneck between the parts of the network tasked with feature learning and those tasked with conditional distribution learning, and offer a generic diagnostic to separate summary failures from inference failures. Finally, we demonstrate that auxiliary supervision improves convergence speed and accuracy on a range of challenging real-world inference problems.

0.27

Sparse Bayesian Neural Networks via Sticky Piecewise Deterministic Markov Processes

cs.LG stat.MLstat.CO ·2026-10-23 ·August Arnstad, Geir Olve Storvik, Filippo Pagani, Leiv Rønneberg
similarity 0.66
Abstract

There is growing evidence that the performance of a neural network depends largely on a small subset of its weights, which are typically identified by training a dense network and pruning it afterwards. Within a Bayesian framework, pruning can be directly incorporated through a spike-and-slab prior, providing a proper treatment of the uncertainties involved, both with respect to pruning and to parameter uncertainty. In Bayesian neural networks (BNNs), sampling by MCMC is notoriously hard. We demonstrate that modern Monte Carlo methods based on piecewise deterministic Markov processes (PDMPs) can be used to sample posteriors of BNNs asymptotically exact whenever numerically constructed rate bounds hold. Spike-and-slab priors can be incorporated through sticky versions of the PDMPs, setting weights exactly to zero during sampling. We examine the predictive performance and convergence of the samplers, and show that sparse BNNs sampled this way can be more robust to input noise than dense ones.

0.15

Cross-Fitting for Neural Posterior Estimation

cs.LG stat.ML ·2026-10-19 ·Jeffrey Regier
top venue similarity 0.65
Abstract · Accepted to NeurIPS 2026

Neural Posterior Estimation (NPE) fits an amortized conditional density estimator to a simulated training set of complete-data realizations. NPE's training objective, log-likelihood, favors posterior approximations that assign arbitrarily high density to the training points, and typical families of amortized conditional density estimators (e.g., normalizing flows, Gaussian encoders) are flexible enough to oblige. The resulting policy of extreme overconfidence translates poorly to held-out data. Early stopping and other regularizers can prevent this degeneracy, but they are limited to selecting posterior approximations along an unfavorable optimization trajectory set by in-sample residuals, where the posterior mean is underfit while the posterior variance is overfit. To avoid this trade-off, we propose Cross-Fitting NPE (CF-NPE), which uses sample splitting to separate conditional-mean estimation from the estimation of the posterior's higher moments. Using $K$-fold cross-fitting, CF-NPE fits $K$ posterior-mean regressors and then fits a conditional density estimator to the resulting out-of-fold residuals. In experiments with synthetic data and in a cosmological application, CF-NPE achieves higher held-out log-likelihood than standard NPE and typically improves calibration. CF-NPE is most useful in high-dimensional, simulation-limited settings.

0.27

A Nonlinear Abstract Domain for Bayesian Neural Networks

cs.PL ·2026-11-14 ·Qiuhan Gu, Gagandeep Singh
similarity 0.65
Abstract

Bayesian neural networks (BNNs) explicitly model epistemic uncertainty by leveraging probability distributions over network parameters, making them better suited for risk-aware decision-making. Yet soundly propagating uncertainty through the network and capturing the entire output distribution to support predictive uncertainty quantification remains a fundamental challenge. Existing approaches are either sampling-based, providing no formal guarantees, or propagate only selected statistical properties, such as expectations and entropy, limiting their applicability to general probabilistic queries. To address these challenges, we present the first nonlinear abstract domain for soundly approximating complete probability distributions of BNNs. The abstract domain builds upon a novel marginal collecting concrete semantics that soundly abstracts the classical joint-measure semantics. It combines support partitioning with exponential-affine bounds to represent probability measures, capturing both discrete and continuous components of mixed distributions as well as nonlinear density shapes over bounded and unbounded supports. We develop sound abstract transformers that can efficiently handle high-dimensional affine transformations and nonlinear ReLU activations, while remaining highly parallelizable. We further extend the framework beyond BNNs to support bounded-loop probabilistic programs with Bayesian inference to demonstrate its generality. We implement the resulting abstract interpreter as a GPU-accelerated analyzer. Experimental results show that the framework significantly improves precision over existing probabilistic reasoning approaches while soundly capturing the complete output distribution, demonstrating strong efficiency and scalability.

0.24

Compressed Simulation-Based Inference: Neural Posterior Estimation for Massive Data via Low-Dimensional Sketches

cs.LG stat.MLstat.COstat.ME ·2026-11-16 ·Michael Wieck-Sosa, Cosma Rohilla Shalizi
similarity 0.65
Abstract

Neural posterior estimation (NPE) is a popular Bayesian simulation-based inference (SBI) approach that uses neural networks trained on simulated data--parameter pairs to approximate the posterior given a user-chosen prior. For high-dimensional observations, NPE methods typically approximate the conditional density of the parameter given a user-chosen or learned summary statistic, rather than the entire dataset, due to the curse of dimensionality. However, both approaches face challenges: hand-crafting good summaries for complex models requires considerable effort from scientists, while learning them from massive datasets can be prohibitively computationally expensive. To address these challenges, we propose approximating the conditional density of a parameter given low-dimensional sketches of the data. We show that, for a model with a $p$-dimensional parameter, only $2p+1$ random sketches are needed for posterior consistency. We compare with state-of-the-art embedding networks for NPE, and we find that our method \emph{matches} the best embedding network's performance while \emph{increasing computational efficiency}.

0.06

Learning to Bias: Machine Learning-Enhanced Particle Filters

cs.LG stat.ML ·2026-09-24 ·Apoorv Srivastava, Eric Darve
similarity 0.65
Abstract · 29 pages, 9 figures, including appendices

Sequential inference estimates latent states from noisy and incomplete observations. Particle Filters (PFs), a class of Monte Carlo methods based on importance sampling, provide a flexible framework for this task, but often suffer from poor sample efficiency and unfavorable scaling with dimension, partly due to suboptimal proposal distributions. We address these challenges by integrating learned proposals into the PF framework. We introduce Neural Optimal Particle Filters (NOPFs), which learn an amortized approximation to the optimal proposal from offline simulated one-step conditioning tuples. The learned proposal is used as a drop-in replacement in standard PF updates, with samples corrected by standard importance weights so that the method asymptotically targets the same filtering distribution under standard support and density-evaluation assumptions. Across stochastic nonlinear benchmarks of varying inference complexity, NOPFs improve sample efficiency and distributional accuracy over standard PF baselines with modest computational overhead. The approach integrates data-driven proposal learning into classical inference without altering the underlying filtering objective.

0.34

Efficient Deep Gaussian Process Surrogates via Mini-Batch Latent MAP Estimation

cs.LG stat.ML ·2026-10-27 ·Chun-Yi Chang, Chih-Li Sung
similarity 0.64
Abstract · 35 pages including references, 11 pages appendix, 5 figures

Deep Gaussian processes (DGPs) are flexible surrogate models for computer experiments with nonstationary response surfaces, but inference over their latent representations remains computationally challenging. Existing methods typically rely on variational inference or Markov chain Monte Carlo (MCMC), which impose variational-family assumptions or require costly posterior sampling. We propose the latent maximum-a-posteriori deep Gaussian process (LaMP-DGP), which uses gradient-based optimization to obtain MAP estimates of the latent representations and enables efficient probabilistic prediction through linked-GP calculations. To scale LaMP-DGP to large datasets, we develop a mini-batch optimization algorithm and, under a polynomial mini-batch bias-decay condition, establish a stationarity bound for latent-state optimization that motivates a growing batch-size schedule. Across numerical experiments and real-data analyses, LaMP-DGP provides consistently competitive predictive performance relative to doubly stochastic variational inference, stochastic gradient Hamiltonian Monte Carlo, and fully Bayesian Vecchia DGP inference. Computationally, LaMP-DGP is substantially faster than the Vecchia MCMC approach while remaining broadly competitive with the stochastic-gradient alternatives. These results position LaMP-DGP as a competitive and computationally efficient alternative for DGP inference. An R package, "LaMPDGP", is available for implementing the proposed method.

0.15

Thompson Sampling using Prior-fitted Diffusion Transformers

cs.LG cs.AIstat.ML ·2026-11-10 ·Sihwa Park, Jingsen Zhu, Vinamr Jain, Sheng-Yen Chou, Alexander Terenin
top venue similarity 0.64
Abstract

Thompson sampling is natural randomized strategy for optimizing unknown functions in a black-box manner. In continuous global optimization, Thompson sampling has largely been restricted to Gaussian process priors, as these are essentially the only class for which computing the posterior distribution has been tractable in practice. In this work, we introduce \emph{prior-fitted diffusion transformers}, which are diffusion models that perform Thompson sampling in-context at inference time, after being pre-trained using the prior-fitted network paradigm. To apply these models, we develop a set of practical non-Gaussian priors for general black-box functions---including those that have characteristics that cannot be achieved under Gaussianity, such as priors over unimodal functions. We also quantify the tradeoff between model size and the number of pre-training iterations, and show that these models follow scaling laws akin to those observed in transformers defined over other modalities. Across a set of global optimization benchmarks, we show that prior-fitted diffusion transformers with non-Gaussian priors can achieve improved performance compared to traditional pipelines built on Gaussian processes.

0.39