CS Paper Radar

Sufficiently Reduced Distributional Regression

Alexander Henzi, Tiange Liu, Xinwei Shen

cs.LG stat.MEstat.ML ·submitted 2026-09-24 09:27 UTC · v1
theory

Abstract

We propose Sufficiently Reduced Distributional Regression (SRDR), a generative method that combines conditional distribution estimation with nonlinear sufficient dimension reduction (SDR). It builds on a characterization of sufficiency through strictly proper scoring rules: a dimension reduction is sufficient if and only if predicting the response from the reduced covariates incurs no loss in expected score relative to the full covariates. Sufficient dimension reduction thus becomes a risk minimization problem. SRDR jointly trains a dimension reduction map and a generative prediction model by minimizing the energy score, which can be estimated by sampling without density evaluation or adversarial training. The framework extends to multi-environment data and to classification. We prove that the estimated conditional distributions converge in energy distance to the true ones, which implies that the learned representation is asymptotically sufficient. In simulations and applications to CT slice localization, superconductivity, and digit classification, SRDR recovers low-dimensional sufficient structure and matches or outperforms state-of-the-art nonlinear SDR methods in representation quality and predictive performance.

Closest papers

Nearest neighbours by meaning across the whole index.

Belted Engression: Sufficient Dimension Reduction for Generative Distributional Regression

cs.LG cs.AImath.STstat.ME+1 ·2026-09-20 ·Wenxi Tan, Bing Li, Lingzhou Xue
theory similarity 0.77
Abstract · 32 pages, 6 figures

Modern conditional generative models face significant challenges when learning complex covariate dependencies. While sufficient dimension reduction (SDR) provides a principled approach to compress these dependencies, traditional SDR frameworks were not formulated for conditional generation. To bridge this gap, we propose Belted Engression, a unified and architecturally parameter-efficient framework for generative distributional regression. Our approach establishes an end-to-end compress-then-generate paradigm driven by sufficient representation learning, embedding a structural bottleneck into the generative architecture. Theoretically, we prove that the standard SDR condition is equivalent to a law-preserving generative factorization, which is achieved at the global optimum of the population Belted Engression objective. Furthermore, by uncovering a localized Bernstein-type control for the energy-score loss, we establish finite-sample convergence rates that are sharper than those of existing results. We also prove that this belted architecture is strictly smaller, operating with an asymptotically vanishing parameter count relative to the unstructured baseline. Extensive simulations and real-world applications demonstrate that Belted Engression achieves superior distributional prediction and SDR recovery with fewer trainable parameters.

0.23

Sufficient Dimesion Reduction via Generalized Stein's Lemma

cs.LG stat.MLstat.ME ·2026-08-15 ·Ye Tian
similarity 0.69
Abstract

Sufficient dimension reduction (SDR) seeks the minimal subspace of the predictors that captures the full conditional distribution of the response, which is known as the central subspace (CS). When the response is multivariate, the problem becomes considerably more challenging, particularly when the sample size is limited. Existing methods face different limitations:inverse regression approaches rely on strong distributional assumptions and matrix inversion, and their multi-response extensions suffer from severe slice sparsity; forward regression methods depend on computationally intensive iterative smoothing whose cost grows with the response dimension; and deep learning-based approaches demand large amounts of labeled data. To circumvent these shortcomings, we propose an SDR framework based on the generalized Stein's lemma. Our method constructs a cross-moment matrix between the multivariate response and the marginal score function of the predictors, and recovers the CS via its singular value decomposition. The proposed method does not rely on the linearity condition, avoids matrix inversion as well as iterative smoothing, and can leverage unlabeled data when available. We establish convergence guarantees for the proposed estimator under standard regularity conditions. Moreover, we propose a practical rank-selection algorithm to estimate the dimension of the CS. Extensive simulation studies and a real data application demonstrate that the proposed methods consistently outperform existing approaches across a variety of settings, particularly in moderate-dimensional, label-scarce scenarios with high noise levels.

0.04

Nonparametric autoregressive estimation of probability distributions with parallel gradient boosting

cs.LG stat.ML ·2026-10-22 ·Rémy Chapelle
similarity 0.66
Abstract

Nonparametric estimation of joint probability distributions is a core problem in statistics, which is notoriously challenging with high-dimensional data. Such challenges also arise with conditional distribution estimation and have led to flexible algorithms that aim to tackle the ''curse of dimensionality'' by exploiting predictive structures of lower dimension than the ambient space. In this paper, we study the use of such conditional distribution estimators to estimate joint distributions by factorizing the latter in an ''autoregressive'' way. We show that for loss functions satisfying a subadditivity property, bounds on the convergence rate of an autoregressive estimator can be derived from bounds on the convergence rate of the conditional distribution estimators. These bounds relate the difficulty of estimating the joint distribution to that of estimating the most challenging factor in the autoregressive factorization. When the conditional estimators adapt to sparsity in their conditioning sets, we characterize this difficulty through variable elimination in an undirected dependence graph. This allows bounding the rates of convergence by quantities related to the sparsity of the graph, and provides an additional motivation for the autoregressive approach. We then evaluate the practical performance of an autoregressive estimator based on parallel gradient boosting, a flexible nonparametric method for estimating conditional distributions. The estimator demonstrates competitive estimation performance on various simulated and real datasets, even without optimizing the factorization order. Together, these results support nonparametric autoregressive estimation as a practical approach to flexible joint distribution estimation.

0.17

Conditional-Independence-Regularized Distributional Autoencoders for Mixed-Type Data

cs.LG stat.MEstat.ML ·2026-08-20 ·Siyuan Tang, Gongjun Xu, Ji Zhu
publishedcode/data similarity 0.64
Abstract · Accepted by STAI-X 2026

Mixed-type data containing both numerical and categorical variables arise in many scientific and real-world applications. Existing representation learning and generative modeling approaches typically focus either on reconstruction accuracy or unconditional data generation, but often fail to recover the full conditional distribution of the data while preserving interpretable structural relationships between heterogeneous variable types. In this work, we introduce Conditional-Independence-Regularized Distributional Autoencoders, a framework for learning low-dimensional representations of mixed-type data through conditional distribution matching and structural regularization. Our method combines an energy-score-based objective for numerical variables, a likelihood-based objective for categorical variables, and an auxiliary conditional independence regularization term encouraging the learned representation to capture the dependence between numerical and categorical components. We provide theoretical analysis showing that the optimal representation balances unexplained numerical variability, conditional entropy of categorical variables, and residual conditional dependence. Empirically, the proposed method achieves strong performance on both synthetic and real-world datasets, substantially improving categorical distribution recovery, achieving competitive overall conditional distribution recovery, and preserving mixed-type dependence structure. The code has been made available at GitHub.

0.36

Training Calibrated Conditional Generative Models with a Sample-Only Coverage Loss

cs.LG stat.MEstat.ML ·2026-11-11 ·Benjamin Remy, Raphael Rossellini, Chihway Chang, Rebecca Willett
theory similarity 0.64
Abstract · 32 pages, 7 figures

Conditional generative models have become the tool of choice for sampling posteriors in scientific inverse problems, where we seek to sample images or signals conditioned on a set of indirect and noisy measurements. Yet they are often overconfident, which is a critical failure in scientific applications where accurate uncertainty quantification is essential. Existing regularizers fall short in complementary ways: differentiable coverage objectives require the model's density and do not identify the posterior, while strictly proper scoring rules such as the energy score need only samples but do not directly enforce coverage. We close this gap by introducing a differentiable loss function based on the expected coverage probability, estimated from samples alone. It applies to any conditional generative model, with no need for tractable densities or architectural changes. We prove that it is a strict divergence, vanishing if and only if the model equals the true posterior. On simulation-based inference benchmarks, it matches a density-based coverage loss without evaluating the density. On high-dimensional imaging inverse problems, it yields better-calibrated posteriors than scoring-rule regularizers, with little change in the reconstruction quality of point estimates.

0.22

Estimating Spectrum-Free Information Bounds for Generative Surrogates Beyond Conditional Collapse

cs.LG hep-phhep-ex ·2026-11-05 ·Chaitanya Dumbre, Sourabh Dube
benchmark similarity 0.64
Abstract · 20 pages, 7 figures

Conditional generative surrogates in high-energy physics, such as models that predict reconstructed jets from partons, are often evaluated on $1\text{D}$ marginal divergences among other metrics. Taken alone these metrics can reward conditional collapse. On a CMS simulation benchmark, a sampler that ignores the conditioner beats a trained Mixture Density Network (MDN) on all four target marginals under both KL and Wasserstein-1, and cannot be stably separated from a Conditional Flow Matching (CFM) model on two of the four. We instead score the models with the Continuous Ranked Probability Skill Score (CRPSS) from forecast verification, taking the exact marginal as the reference. We show how global skill depends on the evaluation spectrum, and we also report it within bins of a conditioning variable, where that dependence largely goes away. We then introduce a data-driven $k$-Nearest-Neighbor ($k$-NN) extrapolation that estimates the best CRPS the conditioning information allows. Both models agree with this estimate on all four targets, which suggests that current architectures have reached the information available in the conditioner, at least as far as these per-target scores can tell.

0.24

Optimal Transport Dropout for Structured Predictive Uncertainty

cs.LG math.NA ·2026-09-27 ·Giacomo Lorenzon, Francesco Regazzoni
similarity 0.63
Abstract

Deterministic neural networks and neural operators provide point predictions with no intrinsic measure of reliability. Yet, predictive uncertainty may stem from irreducible outcome variability, finite data, or limitations of the chosen model class. Monte Carlo dropout offers a computationally convenient way to construct a predictive distribution through stochastic feature masking, without training multiple independent networks or explicitly inferring a posterior over model parameters. However, its perturbation law is largely prescribed a priori and typically factorised across latent coordinates. We introduce Optimal Transport Dropout (OTD), which instead learns the predictive mapping and the law of its latent perturbations jointly. Starting from a simple independent reference distribution, OTD transports latent perturbations through a learnable flow and propagates them through the predictive neural network, thereby inducing a structured predictive law. Training uses the strictly proper Energy Score, while a kinetic-action term geometrically regularises the transport. Synthetic benchmarks show that OTD captures multimodal predictive distributions, generates meaningful dispersion when the model is misspecified, and exhibits contracting dispersion as more training data or greater model capacity are provided. For a field-valued partial differential equation surrogate, predictive dispersion strongly aligns with the spatial pattern of prediction errors. On this task, compared with Monte Carlo dropout, OTD yields more accurate predictions and better-calibrated, substantially narrower intervals. On real-world regression benchmarks, it further shows competitive accuracy and better probabilistic predictions compared to established baselines. OTD therefore offers a way to learn structured predictive uncertainty without explicit posterior inference or ensembles of independently trained predictors.

0.31

ReMaD: Tuning-free Domain Adaptation for Classification and Out-of-Distribution Detection

cs.LG ·2026-10-05 ·Elijah Bolluyt, Cristina Comaniciu
similarity 0.63
Abstract

We introduce Reduced-rank Mahalanobis Distance (ReMaD), a novel prototypical distance-based refinement to classification and out-of-distribution (OOD) detection using pretrained models without finetuning. We use embeddings of the target dataset to fit closed-form distribution statistics in the model's latent space which can classify in-distribution samples and detect OOD samples, all without training or prior knowledge of the OOD data. Building on prototype classification and OOD detection, we analyze the distribution properties of large pretrained models when processing new datasets; based on this analysis, we formulate a simple modification to Mahalanobis Distance to adapt models' latent space distributions to new domains by removing unused features, without the finetuning or hyperparameter searches required by other adaptation procedures. We demonstrate the efficacy of this method to adapt existing large pretrained image embedding models to new classification domains outside their trained capabilities by testing across four target datasets, with competitive performance in both classification and OOD detection.

0.15