CS Paper Radar

Ensembles of Random Partition-Based Models under Covariate Shift

Mary Versa Clemens-Sewall, Eliza O'Reilly

cs.LG math.ST ·submitted 2026-11-03 16:47 UTC · v1

Abstract

Distribution shifts, including covariate shifts, are ubiquitous in real-world applications of machine learning. Empirical evidence suggests that, across a variety of models with and without downstream adaptation, ensembles tend to be more robust to covariate shift than single models. We extend the theory of covariate shift severity from single models to ensembles, providing theoretical justification for the empirical trends. We introduce new discrepancy measures of covariate shift severity to control the risk of partition-based kernel estimators. We then show that with sufficient ensemble size, these models can achieve minimax-optimal risk convergence rates under covariate shift. Our results suggest that modelers can expect to need larger ensemble sizes for more severe covariate shifts.

Comment
29 pages, 3 figures

Closest papers

Nearest neighbours by meaning across the whole index.

General Quantification of Covariate and Concept Shifts

cs.LG cs.AIstat.ML ·2026-09-10 ·Hongbo Chen, Li Charlie Xia
top venue similarity 0.65
Abstract · 38 pages, 9 figures, accepted at the 43rd International Conference on Machine Learning (ICML 2026)

Generalization under distribution shift remains a core challenge in modern machine learning, yet existing learning bound theory is limited to narrow, idealized settings and is non-estimable from samples. In this paper, we bridge the gap between theory and practical applications. We first show that existing definition of concept shift breaks when the source and target supports mismatch. Leveraging entropic optimal transport, we propose a key notion: $γ^{*}\!$-concept shifts, and derive a general error bound unifying covariate and $γ^{*}\!$-concept shifts, which applies to broad loss functions, label spaces, and stochastic labeling. We further develop estimators for these shifts with concentration guarantees, and the DataShifts algorithm, which can quantify distribution shifts and estimate the error bound in most applications - a rigorous and general tool for analyzing learning error under distribution shift.

0.44

CVaR anchor regression protects against rare shifts

cs.LG stat.ME ·2026-09-22 ·Malte Londschien
theory similarity 0.59
Abstract

We study prediction in new environments when the training data contains rare, large distribution shifts. Anchor regression penalizes the average of the squared mean residual across environments. It protects against shifts in an ellipsoid determined by the second moment of the training shifts. Covering rare shifts may therefore require a large penalty $γ\gg 1$, expanding the ellipsoid in every direction and reducing accuracy on common environments. We propose CVaR anchor regression, which replaces the average of the squared mean residuals with a tail average. We prove an exact worst-case risk guarantee under a linear structural model that allows for heteroscedastic noise. CVaR anchor regression protects against shifts in a union of ellipsoids determined by second moments of the shifts in the worst $α$-fraction of environments. For $α$ small enough, this is the scaled convex hull of the training shifts and their negatives. It therefore covers all training shifts when $γ\geq 1$. Unlike CVaR or GroupDRO applied directly to prediction risks, CVaR anchor regression does not give environments more weight solely because their noise levels are high. Examples show how the method can improve protection against rare shifts while retaining accuracy on common environments. We illustrate the method on the New York City taxi dataset.

0.25

Robust Satisficing Ensemble: Scalable Model Aggregation Under Distribution Shifts

cs.LG cs.AIstat.ML ·2026-11-07 ·Ahmet Faruk Cetinkaya, Enes Agirman, Cem Tekin
top venuetheory similarity 0.59
Abstract · Accepted at NeurIPS 2026 (Spotlight)

Standard ensemble methods can improve robustness by mitigating individual model stochasticity, but they frequently fail to generalize beyond the training distribution, leaving systems vulnerable to out-of-distribution (OOD) data. We introduce the Robust Satisficing Ensemble (RSE), a scalable probabilistic aggregation framework designed to secure model combinations by minimizing their fragility against distributional changes. By bridging a robust satisficing objective with a quadratic risk surrogate, RSE derives an analytical inner adversary. This fundamental reformulation enables highly efficient weight updates without the computational bottleneck of standard minimax optimization, crucially decoupling optimization complexity from sample size to scale effectively to large datasets. In particular, we prove a geometrically decaying truncation error for the fragility approximation via Lagrange-Burmann expansion and establish stationarity guarantees for RSE. We evaluate RSE across a comprehensive spectrum of distribution shifts, including adversarial label shift (SST-5, TREC), natural subpopulation and temporal shifts (CivilComments, HuffPost). Our results demonstrate that RSE consistently outperforms strong robust baselines, including GroupDRO, IRM, and SRM. These findings establish RSE as a practical, theoretically grounded safeguard for reliable model aggregation in dynamic environments.

0.43

Latent-Space Importance Weighting under Covariate Shift

cs.LG stat.ML ·2026-10-11 ·Yuling Jiao, Lican Kang, Yang Xiang, Jerry Zhijian Yang, Fusheng Zhou
similarity 0.58
Abstract · 37 pages, 2 figures

In this work, we propose a transfer learning algorithm for covariate shift, integrating sample reweighting with representation learning. A key feature of our approach is the use of a lower-dimensional latent space for modeling, in contrast to conventional methods that operate directly in the high-dimensional ambient space. Our method estimates the density ratio of learned latent features for downstream weighted prediction. We establish explicit target-domain excess-risk rates with independent data folds for strictly sufficient learned representations with regular learned-space prediction scores and density ratios. We give dimension comparisons under stated smoothness conditions and a variance comparison for true weights and a fixed predictor. General finite-sample inequalities retain representation-quality errors. Experiments on simulated and real-world datasets demonstrate gains from combining learned representations and latent weighting, and reveal sensitivity to initialization and domain choice.

0.22

Sharp training-conditional coverage for conformal prediction under covariate shift

cs.LG stat.MLmath.STstat.ME ·2026-09-27 ·Mehrdad Pournaderi
theory similarity 0.57
Abstract

Weighted split conformal prediction reweights calibration scores by the likelihood ratio between the test and training covariate distributions and guarantees marginal coverage under covariate shift. We study its coverage conditional on the calibration data. An elementary argument, based on a single concentration inequality at a fixed population quantile, gives explicit training-conditional bounds without unspecified constants, and shows that the relevant scale is not the supremum of the likelihood ratio but a variance proxy built from the chi-squared divergence of the shift and from the average of the ratio over the part of the test population, of probability equal to the miscoverage level, where it is largest. A two-point lower bound shows that the root-m rate and the chi-squared contribution are intrinsic to the shift. Run at an explicitly inflated level, the weighted quantile becomes a deterministic PAC prediction set. We compare it with randomized rejection sampling and with importance-weighted learn-then-test and, through a certified choice of a clipping level for the likelihood ratio, map the regime in which each gives the narrower valid set. The analysis extends to estimated likelihood ratios and to tail functionals estimated from an unlabeled source sample, which yields a fully finite-sample certificate.

0.44

Conformal Prediction under Exponential-Tilt Joint Shift

cs.LG stat.MLstat.ME ·2026-09-25 ·Seungjin Choi
similarity 0.56
Abstract · 23 pages

Conformal prediction can lose coverage when the data distribution changes after deployment. We study adaptation using labeled source data and unlabeled target inputs, allowing both the input distribution and its relationship with outcomes to change. We use Exponential Tilt Reweighting Alignment (ExTRA), introduced for classification by Maity et al. (2023), to estimate structured distribution shifts. We compare using its estimated weights in conformal calibration with additionally tilting the source predictive distribution. Shared learned predictors, estimated weights, calibration samples, and test observations isolate the effect of tilting. Existing theory gives both procedures target coverage with true weights and a common coverage bound with estimated weights. Identification calculations and an analysis of how scoring interacts with weight estimation error help explain why their performance can nevertheless differ. In a synthetic regression setting where the assumed models match the data-generating process and target inputs are informative about the shift, tilting reduces mean set length by about $30\%$ relative to weighting alone, with both methods attaining coverage near nominal. Tilting can instead cause substantial coverage losses in synthetic classification and in regression when target inputs provide little information about the response shift. Real-data experiments also show no consistent benefit. Good coverage from weighted calibration alone does not ensure that adding predictive tilting will preserve coverage. Deciding when to apply this additional adjustment using only source labels and target inputs remains an open problem.

0.41

When do data mixtures improve scaling laws? Insights from high-dimensional regression

cs.LG stat.ML ·2026-09-29 ·Diyuan Wu, Lehan Chen, Theodor Misiakiewicz, Marco Mondelli
similarity 0.56
Abstract

Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an extensive literature on data mixing and reweighting, existing work is largely empirical and it remains unclear when auxiliary data genuinely improves scaling laws rather than merely providing more samples. To gain insight into this question, we study a high-dimensional mixed-data regression model with a shared regression function, heterogeneous covariances and noise levels, and dataset sizes that may grow at different rates. We establish the minimax risk under an ellipsoidal parameter constraint for the general covariance structure and derive deterministic equivalents for the test error of ridge regression under commutative covariances. We then specialize to a target domain and an auxiliary domain with aligned power-law covariance spectra, where the theory yields explicit scaling laws in terms of spectral decay, target regularity, and the relative growth of the two datasets. These laws identify regimes in which combining data mixtures provably yields a faster scaling rate than using either dataset alone. In particular, improving the scaling law requires a specific interplay between spectra and relative sample sizes of the domains. Our numerical experiments on language models exhibit the same qualitative phenomenon: appropriate data mixtures yield a faster decrease in target-domain test loss than training on either domain alone.

0.44

Small area learning

cs.LG stat.MEmath.ST ·2026-10-11 ·Katarzyna Reluga, Nicola Salvati, Setareh Ranjbar, Nikos Tzavidis, Mark van der Laan
similarity 0.56
Abstract · 53 pages, 1 figure, 10 tables

Small area estimation (SAE) traditionally borrows strength across domains through a prespecified working model. We introduce small area learning (SAL), a learner-agnostic framework that formulates SAE as a multi-target semi-supervised learning problem and treats outcome prediction as a nuisance task. Areas may differ through unrestricted covariate shift and structured conditional shift, with information shared through common and area-specific components. We develop standard and cross-fitted estimators for low-complexity and flexible learners, unifying synthetic, model-based, and fully residual-corrected estimation. For area means, we establish consistency and asymptotic normality and provide variance estimators and confidence intervals. We extend the theory to settings with only aggregated population covariates and characterize the resulting aggregation bias. SAL reveals a bias--variance trade-off: partial residual correction can reduce mean squared error, whereas full correction yields estimators robust to working-model misspecification, potentially at higher variance. Simulations yield RMSE reductions of 34\%--49\% relative to direct estimation, with the best configuration varying across settings. An empirical application favours a cross-fitted estimator using a flexible learner under the chosen validation criterion. The numerical studies also illustrate how shrinkage can produce overly optimistic confidence intervals at the expense of strong modeling assumptions.

0.38