CS Paper Radar

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Junpeng Ren, Carlos Misael Madrid Padilla, Yanzhen Chen, Oscar Hernan Madrid Padilla

cs.LG stat.MLstat.ME ·submitted 2026-08-20 16:53 UTC · v1
top venue

Abstract

This paper develops a general transfer learning framework for nonparametric regression with data consisting of multiple groups. Under the assumption that groups share a common structure along with group-specific deviations in additive form, the proposed method employs a two-stage offset learning procedure: the first stage pools data from all groups to estimate an overall mean function, and the second stage estimates offsets for each group, yielding final group-level estimators through additive combination. Upper bounds on the $\mathcal L_2$ error are established for the proposed framework, covering a broad class of nonparametric estimators under mild complexity and noise conditions. When instantiated with deep ReLU networks, explicit convergence rates are derived under hierarchical composition models, demonstrating the ability to overcome the curse of dimensionality. Conditions that enable positive transfer with faster rates are considered, including learning with simpler functions and data augmentation through pooling samples across groups. Various simulations and real-data experiments further validate the effectiveness of the proposed method.

Comment
Accepted at the 43rd International Conference on Machine Learning (ICML 2026)

Closest papers

Nearest neighbours by meaning across the whole index.

Black-Box Knowledge Transfer across Distinct Feature Sets

cs.LG stat.MLstat.ME ·2026-08-10 ·Oh-Ran Kwon, Daeyoung Ham
similarity 0.65
Abstract

Pre-trained black-box predictive functions encode knowledge distilled from massive datasets and extensive computation. However, when the available input features differ from those the black box expects, direct use is infeasible. We introduce a method for transferring predictive knowledge from the black box to a new, heterogeneous input space. Our approach decomposes the target regression function into a transferable component, which the black box can inform, and a non-transferable component, which captures information unique to the new space. We propose a two-step neural network procedure, estimating the transferable component from abundant unlabeled feature pairs that bridge the two input spaces and the non-transferable component from limited labels. We derive prediction risk bounds that improve on those of a non-transfer alternative when the non-transferable component is small or smooth, and the procedure adapts to either case. Under additional conditions, the worst-case risk of our estimator is of strictly smaller polynomial order than the minimax risk of estimation from the labeled data alone. We extend the framework to multiple black boxes, each on its own input space, and show that aggregation can reduce prediction error relative to the best single black box. Simulated and real data demonstrate the practical value of the method.

0.34

Optimal Deep Nonparametric Regression with Uniformly Bounded Lipschitz Constants

cs.LG stat.MLmath.ST ·2026-11-18 ·Jiaxin Shi, Ziyuan Chen, Jian Huang, Hansheng Wang
similarity 0.61
Abstract

Stability under input perturbations is necessary for deep neural network estimators in many applications involving noisy covariates or estimated features. Weight norm constraints are widely used to control sensitivity to input perturbations by bounding the Lipschitz constant of neural networks. However, existing approximation results under fixed bounds on standard norm products are insufficient to establish optimal statistical rates. In this paper, we establish a sharp approximation theory for deep ReLU networks under a split spectral norm product constraint. This theoretical result shows that sufficiently smooth Hölder functions can be uniformly approximated to arbitrary accuracy as the network depth increases, with the norm-product bound held fixed. We then propose a penalized least squares estimator for deep nonparametric regression. The resulting estimator has a uniformly bounded Lipschitz constant while still achieving the optimal nonparametric rate up to a logarithmic factor. As applications, we establish optimal prediction rates for two-stage learning and provide robustness guarantees under adversarial input perturbations. Numerical studies show that the penalized estimator has well-controlled sensitivity to input perturbations and prediction accuracy comparable to that of classical deep network methods.

0.09

Joint Contrastive-Adversarial Representation Learning for Unsupervised Domain Adaptation

cs.LG stat.ML ·2026-10-11 ·Yuling Jiao, Lican Kang, Jerry Zhijian Yang, Zhou Yu, Fusheng Zhou
similarity 0.59
Abstract · 29 pages, 5 figures, and 9 tables

Effective unsupervised domain adaptation (UDA) requires a representation that captures transferable structure from both the labeled source domain and the unlabeled target domain. We introduce Joint Contrastive-Adversarial Representation Learning (JCAR), a two-stage framework that learns a common representation from the union of unlabeled source and target samples by combining contrastive learning with adversarial domain alignment, and then freezes the representation to fit a source-supervised predictor. We establish a non-asymptotic excess-risk bound for this procedure that couples representation and domain-discriminator estimation while separating downstream prediction, with explicit neural-network approximation errors. Under the stated assumptions and common Hölder smoothness $β\geq1$, the bound identifies a sufficiently large unlabeled-sample regime in which the downstream rate is $\widetilde{\mathcal{O}}(m^{-β/(2β+d)})$, where $m$ counts labeled source samples and logarithmic factors are suppressed. The labeled-sample exponent is governed by the latent dimension $d$ rather than the ambient dimension $p$, which enters the pretraining cost. Experiments on Office-31, Office-Home, and DomainNet demonstrate clear transfer gains across diverse domain shifts, with particularly strong performance when representations are learned from scratch.

0.30

Generalized Deep Regression for Repeated Measurements

cs.LG stat.ML ·2026-09-22 ·Kexuan Li
theory similarity 0.58
Abstract

In this paper, we study the estimation of a marginal regression function from independent units with repeated binary, count, or continuous responses using ReLU deep neural networks. In the model, we assume that the dependence is generated by an unobserved random mean function within each unit. We then fit a neural network with a convex generalized regression loss. We show an oracle inequality by separating conditional measurement variation from between-unit variation. In addition, we prove that with $n$ units and $m$ measurements per unit, ReLU networks can attain an integrated mean squared error of order $n^{-1}+(nm)^{-2β/(2β+d)}$, up to logarithmic factors, over $β$-Hölder classes. We also derive a weighted oracle inequality for unequal cluster sizes and a rate for compositionally smooth functions. For pointwise ensemble inference, we give a projection central limit theorem and prove infinitesimal jackknife consistency under an explicit asymptotic linearity condition. Simulations and real data examples are provided to support our theoretical findings and practical implications.

0.31

Two-Stage Score Estimation for Group-Specific Diffusion Models

cs.LG stat.MLmath.ST ·2026-10-14 ·Huitao Yang, Oscar Hernan Madrid Padilla
theory similarity 0.57
Abstract

Generative modeling across related distributions raises a question: how can shared structure support accurate generation for a target distribution with limited observations? We develop a statistical framework for diffusion transfer that analyzes a shared pooled score together with a target-specific offset. Under Besov regularity of the pooled density and a stable hierarchical composition model for the offset, we derive convergence rates that separate the costs of learning these components and identify when a simpler offset yields a faster risk-bound rate than separate fitting. Under suitable boundary and regularity assumptions, we prove that the offset remains bounded and jointly regular as diffusion time approaches zero. These results allow the lower time cutoff to vanish polynomially and provide total-variation guarantees for diffusion models as density estimators. Synthetic simulations and image benchmarks provide a proof of concept for the framework.

0.19

Brownian Heads for Deep ReLU Representations: Activation Mass and the Cost of Same-Sample Selection

cs.LG stat.ML ·2026-09-18 ·Mahdi Mohammadigohari, Nicole Mücke
similarity 0.57
Abstract

Deep representation learning often selects hidden features and fits the final predictor on the same sample, so fixed-feature analysis performed after selection can omit selection cost. We study the conditional empirical Rademacher complexity of deep ReLU representations followed by bounded-norm predictors in additive or Lévy-Brownian RKHSs, termed Brownian heads. For a fixed representation, we derive an exact dual identity and sharp bounds in terms of activation mass, the average norm of the observed hidden vectors. Under same-sample selection, the representation supremum induces a quadratic Rademacher process. Brownian layer-cake and Gaussian-projection identities reduce it to coordinatewise or signed projected threshold traces, separating realized scale from selection complexity. For samples with pairwise-distinct inputs, explicit scalar ReLU families match the finite-trace and VC rates up to universal constants at the realized trace-and-envelope level. Induced-norm contraction also yields architecture-level bounds for rectangular, rank-deficient ReLU networks. Experiments verify the sharp bounds and rates, exhibit a selection gap at fixed activation mass, and assess the predictive feasibility of Brownian heads.

0.09

Deep Barycentric Regression for Optimal Transport Map Estimation and Its Statistical Optimality

cs.LG cs.AIstat.ML ·2026-09-06 ·Kunwoong Kim, Insung Kong, Yongdai Kim
theory similarity 0.57
Abstract

The optimal transport (OT) map provides a geometric transformation for aligning probability distributions and has become a useful tool in machine learning. However, existing estimators of the OT map still exhibit a gap between sharp statistical guarantees and practical parametric estimation based on stable training objectives. Theoretical estimators achieve minimax optimal convergence rates, but they are typically nonparametric and can incur demanding implementation design or inference costs. Practical estimators are parametric and scalable, but their statistical guarantees remain underexplored, and their min-max, adversarial-like training objectives can be sensitive to optimization algorithms. We propose BROT (Barycentric Regression for Optimal Transport), a simple two-step method that first computes the unregularized OT plan and then fits a deep neural network (DNN) to the induced barycentric targets by least-squares regression. Under standard regularity conditions, we prove that the DNN estimator of BROT attains the minimax convergence rate, when the ground-truth OT map is Lipschitz. Numerical studies on synthetic datasets and an image dataset show that BROT provides accurate map estimates, strong target distribution matching, and competitive transport costs, compared to existing estimation methods. Experiments on two downstream tasks, single-cell perturbation prediction and unsupervised domain adaptation, further suggest that the accurate estimation of BROT can translate into stronger task performance.

0.08

Exploring Bad Minima in Transfer Learning

cs.LG cs.CR ·2026-10-30 ·Mina Remeli
similarity 0.57
Abstract · 57 pages, 13 figures, 3 tables. MPhil thesis, University of Cambridge, 2022. Supervised by Ferenc Huszár

Transfer learning is a highly successful learning paradigm in deep learning which leverages knowledge from one task and applies it to another. Despite its success, the necessary conditions for knowledge transfer are unclear and require further research. In this dissertation, our aim is to understand transferability a little better by answering the following question: Does good generalization on a source task necessarily imply transferability to a related target task? We hypothesize the answer is negative, and attempt to find counterexamples via adversarial attacks on the transfer learning procedure. Over three types of adversarial attacks, our empirical results show a strong relationship between generalization on the source task and transferability. These results do not rule out generalization on the source task as a sufficient condition for successful transfer.

0.24