CS Paper Radar

Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysis

Chuang Ma, Tomoyuki Obuchi

cs.LG ·submitted 2026-09-30 10:30 UTC · v1

Abstract

In federated averaging, local objectives can admit multiple optimal heads, making the aggregate depend on which heads clients return. We study this ambiguity in federated multivariate regression with private backbones and a shared linear head, using an unconstrained feature model (UFM) that treats training-sample features as free variables. We introduce a natural selection rule: each client returns the optimal head closest to the broadcast head. We show that global minimization with a vanishing proximal penalty on the head realizes this rule. When the clients' optimal Gram matrices and the initial shared Gram matrix are positive definite, the shared Gram matrix follows a closed recursion and converges to the unique Bures-Wasserstein barycenter of the clients' optimal Gram matrices. Even with this alignment, the limit generally differs from the centralized optimal Gram matrix. We decompose this gap into three positive-semidefinite terms arising from differences in client target means, covariance heterogeneity, and averaging the aligned heads. A correction based on a one-time exchange of target means and covariances recovers the centralized optimal Gram matrix in one round under exact local optimization and the same selection rule. We verify these results numerically in the UFM and test its predictions on five tabular and five image regression datasets using deep networks with feature regularization and long local training. In these experiments, ordinary training approaches the predicted barycenter, while a weak proximal penalty improves endpoint agreement and yields trajectories that closely follow the predicted Gram dynamics. The correction moves the final Gram matrices close to the centralized UFM prediction.

Comment
47 pages, 17 figures, 9 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Federated Nonparametric Regression with Heterogeneous Clients

cs.LG stat.ML ·2026-10-23 ·Xiaoran Jiang, Anand N. Vidyashankar
similarity 0.64
Abstract · 123 pages, 8 figures, 6 tables

Whether a federated predictor benefits an individual client depends on both population heterogeneity and estimation uncertainty. We study this question for Nadaraya--Watson regression in a triangular array with potentially different fitting and evaluation distributions. Each client transmits local kernel summaries, allowing the server to construct a covariate-density-weighted federated estimator without centralizing individual observations. A client combines its fitted local predictor with the federated predictor through a prescribed weight, and gain is the paired reduction in squared prediction loss on an independent audit observation. Exact identities characterize both the gain of the realized fitted pair and its repeated-fitting mean. We also establish pointwise inference for the federated estimator. A two-stage analysis of client gains separates fitting variability from audit variability, retains the covariance created when a client's observations enter both its local estimator and the common server estimator, and yields fixed-block simultaneous inference across clients. Robustness is examined under two perturbation models. A Kullback--Leibler neighborhood provides a variance-sensitive lower certificate in a regular positive-gain-variance regime, while an $L_2$ neighborhood yields an exact least-favorable regression direction and robust value. Exact-model, vanishing-variance, and exact-zero-direction boundaries are kept separate because their scalar inputs are nonregular. The framework thus distinguishes decisions about a realized fitted pair from claims about the learning procedure under repeated fitting.

0.23

Characterizing Client-Independent Stationary Bias for SCAFFOLD and FedAvg in Federated Learning

cs.LG math.ST ·2026-08-27 ·Yi-Ping Tang, Guan-Ju Peng
similarity 0.61
Abstract

Even when all clients optimize the same strongly convex objective, the global model parameters produced by constant-step federated learning may still have a stationary mean that differs from the minimizer. Does increasing the number of clients eliminate this bias, or can some components persist? Answering this question clarifies the limits of client averaging. To this end, we study two federated learning algorithms, \Scaf{} and matched \FedAvg{}, under full participation, a sufficiently smooth objective, and independent bounded additive gradient noise. Analyzing global stationary balance and within-round local fluctuations yields multidimensional expansions with a shared $γ/N$ term and explicit client-independent $γ^2$ coefficients, where $γ$ is the step size and $N$ is the client count. The \Scaf{} coefficient combines the ordinary local-update contribution found in \FedAvg{} with an additional persistent-control contribution. Although zero-sum controls cancel from the direct server increment, they change local second moments, which interact with third-order curvature to shift the stationary mean. For fixed dimension $d$ and local-step count $H$, a remainder of $O_{H,d}(γ^2/N+γ^3)$, uniform in $N$ for sufficiently small $γ$, identifies the second-order coefficients along arbitrary joint small-step, large-client limits after removing the shared leading term. Controlled experiments assess the predicted coefficients. In real-data analyses, finite-tail mean displacement can account for a substantial share of averaged-model parameter error, and a post-hoc comparison shows directional agreement with the derived curvature--noise structure.

0.27

Federated Differentially Private Optimization via Low-Rank Preconditioning

cs.LG cs.CR ·2026-10-22 ·Antti Koskela, Tejas Kulkarni, Elias Raninen
hot direction similarity 0.60
Abstract

We consider differentially private (DP) optimization in the federated setting, where a DP estimate of the feature covariance matrix serves as a Hessian approximation for generalized linear models. Existing central DP methods for covariance estimation have communication cost that grows quadratically in the data dimension, while existing baselines with linear communication cost are only locally private and degrade as the number of clients increases. As a remedy, we propose a federated DP-PCA framework that achieves linear communication cost while providing central DP guarantees under secure aggregation. It consists of a DP rank selection step, a DP power method, and a noisy release of the covariance projected onto the selected low-dimensional subspace. We give error bounds for this covariance estimate and for the resulting low-communication preconditioned DP gradient descent. Empirically, our proposed DP optimization method matches a centralized baseline with quadratic communication cost at a fraction of the communication.

0.31

Personalized Projected Ascent for Federated Stackelberg Optimization

cs.LG ·2026-10-15 ·M Yashwanth, Sharannya Ghosh, Saikiran Bulusu, Anirban Chakraborty
similarity 0.59
Abstract

Federated minimax problems, such as distributionally robust optimization and adversarial training, include an inner maximization variable built from each client's local data. These variables differ in meaning and dimension from one client to the next, so averaging them across clients, as standard federated minimax methods do, is ill-posed. We therefore study a federated minimax problem where the minimization variable $\mathbf{x}$ is shared and aggregated, while each client retains the maximization variable $\mathbf{y}_i$ (personalized). Because these applications constrain $\mathbf{y}_i$ to a compact convex set, we analyze projected ascent directly. We show that, in the nonconvex-concave regime, this projected personalized scheme attains the standard sublinear rate. In the nonconvex-PL regime, however, it hits a stepsize-invariant noise floor due to stochastic $\mathbf{y}_i$ updates. To mitigate this, we propose VR-SGDA, incorporating a variance-reduced ascent step where each client maintains a private variable snapshot. By evaluating current and snapshot gradients on the same sample, the constant variance is replaced by a vanishing drift under a two-timescale regime. VR-SGDA eliminates the noise floor, attaining sublinear leading rates while preserving personalization and projection. Experiments on subpopulation-robustness and NLP adversarial training confirm both the worst-group robustness benefits and the two-timescale tracking mechanism.

0.13

Communication-Efficient Distributionally Robust Federated Averaging with Multi-Source Data

cs.LG math.OC ·2026-11-15 ·Yingzhu Liu, Zhongkui Li, Pengcheng You, Ashish Cherukuri
similarity 0.59
Abstract

Federated learning trains a shared model from distributed client data. The deployment risk is typically specified through two approximations: each unknown local distribution is replaced by its empirical distribution, and the unknown deployment mixture across clients is replaced by empirical or prescribed training weights. Existing robust methods usually address one approximation at a time through local distributional ambiguity sets or worst-case client mixtures. However, joint treatment of both approximations remains underexplored. To address this gap, we construct a global ambiguity set formed by admissible mixtures of client-wise Wasserstein ambiguity sets, yielding a distributionally robust optimization formulation. The construction allows client-dependent ambiguity radii, preserves client-wise separability of the objective function, and yields a high-probability out-of-sample guarantee for the resulting robust problem. We then introduce a penalty-based approximation of this formulation and develop a communication-efficient algorithm called BiDRO-FedAvg, for which we establish a convergence guarantee. We validate the proposed framework through a market-demand prediction simulation.

0.12

Solution-space heterogeneity shapes federated learning dynamics across partial differential equations

cs.LG cs.DC ·2026-09-04 ·Ping Luo, Jiahuan Wang, Ziqing Wen, Kele Xu, Dongsheng Li +1 more
similarity 0.59
Abstract

In natural science applications, federated learning (FL) enables collaborative training of models that approximate solutions to partial differential equations (PDEs) for repeated use in distributed settings. However, the input conditions of FL-PDE training samples involve multiple coupled parameters, making data heterogeneity across clients difficult to analyse. We propose solution-space PDE-Dirichlet, which groups existing solution labels into task-specific bins and distributes samples using the classical Dirichlet allocation rule in FL, providing a common partitioning approach across PDE tasks. Theoretical analysis shows an association between differences in solution-bin composition and gradient disagreement, while experiments on three controlled operators and four public simulation datasets quantify the relationships between this heterogeneity, gradient disagreement and prediction error. Based on this connection, we develop Solution-Bin Controlled Federated Optimization, which uses pooled bin-gradient prototypes to correct the component of client gradients associated with bin composition while preserving the sample-weighted training objective. Experiments on low-viscosity Burgers and Darcy flow show that this correction reduces prediction error compared with federated averaging and narrows the gap from bin-balanced IID training.

0.10

FedRSPO+: A Heterogeneity-aware Algorithm for Decision-focused Federated Learning

cs.LG math.OCstat.ML ·2026-10-06 ·Konstantinos Ziliaskopoulos, Alexander Vinel, Jiaqi Wang
top venue similarity 0.57
Abstract · 10 pages main paper + appendix. Accepted at NeurIPS 2026 main track

Decision-focused learning (DFL) trains predictive models for downstream optimization, but existing methods largely assume centralized data. In cross-silo settings, federated learning offers a natural alternative, yet standard federated methods optimize prediction over decision quality and do not address heterogeneity in downstream objectives or feasible sets. This heterogeneity is especially challenging for DFL because small perturbations in polyhedral problems can cause discontinuous changes in optimal decisions, destabilizing client updates and aggregation. We propose FedRSPO+, a heterogeneity-aware framework for decision-focused federated learning, built on RSPO+, a regularized predict-then-optimize surrogate that smooths the decision map through projection. We show that RSPO+ upper bounds decision error and regret for the regularized decision and, under exact regularization and consistent LP solution selection, for the original LP decision. We further derive cross-client heterogeneity bounds that depend on both objective and feasible-set heterogeneity, vanish at homogeneity, and require no strong convexity. FedRSPO+ uses an annealed, modular training procedure compatible with standard federated personalization and aggregation methods. Experiments on synthetic knapsack, shortest-path, and real-world energy pricing tasks compare against prediction-only federated learning and DFL baselines under varying heterogeneity and communication budgets. Results suggest that smoothing is a useful ingredient for stable collaborative decision learning and provide a heterogeneity-aware foundation for federated DFL.

0.37

Projection-Based Federated Learning on Compact Submanifolds with Geometry-Independent Differential Privacy

cs.LG stat.ML ·2026-10-24 ·Thibault Pautrel, Florent Bouchard, Guillaume Ginolhac, Ammar Mian, Aurélien Bellet
similarity 0.57
Abstract

Federated learning trains a shared model across clients without exchanging their data, yet the updates they transmit still leak information about individual records. Enforcing differential privacy is harder still when the model parameters lie on a Riemannian manifold. Existing methods rebuild the noise added for privacy in the geometry of each new iterate, aggregate through intrinsic maps that often lack a closed form, and prove convergence only in restricted regimes. We introduce DP-RFedProj, a record-level differentially private federated learning method for compact submanifolds that aggregates through closed-form projections and privatises gradients in a fixed ambient geometry. Each client may run any stochastic first-order optimiser, take several local steps, and participate only intermittently. An explicit record-level $f$-DP guarantee holds for the entire training transcript, independently of the manifold, the aggregation scheme and the local optimiser. We establish convergence of DP-RFedProj when each client runs Riemannian SGD locally and the server aggregates by projected averaging, and show that privacy enters the rate through a term scaling with the manifold's intrinsic dimension rather than the ambient one. Synthetic experiments on federated PCA over the Grassmann manifold corroborate these findings. On EEG motor-imagery classification with SPDNet, whose weights lie on the Stiefel manifold, the method outperforms a Euclidean baseline at every privacy budget considered. In both experimental settings, our accounting requires five to six times less noise than that of the closest prior method for the same budget, which translates into higher utility.

0.18