CS Paper Radar

FedFIbOS: Fisher Importance based Optimal Submodelling for Heterogeneous Federated Learning

Yasmeen Afzal, Jeremiah D. Deng, Haibo Zhang

cs.LG ·submitted 2026-09-17 01:41 UTC · v1

Abstract

Heterogeneous federated learning requires clients with diverse computational capacities to collaboratively train a global model, where each client trains a capacity-constrained submodel. Existing methods select submodel parameters using heuristic importance measures---most prominently parameter magnitude---without theoretical justification for why these measures support convergence. We identify a fundamental gap: existing parameter selection criteria lack theoretical grounding in the convergence framework, partial client participation introduces additional estimation effects in the Fisher scores. We propose \textbf{FedFIbOS}: Fisher Importance-based Optimal Submodelling for heterogeneous federated learning, using Fisher Information in a principled criterion derived from minimizing submodel masking error. %We formally establish when magnitude selection is equivalent to Fisher selection fail under non-IID heterogeneous federated learning. We theoretically formulate submodel selection through a Fisher-weighted quadratic masking surrogate and show that the raw Fisher top-$k$ rule implemented by FedFIbOS solves this surrogate under a Fisher-dominant ranking condition. The resulting method retains the convergence structure of the underlying masked federated optimization bound. Fisher scores are efficiently estimated from empirical diagonal Fisher information using squared gradients, enabling stable and adaptive parameter selection without additional optimization overhead. Experiments on CIFAR-10, CIFAR-100, and AGNews under pathological and Dirichlet non-IID settings show FedFIbOS achieves ${\approx}10\%$ higher accuracy than the state of the art, with improvements becoming more pronounced under stronger heterogeneity.

Comment
9 pages, 4 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Diversity-Aware Client Selection for Communication-Efficient Federated Learning

cs.LG cs.AI ·2026-11-06 ·Max Halleryd, Sima Sinaei, Viktor Valadi, Mina Alibeigi, Henrik Abrahamsson +1 more
published similarity 0.67
Abstract · 4 pages, 6 figures, 2 tables. Accepted at SNCNW 2026

Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, making it attractive for privacy-sensitive and bandwidth-constrained applications. A practical challenge in deploying FL is reducing communication overhead while maintaining model performance under heterogeneous, non-IID data distributions. Selecting which subset of clients participates in each training round offers a principled way to address both challenges simultaneously, yet its impact on data diversity and representativeness remains underexplored. In this paper, we investigate three client selection strategies, Power-of-Choice, Fisher information, and Centered Kernel Alignment, evaluating them not only by accuracy and convergence, but through the lens of participation diversity using the Gini coefficient and KL divergence. Under non-IID conditions, restricting participation to 15% of clients per round achieves significant communication savings, and when selection promotes diverse and balanced contributions, it also improves generalization over biased alternatives. This work provides a systematic and reproducible analysis demonstrating that diversity-focused client selection renders communication efficiency and model quality complementary rather than competing objectives.

0.25

FedRSPO+: A Heterogeneity-aware Algorithm for Decision-focused Federated Learning

cs.LG math.OCstat.ML ·2026-10-06 ·Konstantinos Ziliaskopoulos, Alexander Vinel, Jiaqi Wang
top venue similarity 0.66
Abstract · 10 pages main paper + appendix. Accepted at NeurIPS 2026 main track

Decision-focused learning (DFL) trains predictive models for downstream optimization, but existing methods largely assume centralized data. In cross-silo settings, federated learning offers a natural alternative, yet standard federated methods optimize prediction over decision quality and do not address heterogeneity in downstream objectives or feasible sets. This heterogeneity is especially challenging for DFL because small perturbations in polyhedral problems can cause discontinuous changes in optimal decisions, destabilizing client updates and aggregation. We propose FedRSPO+, a heterogeneity-aware framework for decision-focused federated learning, built on RSPO+, a regularized predict-then-optimize surrogate that smooths the decision map through projection. We show that RSPO+ upper bounds decision error and regret for the regularized decision and, under exact regularization and consistent LP solution selection, for the original LP decision. We further derive cross-client heterogeneity bounds that depend on both objective and feasible-set heterogeneity, vanish at homogeneity, and require no strong convexity. FedRSPO+ uses an annealed, modular training procedure compatible with standard federated personalization and aggregation methods. Experiments on synthetic knapsack, shortest-path, and real-world energy pricing tasks compare against prediction-only federated learning and DFL baselines under varying heterogeneity and communication budgets. Results suggest that smoothing is a useful ingredient for stable collaborative decision learning and provide a heterogeneity-aware foundation for federated DFL.

0.37

FedFiOT: Fisher-Weighted Optimal Transport for Model Merging in One-Shot and Few-Round Federated Learning

cs.LG cs.AI ·2026-11-04 ·Quan Minh Nguyen, Hoang M. Ngo, Bhakti Chowkwale, Yasaman Keshtkarjahromi, My T. Thai
similarity 0.66
Abstract

One-shot and few-round federated learning (FL) reduce communication burden by constructing a global model within one or a few communication rounds, but often achieve lower performance than conventional multi-round FL. Without frequent synchronization, client models may drift apart, causing neurons that learn similar features to occupy different positions across clients. At the same time, local data heterogeneity can cause corresponding parameters to differ in their importance to each client's predictions. Together, these challenges cause direct averaging to combine mismatched neurons and assign equal weight to parameters despite differences in their importance. We propose FedFiOT, a Fisher-weighted optimal transport framework for one-shot and few-round FL that leverages Fisher information to quantify parameter importance and guide neuron alignment and model aggregation. FedFiOT jointly optimizes neuron correspondences and global model parameters through alternating minimization of a shared Fisher-weighted objective. Theoretically, we show that FedFiOT's optimization procedure monotonically decreases the layer-wise merging objective and establish a stationarity bound that quantifies the effects of client heterogeneity and nonuniform Fisher weighting.Experiments across multiple datasets demonstrate consistent improvements over competing baselines in both one-shot and few-round federated learning settings.

0.27

Beyond Parameter Space: NTK-Guided Personalized Aggregation for Robust Federated Learning

cs.LG ·2026-08-12 ·Mirko Konstantin, Stefan Zachow, Anirban Mukhopadhyay
similarity 0.65
Abstract

Federated learning (FL) enables collaborative model training across distributed clients while keeping data local. A central challenge is determining which client updates are beneficial for aggregation with respect to each client's target domain. Existing methods typically address this problem in parameter space by comparing model parameters or gradients. However, parameter-space similarity can be a poor proxy for predictive behavior, especially under heterogeneous, non-IID data. Consequently, updates that are misaligned with a client's target domain, including those caused by heterogeneous data or malfunctioning clients, may degrade local model performance. We propose Local Inference Guided Aggregation for Heterogeneous Training Environments to Yield Enhancement Through Agreement and Regularization (LIGHTYEAR), a federated learning framework that performs update selection in function space. LIGHTYEAR uses an NTK-based agreement score to characterize predictive behavior and determine a personalized aggregation set for each client. By relating model parameters to local predictive responses, the Neural Tangent Kernel (NTK) provides a more expressive criterion for update selection than parameter-space similarity alone. Because function-space information is not available before aggregation in conventional centralized FL, LIGHTYEAR uses a peer-to-peer (P2P) topology in which clients exchange updates directly and evaluate incoming models on private validation data. Each client selects only updates that are beneficial for its own target domain and aggregates them using a regularized rule that improves stability under heterogeneity. Across five datasets and nine baseline methods, LIGHTYEAR consistently outperforms centralized FL baselines and existing P2P approaches.

0.04

From Task Mixtures to Specialized Experts

cs.LG ·2026-09-30 ·Hojat Allah Salehi, Mehrdad Mahdavi, Andrew Arash Mahyari, M. Hadi Amini
similarity 0.63
Abstract · 63 pages, 9 figures

In collaborative foundation model fine-tuning, client data is rarely homogeneous. Instead, clients typically possess unknown mixtures of distinct data distributions, or tasks. Conventional federated learning primarily addresses heterogeneity across clients without explicitly resolving latent task mixtures within each client. We study this setting as compound heterogeneity, where data is heterogeneous both across and within clients. We study adaptation over a common frozen representation and show that, when tasks share the same feature geometry, the optimal model for a client's task mixture under squared loss is a convex combination of the optimal models for its underlying tasks. Thus, a single locally trained model represents the client's overall task mixture, while individual inputs may be drawn from different underlying task distributions. This motivates routing inputs to specialized experts, and we show that, when the task optima form a simplex, task-aligned routing achieves lower risk than any single adapted model for genuinely mixed clients. With access to a small set of task-labeled public samples, we derive a convex program to recover task experts and match them to their corresponding tasks. Our routing analysis shows that effective specialization requires input-dependent expert selection aligned with each client's task mixture. Motivated by this analysis, we propose FedSEE. Across our experiments, FedSEE avoids the negative transfer observed in the evaluated baselines and improves performance by 2.9 points overall and 3.7 points for the worst-served quartile.

0.23

FedeRage: Provably Convergent Agnostic Federated Learning under General Client Drift

cs.LG cs.DCeess.SPeess.SY ·2026-09-17 ·Herlock Rahimi, Dionysis Kalogerias
theory similarity 0.62
Abstract

Federated learning (FL) enables collaborative model training without sharing raw data, but its performance degrades under non-IID data and stochastic client participation. Remedies built on classical Federated Averaging (FedAvg) typically presuppose that client participation probabilities are known to the server, which is rarely the case in deployed systems. We first discuss and then characterize the optimization problem that \emph{distributionally agnostic} FedAvg actually solves when participation is entirely unknown, possibly highly skewed, and of variable size across rounds: uniform aggregation is shown to minimize a well-defined stochastic objective, weighted by the participation-induced marginal, at a standard $\mathcal{O}(1/\sqrt{T})$ rate for convex and possibly nonsmooth losses. Building on this characterization, we propose \emph{Federated Risk-Averse Averaging} (\textsc{FedeRage}), a risk-averse extension of FedAvg that embeds the \emph{Conditional Value-at-Risk} (CVaR) into the local objective within a natural distributionally robust optimization (DRO) framework. \textsc{FedeRage} implicitly upweights high-loss and infrequently participating clients while adding only a \emph{single scalar per-client}, and admits an $\mathcal{O}(κ/\sqrt{T})$ rate in which the factor $κ$ is the upper bound on the ``price" of risk aversion. In contrast with aggregation-alignment schemes based on optimal transport, which require the availability distribution as an input, \textsc{FedeRage} remains agnostic to it. Several experiments on three heterogeneous benchmarks indicate consistent improvements over state-of-the-art methods in accuracy, fairness, and convergence speed.

0.22

When Is Shallow Enough? Adaptive Split Federated Learning with Client-Specific Sufficiency Estimation

cs.DC cs.AIcs.LG ·2026-08-16 ·Wenhao Yuan, Chenchen Lin, Wentao Hu, Jian Chen, Jinfeng Xu +2 more
top venue similarity 0.62
Abstract · Accepted by CIKM2026 (Full Research Track)

\textit{Split Federated Learning} (SFL) enables distributed model training by splitting networks between the server and clients. However, under client heterogeneity, the conventional static split strategy may be suboptimal because clients can differ in data distributions, adaptation dynamics, and representation learning progress, making a single split point insufficient to accommodate client-specific training states. In this paper, we propose \textsc{FedSGA}, a \textbf{S}ufficiency-\textbf{G}uided \textbf{A}daptive split \textbf{Fed}erated learning framework that addresses this question through client-specific shallow sufficiency estimation. First, we introduce a client-specific adaptation channel based on private prompt tokens, which tracks local adaptation dynamics separately from the shared backbone and provides a lightweight signal for detecting whether client adaptation remains active. To further avoid repeated online probing over multiple candidate depths, we design a shallow sufficiency estimator that combines cross-client semantic alignment, temporal interface stability, and prompt-state variation to estimate whether the shallowest split is already sufficient. Finally, we introduce a split-compatible interface harmonization module that projects activations from different split depths into a shared semantic space, improving the comparability of heterogeneous client interfaces before server-side prediction. Extensive experiments on multiple heterogeneous benchmarks demonstrate the effectiveness of \textsc{FedSGA} in improving model performance compared with state-of-the-art methods while reducing unnecessary client-side computation.

0.29

FedLoRD: Personalized Federated Learning based on Low-Rank Structures of Model Parameter Differences

cs.LG eess.SP ·2026-10-13 ·Tsutahiro Fukuhara, Junya Hara, Hiroshi Higashi, Yuichi Tanaka
benchmark similarity 0.61
Abstract

We propose FedLoRD, a personalized federated learning (PFL) method based on the low-rank structure of the differences between the global and personalized model parameters. PFL is a branch of federated learning that aims to learn models tailored to the local data of each client under biased local data distributions. In its local training, each client learns a personalized model by regularizing the difference between its model parameters and the global ones sent from the server. The regularization function needs to be designed with priors on the model parameter difference; however, the existing PFL methods rely on empirical choices (e.g., $\ell_1$- and $\ell_2$-norms), which do not take into account the actual characteristics of the difference. The resulting mismatch of priors can lead to suboptimal personalized models. To address this issue, we propose a PFL method whose regularizer is built on an observed characteristic of the model parameter differences: they are approximately low-rank at every layer. In our method, we first formulate a local training problem with nuclear norm regularization on the layer-wise differences. We further rewrite this problem via the factorized representation of the nuclear norm so that it can be efficiently solved by a plain stochastic gradient descent method. Image classification experiments on three benchmark datasets demonstrate that FedLoRD outperforms several existing PFL methods.

0.10