CS Paper Radar

FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learning

Van Truong Vo, Khoa Nguyen, Taehong Kim

cs.LG cs.AIcs.CVcs.DC ·submitted 2026-08-10 07:27 UTC · v1
publishedcode/data

Abstract

Decentralized intelligence systems with heterogeneous devices and limited coordination increasingly rely on decentralized federated learning (DFL). However, DFL suffers from convergence inefficiency under data heterogeneity due to the use of a uniform learning rate (LR) that ignores layer-specific optimization needs. Foundational layers are responsible for maintaining network consensus, while specialized layers adapt to local data characteristics, leading to conflicting gradients and degraded performance under non-IID conditions. To address this fundamental tension, this work introduces FedA2L, a method that dynamically adjusts layer-wise LRs based on model divergence signals. By leveraging local update intensity and network consensus constraints, FedA2L seamlessly integrates into existing DFL protocols without additional communication or coordination. Extensive evaluations across DFL algorithms, various model architectures, and datasets demonstrate that FedA2L achieves up to 4.94 times faster convergence than vanilla DFL and reduces communication rounds by up to 59% compared to scheduler-based baselines. Furthermore, FedA2L exhibits resilience to severe data heterogeneity, larger network sizes, and sparse topologies, reducing communication overhead and establishing it as a versatile optimization tool for resource-constrained or large-scale distributed learning in edge and IoT deployments. The code is released at https://github.com/nclabteam/FedA2L.

Journal ref
Future Generation Computer Systems 186 (2026) 108743
DOI
10.1016/j.future.2026.108743

Closest papers

Nearest neighbours by meaning across the whole index.

Latent Information Sharing for Accelerating Federated Learning

cs.LG ·2026-10-01 ·Seungjun Lee, Ensieh Khazaei, Dimitrios Hatzinakos, Baturalp Buyukates, Sunwoo Lee
similarity 0.72
Abstract

Federated learning (FL) is a communication-efficient distributed learning paradigm. However, client drift remains one of the most critical challenges, hindering the efficient training of a global model. In this study, we propose a novel latent information sharing scheme that directly mitigates data heterogeneity across clients. Our theoretical and empirical results show that sharing a small amount of hidden-layer activations significantly improves training efficiency while preserving convergence guarantees and data privacy. Furthermore, we compare our method with existing FL approaches designed to address client drift, including FedProx, SCAFFOLD, FedPVR, FedProto, and SplitFed, and demonstrate superior model accuracy under a fixed round budget without incurring excessive communication overhead. Overall, this work presents a promising new knowledge aggregation scheme and provides a comprehensive analysis of the impact of activation sharing on federated optimization.

0.30

FedPCD: Priority-Constrained Federated Aggregation

cs.LG ·2026-10-09 ·Yousef Irshaid, Dara Varam, Mohamed I. AlHajri
similarity 0.69
Abstract · 30 pages, 17 figures, 5 tables

Federated learning (FL) seeks to optimize a shared global model from decentralized client data. While effective for average performance, this process can produce unfair performance across clients, particularly under statistically heterogeneous data distributions. Existing fairness-aware and multi-objective FL methods address this through client re-weighting, gradient correction, or common-descent formulations, but often change the global optimization direction or treat client objectives symmetrically. We propose FedPCD, a priority-constrained federated optimization framework that separates the global optimization objective from client protection. FedPCD preserves the conventional federated aggregate as the primary objective, while allowing only the minimum correction necessary to meet client performance constraints. Across heterogeneous FL benchmarks, FedPCD consistently outperforms state-of-the-art baselines, achieving the highest global accuracy across all configurations and leading worst-10% client accuracy in nearly all. FedPCD further preserves competitive trade-offs between global accuracy and client performance variability. Beyond this trade-off, FedPCD demonstrates faster empirical convergence, reaching matching or superior performance in substantially fewer communication rounds.

0.10

Resilience Beyond Stationary Client Unavailability: Unlocking Efficient and Unbiased Federated Learning

cs.LG cs.DCmath.OC ·2026-09-04 ·Ming Xiang, Stratis Ioannidis, Edmund Yeh, Carlee Joe-Wong, Lili Su
similarity 0.68
Abstract · Journal of Machine Learning Research

Due to resource constraints or external and internal uncertainties, clients in real-world federated learning systems are often intermittently available edge devices. In highly dynamic environments, the parameter server lacks prior real-time knowledge of clients' availability, making it challenging to adapt traditional federated learning algorithms to be resilient to uncertainties in client availability. If not carefully addressed, complex client availability can introduce significant bias, potentially harming the performance of the trained model. Most prior work either fails to account for non-stationary client availability dynamics or demands significant memory and computational overhead. This paper aims to develop efficient federated learning algorithms that are provably resilient to heterogeneous and non-stationary stochastic client availability. We propose FedSWE, which admits novel algorithmic structures to (i) compensate for missed computations, (ii) stabilize and diffuse the global updates over rounds, and (iii) evenly mix the local updates through implicit gossiping, despite being agnostic to non-stationary dynamics. Compared with the standard FedAvg, FedSWE introduces light additional memory and computation overhead. We show that FedSWE converges to a stationary point of non-convex objectives while achieving the desired linear speedup property in certain special cases. We corroborate our analysis with numerical experiments over diversified client unavailability dynamics on real-world data sets.

0.19

Bridging Update Coordination and Flatness in Federated Learning

cs.CV ·2026-11-11 ·Chongwei He, Shichen Huang, Jiayun Li, Deyuan Lin, Wangze Jiang +4 more
similarity 0.68
Abstract · 16 pages, 6 figures

Federated learning combines complementary knowledge across clients to alleviate local data limitations without sharing raw samples. However, data heterogeneity can cause conflicting updates and hinder generalization. Update coordination reduces interference among client updates, while flatness optimization reduces sensitivity to parameter perturbations to improve generalization. Existing joint designs largely target specific algorithms, limiting flexible combinations of these approaches. To address this problem, we propose a unified federated learning framework that bridges update coordination and flatness optimization. Clients separately accumulate descent signals and flatness corrections along the same local trajectory. The server coordinates both groups with independent states before fusing their outputs into a global update. Explicit conversion between trajectory signals and model updates enables different coordination strategies to work with compatible local or globally guided flatness mechanisms without changing the inference architecture. Experiments show that the evaluated combinations generally improve average and worst-client accuracy over update coordination alone, demonstrating the effectiveness of the proposed framework in combining these complementary optimization mechanisms.

0.21

Tram-FL: Reducing Communication and Computation Costs through Sequential Model Circulation in Decentralized Federated Learning

cs.LG cs.DCcs.NI ·2026-10-06 ·Kota Maejima, Takayuki Nishio, Asato Yamazaki, Yuko Hara-Azumi
similarity 0.68
Abstract · 12 pages, 7 figures, 5 tables. This work has been submitted to the IEEE for possible publication

Conventional decentralized federated learning (DFL) often focuses on clients, with each client maintaining a model copy, performing updates individually, and undertaking model exchange and integration. While fully leveraging computational resources can shorten training times, it can also lead to significant computational and communication waste. This is especially pronounced with non-independent and identically distributed (non-IID) data, where achieving high model accuracy demands extra resources. This research shifts focus to the model itself, aiming to realize DFL with minimal computation and communication costs. To this end, we propose Tram-FL (Traveling Model Training Mechanism for Decentralized Federated Learning), a mechanism designed to efficiently address these challenges. It sequentially trains a single model by circulating it among nodes. We address the training scheduling problem in model circulation-based training, specifically determining which nodes should update the model and the number of updates to perform. This is approached by considering the model's circulation route and update iteration allocation, for which we propose simple yet effective methods. Additionally, with quantized momentum, Tram-FL achieves high accuracy with fewer model circulations while controlling communication load per transmission. Experimental results show that the proposed algorithm, even with non-IID data, converges to a global model with reduced communication and computation.

0.18

Communication-Efficient Clustered Federated Learning in Non-IID Wireless Networks

cs.LG ·2026-10-28 ·Martin Isaksson, Rickard Cöster, Šarūnas Girdzijauskas, Seif Haridi
similarity 0.67
Abstract · 64 pages (25 main text + appendices)

Current and emerging mobile networks face increasing privacy, scalability, and performance challenges as distributed learning becomes essential for advanced 6G services. These challenges are amplified by the presence of non-IID data, naturally arising in heterogeneous wireless environments, that degrade the performance of standard Federated Learning (FL) approaches. To address non-IID data across clients, we propose a Clustered Federated Learning (CFL) framework that assigns clients to specialized models during training. Data-domain clustering initializes model-based clustering, enabling identification and exploitation of client similarities without sharing raw data while reducing communication cost. Motivated by mmWave beam selection, we evaluate our methods on a wireless simulation dataset (DeepMIMO) and complement it with two non-IID benchmarks (FEMNIST, Rotated CIFAR-10) to isolate different non-IID characteristics. Our best configurations improve accuracy over IFCA by 4.8 pp on DeepMIMO and 10.6 pp on FEMNIST while reducing downlink data volume by 83.1 % on FEMNIST compared to IFCA. We demonstrate that different clustering strategies suit different non-IID characteristics. These results show that our CFL methods enable resource-efficient, and high-performance distributed learning in future 6G networks.

0.28

Decentralized Federated Learning for Heterogeneous Multi-Task Semantic Communication

cs.AI cs.LG ·2026-08-15 ·Lin Yin, Tiejun Lv, Weicai Li, Xi Yu, Xiaoyu He
top venue similarity 0.67
Abstract · 17 pages, 9 figures, Accepted by IEEE Transactions on Communications

Collaborative training in distributed semantic communication (DSC) networks typically relies on decentralized federated learning (DFL). However, pushing topology-agnostic aggregation into heterogeneous, multi-task environments creates a fundamental bottleneck: it drives negative transfer and overconsensus bias (OCB). This paper introduces a personalized DSC framework that cuts off this cross-task interference. At the node level, a policy-driven multi-path routing mechanism separates task-specific features from shared representations to preserve local fidelity. Across the network, we deploy a "communicationwhile- aggregation" protocol. It calibrates a column-stochastic consensus matrix using task affinities. This limits the system to absorbing complementary knowledge while actively blocking mismatched parameter updates. To bound the convergence, we derive a unified Lyapunov drift analysis. We reveal a strict Ushaped trade-off: deeper topological mixing reduces variance but amplifies structural OCB. Resolving this tension yields a closed-form expression for the optimal aggregation depth. We evaluate the proposed framework on NYU-v2, where the results reveal a clear trade-off between insufficient aggregation and excessive topological mixing. At the analytically derived optimal aggregation depth, our method achieves a 4.77% global relative improvement over the no-aggregation baseline and outperforms decentralized FedAvg, FedAMP, and heuristic max aggregation. We further evaluate the framework on Taskonomy and imperfect wireless links to examine the effects of network-size variation and wireless-link reliability.

0.35

Accelerating Sharded Data Parallelism at Scale with Federated Learning

cs.DC cs.AIcs.PF ·2026-09-17 ·Gianluca Mittone, Marco Aldinucci
published similarity 0.66
Abstract

The symbiotic scaling of artificial intelligence models and high-performance computing systems continually creates algorithmic challenges in their convergence. Foundation models (FMs) are a crucial example, requiring months-long training on thousands of cutting-edge GPUs. Sharded data parallelism (DP) is the dominant strategy to accelerate such computations by splitting data and models across multiple GPUs. However, it incurs prohibitive communication overhead when deployed at scale, particularly on multi-tier interconnects with heterogeneous performance. Inspired by the efficient communication principles of federated learning (FL), this work introduces two hybrid algorithms - FL+FSDP and FL+HSDP - interleaving sharded DP with FedAvg-style aggregations. Such approaches decouple large DP deployments into smaller, loosely-coupled federation groups, requiring minimal inter-group traffic while keeping the global batch size bounded by the groups' size. Formal analysis of communication costs and experimental validation prove their scalability and flexibility. A Llama3.1 8B pre-training on 512 A100 GPUs shows that, under identical hyperparameters, FL+FSDP and FL+HSDP achieve up to 8.04 faster data processing and 4.48 lower evaluation perplexity than their counterparts, demonstrating superior computational efficiency and improved model quality. These properties stem from reduced communication overhead and the bounded growth of the global batch size relative to the federation group size.

0.36