CS Paper Radar

Effect of Spherical Parameter Space on Number of Non-Forgetting Tasks and Capacity of Continual Learning

Nazanin Mirhosseini

cs.LG cs.ITstat.ML ·submitted 2026-10-12 22:13 UTC · v1
theory

Abstract

We propose a continuous learning formulation for $T$ non-forgetting tasks with the spherical parameter space $\mathbb{S}^{d-1}(\sqrt{d})$ in asymptotic regime $d\rightarrow \infty$. Each task $i$ has an optimum $\bmΘ_i$, minimizing its corresponding loss function $R_i(\bm{w})$ on sphere. We reduce each task $i$ to an acceptable loss region modeled by a geodesic ball $B_{\bmΘ_i}(r_i(d))$ on $\mathbb{S}^{d-1}(\sqrt{d})$ centered at $\bmΘ_i$ with radius $r_i(d)$. To model task dependencies (similarities), task optima $\bmΘ_i$ are distributed by a Neyman-Scott clustered point process $Φ$. Cluster centers $\{\bm{Y}_j\}_{j=1}^{N_p}$ form a homogeneous Poisson point process on sphere, while intra-cluster task optima follow a von Mises-Fisher distribution with mean $\bm{Y}_j$ and concentration $κ$. Each task $t\in \{1, ..., T\}$ learning algorithm is considered to be a channel $Q_{t}(\mathrm{d}\bm{w}_t | \bm{w}_{t-1}, \mathcal{K}_{t-1}(\bmθ_{1:t-1}), \bmθ_t)$, where $\bm{W}_t$ is the learned parameter of task $t$ and $\mathcal{K}_{t-1}(\bmΘ_{1:t-1})=\bigcap_{i=1}^{t-1}B_{\bmΘ_i}(r_i(d))$. Since in continual learning, the learner does not need to reach $\bmΘ_t$ exactly to successfully learn task $t$, to avoid forgetting as well as reliable learning, it only needs for the learned parameter vector $\bm{W}_{t}$ to enter $B_{\bmΘ_t}(r_t(d)) \cap \mathcal{K}_{t-1}(\bmΘ_{1:t-1})$. Using this model, we derive an upper bound on $T$ ensuring $\mathbb{P}[\mathcal{K}_{T}(\bmΘ_{1:T}) \neq \emptyset] \rightarrow 1$ as $d\rightarrow \infty$ and $κ/d \rightarrow α$, where $α>0$ is a constant. Finally, for $T$ tasks sampled uniformly from $Φ$, we derive an upper bound for total information theoretic capacity under the non-forgetting constraint for each task.

Closest papers

Nearest neighbours by meaning across the whole index.

Dimension-Optimal Dynamic Regret in Exp-Concave Online Learning

cs.LG math.OC ·2026-10-14 ·Yanyi Li, Wendao Wu, Haihan Zhang, Qinshi Wang, Chenheng Zhang +4 more
theory similarity 0.58
Abstract

Tracking changing comparators is a central problem in online learning. For exp-concave losses, previous work established the $T^{1/3}P_T^{2/3}$ dependence on the horizon $T$ and Euclidean path length $P_T$ for fixed dimension, while the optimal cost of dimension remained unresolved. We resolve this question by establishing a sharp $d^{1/3}$ curved dimension factor, improving the $d^{2/3}$ factor of recent switching and interval reductions [1, 2]. To achieve this improvement, we develop a proper, full-information learner on compact convex subsets of $\mathbb{R}^d$ that adapts to the unknown path length and, under fixed dimension-independent diameter, gradient, and curvature parameters, guarantees $$\operatorname{Reg}_T(u_{1:T}) \le \widetilde{O}\left(d+\min\left\{d^{1/3}T^{1/3}P_T^{2/3},\sqrt{TP_T},T\right\}\right).$$ A matching lower bound on a diameter-one Euclidean ball establishes the minimax frontier up to logarithmic factors when $T\gtrsim d^2$, including the transition from the curved rate to larger path budgets. The lower bound already holds for losses that depend on only one coordinate per round, isolating an unavoidable cost of learning across directions. Our upper bound follows from square-root Gaussian KL control of comparator movement and a cancellation argument that pays for history restarts. The curved learner maintains $O(\log^2 T)$ Gaussian states, uses one gradient per round, and runs in time near-linear in $T$ under a Mahalanobis-projection oracle.

0.14

Task Vector Descent: Learning from Non-IID Batches

cs.LG cs.CL ·2026-10-04 ·Anton Baumann, Jonas Hübotter, Zeynep Akata, Andreas Krause
hot direction similarity 0.58
Abstract

A central challenge in continual learning is to acquire new knowledge without forgetting what the model has already learned. This challenge appears in language model training when training data comes from various domain-, user-, or task-specific distributions that are encountered unevenly over time. In such settings, successive minibatches are temporally clustered by distribution instead of being sampled i.i.d. from the overall data mixture. Training on temporally clustered data induces a stability-plasticity tradeoff. Adapting the model to the active distribution can improve the model on the active distribution but may lead to a performance degradation on data it previously trained on. We find that this tradeoff intensifies with longer exposure to the same distribution. We therefore ask if the parameter displacement produced by such a sequence (the task vector) should be fully retained or applied partially. We compare applying the full displacement ($λ=1$) with partial integration, which scales the task vector by $λ$ before applying it to the continuing model and scales the optimizer state by the same coefficient. Across continual pretraining, pretraining from random initialization, supervised post-training, and reinforcement post-training, we find that intermediate values of $λ$ often improve average continuing-model performance relative to full integration, particularly after longer same-distribution sequences. In continual-pretraining experiments with both controlled streams and naturally defined adaptation sequences, task-vector scaling outperforms full integration at the matched learning rate, showing that its benefits are not reproduced by learning-rate scaling alone.

0.27

Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability

cs.LG ·2026-10-22 ·Guodong Zheng, Yan Sun, Peng Wang, Enneng Yang, Jiafei Lyu +3 more
similarity 0.57
Abstract

Understanding catastrophic forgetting, in which learning new tasks degrades performance on previously learned tasks, remains a fundamental challenge in continual learning (CL). Existing theoretical analyses often rely on specific model or task structures and offer limited insight into how iterative optimization affects forgetting. To address this gap, we develop an algorithmic stability framework for expected population forgetting under SGD without prescribing a particular model architecture. Specifically, we decompose forgetting into intra-task and cross-task generalization gaps and empirical forgetting. To bound the generalization gaps, we track how a sample perturbation affects updates within a task and propagates through subsequent tasks. We consider constant, intra-task decaying, and continuously decaying learning-rate schedules. Comparing the two decaying schedules allows us to characterize how restarting learning-rate decay at task boundaries affects the propagation of historical sample perturbations. For empirical forgetting, we derive bounds for all three schedules by retaining cumulative gradient alignment along the optimization trajectory, without requiring convexity. Combining these results yields upper bounds on expected forgetting that capture the effects of training set size, the number of updates per task, the total number of tasks, learning-rate schedules, and cumulative gradient alignment. Finally, we conduct experiments on Split CIFAR-100 and Split MiniImageNet to examine the empirical implications of our theoretical findings.

0.14

Loss-Parameterized Fisher Width Along Learning Trajectories

cs.LG math.ST ·2026-08-21 ·Vu Khac Ky
similarity 0.57
Abstract · 44 pages, 2 figures

Fisher width measures the Gaussian width of a probe set after deformation by the local Fisher geometry. We study its evolution along learning trajectories and ask when training loss can serve as an effective coordinate for this quantity. We first derive an exact trace--shape factorization and a deterministic stability bound for fixed compact probes. In a population Gaussian-teacher logistic model, the teacher-aligned state is extremal on every loss level below $\log 2$: it has minimal parameter norm and maximizes both Fisher trace and Euclidean-ball Fisher width. We then show that population gradient flow asymptotically selects this branch, with explicit rates for the aligned and orthogonal coordinates. This yields, for $d\geq2$, \[ \frac{w_F(B_2^d;θ(t))} {\sqrt{L(θ(t))}} \longrightarrow \frac{\sqrt6}π\mathbb E[χ_{d-1}]. \] Controlled full-Fisher experiments support the matched-loss branch and the population predictions. In a nonlinear MLP with a diagonal model-Fisher approximation, GD and SGD remain close at matched loss, whereas Adam follows a substantially displaced branch; the fixed probes tested retain highly similar temporal shapes. These results support a branchwise, rather than universal, loss parametrization of Fisher width.

0.28

Geodesic-SVDD Latent Mixup for One-Shot Distributed Tabular Learning

cs.LG ·2026-10-24 ·Andrea Moleri, Christian Internò, Markus Olhofer, Barbara Hammer, Ali Raza
published similarity 0.55
Abstract

Distributed Learning on structured data is a key and widely adopted approach in privacy-sensitive sectors, ranging from financial fraud prevention to federated healthcare diagnostics. While Gradient Boosted Decision Trees are widely used in centralized settings due to their efficiency, they are incompatible with standard gradient-based federated learning protocols because of their non-differentiability. Conversely, standard deep classifiers that are amenable to federated optimization often suffer from catastrophic forgetting under label distribution skew. In this work, to address these gaps we propose Geodesic-SVDD, a framework for Distributed Spherical Prototype Learning that recasts each client's task as one-class density estimation on the hypersphere, so that local models can be aggregated as probabilistic manifolds in a single round. Specifically, for each client, we employ a Residual Spherical Encoder and Geodesic Latent Mixup that enforces strict hyperspherical constraints ($\|φ(x)\| \equiv 1$), preventing the manifold collapse observed in linear approximations. Furthermore, leveraging this insight, we propose a novel One-Shot "Product of Experts" protocol that enables privacy-preserving aggregation without iterative synchronization. Finally, we mechanistically analyze the trade-off between interpolation strategies, demonstrating that while linear mixup offers an enticing computational shortcut, it destabilizes optimization gradients by orders of magnitude ($10^9\times$). Across five tabular benchmarks and four levels of label skew, Geodesic-SVDD attains the highest mean AUROC among seven federated methods, outperforms federated tree ensembles by up to 21 AUROC points on Covertype, and its One-Shot variant matches centralized Deep SVDD under IID and mild skew with a single communication round.

0.44

PaLoRA: Paced Low-Rank Adaptation for Continual Learning

cs.LG ·2026-10-03 ·Yuxuan Li, Fanhu Zeng, Hao Tang
top venuecode/data similarity 0.55
Abstract · Accepted at NeurIPS 2026. Code: https://github.com/liyuxuan-github/PaLoRA

LoRA-based continual learning methods mitigate catastrophic forgetting through various mechanisms, yet nearly all complement these with small learning rates as a heuristic to restrict gradient scaling magnitude. Such fixed heuristics lack theoretical guidance on how the strength of this restriction should evolve as tasks accumulate. We reveal that even under directional constraints such as nullspace projection, finite-precision updates inevitably leak into the subspace of accumulated prior knowledge along multiple directions. While small learning rates attenuate such leakage, they cannot prevent the accumulated forgetting from intensifying as the effective rank of historical knowledge grows. We show that the optimal magnitude restriction should adaptively increase with this effective rank to balance stability and plasticity, i.e., preservation of previous knowledge and acquisition of new task information. Under an anisotropic leakage model, we derive a pacing law $s^*=\sqrt{R/c}$ that characterizes the optimal scaling of gradient steps, i.e., the magnitude restriction itself, where $R$ is the effective rank of past updates. Based on this insight, we propose PaLoRA, which compresses historical knowledge via adaptive SVD truncation, projects gradients onto the nullspace of prior tasks, and applies rank-aware adaptive pacing. Experiments demonstrate consistent improvements over prior methods, with particularly strong performance in long-horizon settings, achieving substantial gains of 4% accuracy on challenging 50-task ImageNet-A and ImageNet-R benchmarks.

0.74

Fast Multidimensional Approximate Agreement with Optimal Resilience Using Ball Validity

cs.DC ·2026-09-07 ·Tijana Milentijević, Stefan Schmid
theory similarity 0.54
Abstract

Multidimensional approximate agreement requires $n$ processes with inputs in $\mathbb{R}^d$ to output vectors close to each other, despite up to $t$ Byzantine faults. Under convex validity, outputs must lie in the convex hull of the correct inputs, which leads to resilience thresholds that grow with the dimension. We instead study Minimum Enclosing Ball (MEB) validity and its multiplicative relaxation $c$-MEB validity. Our first contribution is Adaptive MEB Contraction, a coordinate-free algorithm that, in the synchronous model with $n>(d+1)t$ contracts the correct MEB radius by $1/\sqrt2$ per round for $α=1$ while satisfying $\sqrt2$-MEB validity. We also give an example showing that this bound is tight for our analysis. Our main technical contribution is a dimension-free inflation theorem: if every $β$ balls in a finite family of Euclidean balls have a common point, then inflating each radius by $\sqrt{β/(β-1)}$ guarantees a common intersection. Applying the theorem to the candidate balls defining the local MEB-safe areas results in a synchronous algorithm with optimal resilience $n>3t$, contraction factor $\sqrt3/2$ and $\sqrt6$-MEB validity. To the best of our knowledge, this is the first multidimensional approximate agreement algorithm with optimal resilience, constant $c$-MEB validity and coordinate-free contraction. We further extend the approach to the asynchronous setting. Without inflation we obtain resilience $n>(d+2)t$, contraction factor $\sqrt{2/3}$ and $\sqrt6$-MEB validity, whereas with inflation for $n>4t$ we get the contraction factor $\sqrt{15}/4$ and $2\sqrt{10}$-MEB validity. Finally, we compare our guarantees with existing algorithms, including Minimum-Diameter Averaging (MDA), for which we derive MEB-validity guarantees. Our algorithms achieve strictly better resilience while providing substantially stronger MEB-validity guarantees than MDA.

0.44

Formalizing Forgetting: Theory of Forgetting for Online Continual Learning

cs.LG ·2026-10-10 ·Renjie Cao, Yifei Cheng, Li Shen
similarity 0.54
Abstract

Online continual learning (OCL) requires a learner to acquire a sequence of tasks in a single pass over the data, and catastrophic forgetting of earlier tasks remains its central challenge. Despite the many methods proposed to mitigate forgetting, the theoretical understanding of forgetting in OCL is still limited: existing analyses mostly focus on linear, overparameterized, or neural tangent kernel regimes, and it remains unclear which factors govern forgetting and how they affect it. To fill this gap, we analyze single-pass Shuffle SGD under strongly convex and general convex objectives and derive explicit upper bounds on the expected final forgetting. Further analysis of these bounds shows how the per-task data amount, task similarity, and task ordering enter them, and that their effects can differ across objective classes. Moreover, experiments with neural networks on Split-CIFAR-100 and PMNIST show that the trends suggested by our theory also appear in practice, indicating that these insights extend beyond the convex setting.

0.20