CS Paper Radar

An Impossibility Result for ETF Target Geometry, and an Equinorm Remedy for Imbalanced Analytic Continual Learning

Berk Sezer

cs.LG ·submitted 2026-10-17 19:48 UTC · v1
published

Abstract

Analytic continual learning (ACL) methods such as ACIL and GACL fit a ridge-regression classifier head on frozen features via the Sherman-Morrison-Woodbury identity, achieving exemplar-free, exact updates. A separate line of work, Progressive Neural Collapse (ProNC), shows standard continual classifiers benefit from a progressively expanding Simplex Equiangular Tight Frame (ETF) target. It is natural to ask whether fusing the two ideas, exact analytic updates and ETF targets, would yield a better closed-form continual learner. We show this is not the case as the fusion turns out to be identical to plain one-hot ACIL. Further, Simplex ETFs are not nested across class counts, so the literal "progressively expand the ETF" construction stops yielding an ETF as new tasks arrive, and it introduces catastrophic forgetting ($4\times$ that of a classic ACIL head). We therefore separate an ETF's two defining properties, equiangularity and equinormality, and apply each on its own to the classifier weights rather than the targets, where the invariance above does not hold. Equinorm post-processing of the fitted weights recovers both tail-class and overall average accuracy across the long-tailed regime from 5:1 to 100:1 imbalance (e.g. +32.0 tail-accuracy points and +16.0 average-accuracy points at 100:1 on CIFAR-100/ViT), transferring in sign across two backbones and two datasets, at a mild cost of 5.4 average-accuracy points on CIFAR-100/ViT on near-balanced streams (below $\approx 3{:}1$). Equiangular geometry, by contrast, is exactly inert on the target side but actively harmful on the weight side, usually losing to plain one-hot. Hence, the generally useful part of Neural Collapse geometry in Analytic Continual Learning is equinorm post-processing of the weights, becoming more useful as class imbalance grows; and equiangularity is at best a no-op and induces catastrophic forgetting at worst.

Comment
Accepted at the NeurReps 2026 Workshop (Extended Abstract Track), NeurIPS 2026

Closest papers

Nearest neighbours by meaning across the whole index.

DuoCL: Dual-Path Online Continual Learning

cs.LG ·2026-11-17 ·Jebacyril Arockiaraj, Dhruv Parikh, Rajgopal Kannan, Viktor Prasanna
similarity 0.66
Abstract · Technical Report

Online continual learning (OCL) requires learning from a data stream whose arrival rate is independent of model training speed. When training cannot keep pace, computationally expensive gradient-based methods update less frequently, limiting adaptation. Analytic methods reduce this overhead, but their frozen backbones cannot adapt representations to changing distributions. We propose DuoCL, a dual-path OCL framework whose fast analytic path accumulates feature statistics from every incoming minibatch while its slow path periodically adapts the backbone. The slow path adjusts its sample count to the available compute budget without interrupting fast-path learning. Feature alignment compensates for drift in the accumulated classifier statistics. Across four datasets and three computational budgets, DuoCL improves online accuracy by an average of 9.67 percentage points over the strongest gradient-based baseline in each setting. DuoCL consistently outperforms analytic learning under distribution shifts, with gains of up to 4.00 percentage points on CIFAR-C, and achieves higher accuracy than prior methods across disjoint class-incremental, domain-incremental, and Si-Blurry settings.

0.27

SPARCL: Spectral Partitioned Analytic Continual Learning

cs.LG ·2026-08-21 ·James Hartley, Zeropy Surio, Daniel Whitmore, Hannah Clarke, Thomas Reed
similarity 0.66
Abstract · arXiv admin note: This submission has been withdrawn by arXiv administrators due to incorrect author affiliation listing

Analytic continual learning has emerged as a strong exemplar-free alternative to gradient-based class-incremental learning because it replaces iterative optimization with closed-form ridge updates. Yet the usual forgetting narrative, centered on stochastic gradient overwriting, does not explain why analytic methods still drift on old classes despite exact recursive solvers. We identify the culprit as spectral interference: the joint ridge classifier for all tasks shares the inverse autocorrelation operator $(R+λI)^{-1}$, so incoming task samples that load onto old dominant eigendirections dilute the spectrum and perturb old-class logits even when old labels are never revisited. Based on this view, we propose SPARCL, a spectral partitioned analytic continual learner that decomposes the running autocorrelation into a high-energy core and a residual complement, freezes old-class classifier components in the core subspace, and updates only the residual block through recursive least squares with an optional residual random-projection expansion. This yields a simple closed-form update with a provable invariance guarantee for the core contribution of old logits. Across CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A under a frozen ViT-B/16 protocol, SPARCL closes most of the gap from classical analytic learners to strong representation matchers, while remaining complementary to sparse feature-decorrelation approaches such as Fly-CL.

0.22

One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning

cs.CV cs.LG ·2026-08-31 ·Yunxiang Fu, Meng Lou, Yizhou Yu
code/data similarity 0.63
Abstract

Class-incremental learning (CIL) requires a model to incrementally learn tasks that contain new classes without accessing earlier training data while preserving the ability to recognize all seen classes. Recently, pretrained-model-based approaches have become prevalent by adapting a frozen backbone with additional lightweight trainable modules. Existing methods, however, exhibit limitations: task-specific adapters learn explicit per-task representations but are parameter- and computation-inefficient, while LoRA-based merging methods combine per-task LoRA parameters into a single model whose static aggregated weights cause representation interference during inference. To address these problems, we present \textbf{FACET}: task-conditioned \textbf{F}e\textbf{A}ture transformation with \textbf{C}ondition\textbf{E}d feature consis\textbf{T}ency, achieving excellent parameter efficiency while producing highly discriminative features during inference. When continually trained on a task sequence, FACET learns a single shared adapter that employs a dynamic task-conditioned feature transformation, shaping the overall feature distribution of the adapter into a mixture of overlap-reduced task-specific components. On the other hand, we propose an efficient replay-free task-conditioned feature consistency loss, aiming to mitigate catastrophic forgetting of the learned mixture distribution in the adapter's feature space. Even when maintaining only a single adapter, FACET demonstrates robust scalability. On both very long task sequences (e.g., 200 tasks) and standard short task sequences (e.g., 20 tasks), our method achieves superior performance while using significantly fewer trainable parameters and GFLOPs. The code will be made open source upon acceptance.

0.46

Out-of-Distribution Detection in Continual Learning

cs.LG cs.CV ·2026-10-19 ·Nimeshika Udayangani, Flora D. Salim, Christopher Leckie, Sarah Erfani
top venue similarity 0.61
Abstract · Accepted at the 40th Annual Conference on Neural Information Processing Systems (NeurIPS 2026)

Out-of-distribution detection in continual learning (OOD-CL) is highly challenging, as it requires (i) learning a sequence of tasks without catastrophic forgetting, and (ii) identifying unknown samples (OOD) without access to known (in-distribution (ID)) data from past tasks, where standard confidence-based OOD scores tend to degrade. Existing methods largely rely on a rehearsal buffer of past data or assume closed-world settings, limiting their scalability and reliability in real-world scenarios. In this paper, we propose a novel OOD detection framework for rehearsal-free continual learning, inspired by Kolmogorov-Arnold Networks (KAN), which mitigate catastrophic forgetting via their inherent local neuroplasticity. To better leverage KAN in OOD-CL, we introduce a task-adaptive classifier architecture, TA-KAN, which mitigates recency bias and enhances ID--OOD separability through task-specific control of activation locality. Furthermore, we propose a geometry-guided OOD score that complements TA-KAN classifier confidence without relying on past ID data. Our method significantly improves OOD detection while maintaining strong ID classification performance, achieving state-of-the-art results in OOD-CL.

0.32

Formalizing Forgetting: Theory of Forgetting for Online Continual Learning

cs.LG ·2026-10-10 ·Renjie Cao, Yifei Cheng, Li Shen
similarity 0.61
Abstract

Online continual learning (OCL) requires a learner to acquire a sequence of tasks in a single pass over the data, and catastrophic forgetting of earlier tasks remains its central challenge. Despite the many methods proposed to mitigate forgetting, the theoretical understanding of forgetting in OCL is still limited: existing analyses mostly focus on linear, overparameterized, or neural tangent kernel regimes, and it remains unclear which factors govern forgetting and how they affect it. To fill this gap, we analyze single-pass Shuffle SGD under strongly convex and general convex objectives and derive explicit upper bounds on the expected final forgetting. Further analysis of these bounds shows how the per-task data amount, task similarity, and task ordering enter them, and that their effects can differ across objective classes. Moreover, experiments with neural networks on Split-CIFAR-100 and PMNIST show that the trends suggested by our theory also appear in practice, indicating that these insights extend beyond the convex setting.

0.20

Stability-Plasticity Balance via Singular-Vector Selection in LLM Continual Learning

cs.LG cs.AI ·2026-10-08 ·Lingxiang Wang, Hainan Zhang, Liang Pang, Hongwei Zheng, Zhiming Zheng
similarity 0.61
Abstract

Domain-specific continual adaptation of LLMs risks catastrophic forgetting, creating a fundamental tension between acquiring new capabilities and preserving those learned during pretraining. PEFT mitigates this problem by restricting the number of trainable parameters, but existing methods lack a principled unit for deciding where plasticity should be allocated and stability should be preserved. We identify the singular-vector channel as a natural unit for managing this trade-off. Each channel represents an input-output transformation, which can be updated to acquire new knowledge or fixed to preserve pretrained capabilities. Based on this perspective, we introduce SVC, a parameter-efficient continual-learning method that selectively updates Singular-Vector Channels. Before fine-tuning, SVC uses domain-specific data to estimate each channel's adaptation benefit and a fixed public general-domain corpus only as a history activation proxy for estimating forgetting cost. It then adaptively selects trainable channels based on these scores via knee-based cost screening, Pareto-front filtering, and Otsu thresholding. Experimental results across four LLM families and eight downstream tasks show that SVC better preserves pretrained capabilities while achieving strong downstream performance relative to existing PEFT baselines. Further analysis of channel scoring and selection demonstrates that selective plasticity at the singular-vector-channel level enables effective continual LLM adaptation.

0.19

DROD: Dual-Ratio Ordered Dropout for Federated Continual Learning on Heterogeneous Devices

cs.LG cs.DC ·2026-10-18 ·Berk Sezer, Özgür Soysal
similarity 0.60
Abstract · 18 pages

Federated learning for capacity-heterogeneous clients and federated continual learning are each well studied on their own, but their intersection -- clients holding differently sized models while every client's data stream also drifts over time -- remains underexplored. We study this setting directly, training a ResNet-18 fleet from scratch on Split-CIFAR-100 (10 tasks, class-incremental), and introduce DROD (Dual-Ratio Ordered Dropout), which builds on FjORD's Ordered Dropout by using two independent reduction axes to extract sub-models: a shortcut ratio and a branch ratio. On a shared parameter/FLOP budget ladder, DROD achieves the highest mean accuracy among the heterogeneous-model federated learning strategies we compare. Its advantage over the two strongest baseline geometries persists across every replay-buffer size we test and under ER, DER++, and BiC; under EWC, all three geometries collapse alike.

0.16

From Joint Risk to Decision Invariance: Decision-Preserving Regularization for Exemplar-Free Class-Incremental Learning

cs.CV ·2026-11-16 ·Qi Zhu, Ziang Gan, Wanting Zhang, Libao Zhang
code/data similarity 0.60
Abstract

Exemplar-free continual learning seeks to minimize joint risk over all seen tasks, yet historical data are unavailable for direct optimization. We observe that, for classification, preserving the model's decisions on previously learned classes helps retain old knowledge. This motivates Decision-Preserving Regularization (DPR). Penalizing changes in all pairwise old-class logit margins reduces algebraically to a centered relative-logit penalty. We further apply spectral whitening to remove the classifier's spectral scale bias. Because the global classifier supplies the decision constraints for the next stage, it must remain calibrated across all seen classes despite the drift of stored old-class statistics during representation updates. We therefore fit a closed-form affine map from paired current-task features to transport old-class Gaussian means and covariances into the updated feature space, then calibrate the global classifier using these distributions and current-task features. Across CIFAR-100, Tiny-ImageNet, ImageNet-Subset, and ImageNet-1K under cold-start protocols, comparisons with state-of-the-art methods show that DPR achieves higher accuracy in most settings while remaining highly efficient. Code is available at https://github.com/bnufw/dpr.

0.32