CS Paper Radar

Stability-Constrained Approximation in Spline KANs: Exact Layer Balancing and Budget-Compatible Saturation

Aleksander Tankman

cs.LG stat.MLmath.NA ·submitted 2026-09-14 20:17 UTC · v1
theory

Abstract

Deep spline superposition networks face a tension between approximation order and stability across depth. We study approximation under a hard layerwise Lipschitz budget, and organise it around two quantities: the factorisation stability complexity of a given deep factorisation, and the budget-compatible approximation complexity of a discretisation operator. First, we solve exactly the finite-depth diagonal balancing problem for a fixed chain of nonnegative envelope matrices: the optimal uniform layer budget equals $\|M_{L-1}\cdots M_0\|_{\infty\to\infty}^{1/L}$, attained by an explicit one-pass minimiser, for rectangular layers, with a complete treatment of degeneracies and non-attainment. The optimum can be arbitrarily larger than the Lipschitz constant of the network itself, because passing to envelopes destroys sign cancellation. Second, we give a constructive spline discretisation theorem preserving the budget up to a controlled slack, with an explicit grid threshold. Conversely, for linear spline-valued operators that preserve the budget exactly, we prove budget-compatible minimax lower bounds on classes constrained simultaneously in the first and third derivative norms -- a constraint pair that is forced by the problem and that rules out the usual scaling escapes. Finally, we show that the corresponding layer errors need not cancel under composition: for every operator of the class there is a stable depth-$L$ tower realising a constant fraction of the accumulated error, so the linear-in-depth accumulation of the upper bound is not a proof artefact.

Comment
33 pages. Ancillary code reproduces the numerical constants of Appendix B. Complements arXiv:2604.26444, which constructs deep KAN representations with controlled layer-wise Lipschitz product; here the allocation of that product across layers is solved exactly, and the cost of preserving it under spline discretisation is bounded from both sides

Closest papers

Nearest neighbours by meaning across the whole index.

On the Exponential Circuit Imbalance of the Ben-Tal Nemirovski Approximation

cs.CC math.OC ·2026-08-12 ·Jonah Bondar, Stephen Vavasis
theory similarity 0.59
Abstract

Dadush et al.\ (2024) recently developed a scaling-invariant layered least squares algorithm for linear programming whose complexity depends on the optimal condition measure $\barχ_A^*$. Their work builds on Vavasis and Ye's (1996) algorithm whose running time depends only on the constraint matrix $A$ through the condition number $\barχ_A$. Monteiro-Tsuchiya (2003) defined the optimal condition number $\barχ_A^*$ as the maximum $\barχ_{AD}$ achievable over all positive diagonal column rescalings $D$. Dadush et al.\ (2024) introduced the optimal circuit imbalance measure $κ_W^*$, which serves as a lower bound for $\barχ^*_A$. Instances with artificially large optimal circuit imbalance measures $κ_W^*$ can be easily constructed; however, finding naturally occurring examples where this optimal scaling-invariant measure grows exponentially is of independent interest. In this paper, we show that the Ben-Tal Nemirovski (BN) linear programming approximation of the unit disk provides such an example. By explicitly constructing circuits in the kernel of the BN formulation, we prove that the optimal circuit imbalance measure $κ_W^*$ grows exponentially in the number of approximation steps. Since $κ_W^*$ lower bounds $\barχ_A^*$, our result demonstrates that the BN approximation yields an exponentially ill-conditioned family of constraint matrices.

0.04

Controlled Block Proximal Linearized Method for Nonconvex and Nonsmooth Optimization

cs.LG math.OC ·2026-10-25 ·Weifeng Yang
theory similarity 0.57
Abstract

We consider multiblock nonconvex and nonsmooth optimization without assuming bounded iterates or coercivity of the original objective. We propose the Controlled Block Proximal Linearized (CBPL) method, which incorporates a control potential into block updates while retaining the original objective, variables, and constraints. Under explicit control design conditions and the KL property, the control yields boundedness, finite length, whole-sequence convergence of the original variables, and convergence of objective values. At the limit, one limiting subgradient of the original objective vanishes on all effective comparison directions. We apply CBPL to graph-regularized nonnegative matrix factorization (NMF), CP decomposition, and independent-core matrix tri-factorization (NMTF) with $\ell_0$ norm sparsity constraints. We characterize their reconstruction-preserving scale degeneracies, proving that scale criticality is equivalent to graph collapse and that full criticality forces such collapse. Explicit controls provide compact sublevel sets and a uniform positive lower bound on graph action, including with an independent Tucker core, while retaining genuine reconstruction-changing directions. Each block admits an explicit projection and a computable safe step parameter. For ordinary NMF and CP models, whole-sequence convergence to a full critical point follows. Experiments with direct safe updates demonstrate convergence, sustained graph action, and controlled factor and core scales.

0.15

BMuon: Budgeting Muon's Spectral Geometry for Structured Compressibility

cs.LG ·2026-10-16 ·Rida Assalouh, Tom Jacobs, Rebekka Burkholz, Yani Ioannou, Umut Şimşekli
similarity 0.56
Abstract · 54 pages, 6 figures

Muon has emerged as a strong optimizer for neural network training and admits an interpretation as steepest descent under a spectral-norm geometry. Yet we find that Muon-trained Transformers can degrade sharply when their MLP hidden channels are pruned after training. To reconcile Muon's optimization performance with structured compressibility, we introduce Budget Muon (BMuon), which modifies Muon's spectral geometry so that hidden channels across blocks compete for a shared allocation budget. The resulting oracle is exactly normalized steepest descent under a family of norms that interpolate between Muon's geometry and structured sparse geometries. Our principal sparsity control is an interpretable global channel-allocation budget, rather than a setting-dependent sparsity-penalty coefficient. For linear factorization models, we show that Muon admits minimum-cost exact representations with redundant hidden width, whereas under the BMuon oracle geometry and a sufficiently scarce budget, every minimum-cost representation is width-minimal. For practical training, we derive a minorization-maximization surrogate for the exact allocation problem. On ImageNet-1K with ViT-S/16, BMuon improves over Muon under gradual structured MLP pruning, and its learned allocations provide a more effective pruning criterion than weight magnitude. On a 1B-parameter LLaMA-style model trained on FineWeb, BMuon improves over Muon under post-training MLP channel pruning.

0.28

Tight Asymptotic Bounds on the Maximum Number of Linear Regions of Piecewise-Linear Kolmogorov--Arnold networks

cs.LG ·2026-11-05 ·Ye Su, Yong Liu
theory similarity 0.56
Abstract

Schoots et al.~\citep{schoots2025relating} (AISTATS 2025) established a generally loose upper bound on the number of linear regions of continuous piecewise-linear (CPWL) Kolmogorov--Arnold networks (KANs), without providing a lower bound. We consider a $T$-layer network with width vector $\mathbf n=(n_0,\ldots,n_T)$, input dimension $n_0=d$, output width $n_T=1$, and at most $k\geq2$ affine segments per edge. Our matching upper and lower bounds determine the tight growth order $Θ_{\mathbf n}(k^{\sum_{\ell=1}^{T}ρ_{\ell-1}})$ in $k$, where $ρ_{\ell-1}:=\min_{0\leq s\leq\ell-1}n_s$. When every hidden width is at least $d$, the tight order simplifies to $Θ_{\mathbf n}(k^{dT})$. The upper bound of \citep{schoots2025relating} is $k^{\sum_{\ell=1}^{T}n_{\ell-1}n_\ell}$. Our upper bound improves it by a factor of $Θ_{\mathbf n}\!\left(k^{\sum_{\ell=1}^{T}n_{\ell-1}n_\ell-\sum_{\ell=1}^{T}ρ_{\ell-1}} \right)$, with strict asymptotic improvement whenever a hidden layer has width greater than one. We further compare CPWL KANs with rectified linear unit (ReLU) networks using dense parameterizations with at most $Λ$ parameters per model and the same fixed nonlinear depth $T$. Counting trainable knot locations and optimizing the widths and $k$, we prove that the maximum region count of CPWL KANs divided by that of ReLU networks is $Θ_{d,T}(Λ^{d\lfloor T/2\rfloor})$ as $Λ\to\infty$, yielding a polynomial advantage for KANs when $T\geq2$. These tight bounds characterize the attainable geometric complexity of CPWL KANs, providing a foundation for understanding their expressive power and guiding architectural design.

0.32

What Must Survive? Exact Task-Information--State Frontiers Under Partial Task Revelation

cs.LG cs.IT ·2026-09-18 ·Ronald Katende
similarity 0.55
Abstract · 12 pages, 2 figures, 1 table. Substantially expanded version: added sharp approximate bounds, overlap-deficit theory, approximation guarante…

A state may need to be compressed before its exact downstream task is known. For a finite family of linear task operators $\{T_u\}_{u\in\U}$, we characterize exactly how much state must survive when only one of $K$ coarse task messages is available at compression time: $$p^*(K)= \min_{\substack{\Pcal\text{ partition of }\U\\|\Pcal|\le K}} \max_{C\in\Pcal}\rank(T_C),$$ where $T_C$ stacks the tasks that remain unresolved within cell $C$. Thus task information reduces retained state precisely by separating tasks whose joint task-visible subspace is expensive. We derive the dual bit frontier, an irreducible common-core floor, and a sharp singular-value characterization for nonzero tolerance. For overlapping task families, a cumulative overlap deficit places the exact frontier within an additive $Δ$ of weighted load balancing. Optimal partitioning is strongly NP-hard, yet direct-sum families admit classical approximation guarantees and a PTAS. We stress-test the theory on learned task operators from CIFAR-10, Fashion-MNIST, SVHN, and Burgers observables. For Burgers with $K=2$ and $\eps=0.35$, a constructive partition attains $p=4$, while balanced random partitions require median $p=6$, with held-out worst standardized RMSE $0.0368$. The results distinguish uniform task sufficiency from distributional and decision-level performance.

0.37

Dec-BFTRL: Squre-Root Regret for Decentralized Online Upper-Linearizable Optimization under Separation Access with Application to Continuous Submodular Maximization

cs.AI math.OCcs.LG ·2026-08-31 ·Yiyang Lu, Mohammad Pedramfar, Vaneet Aggarwal
similarity 0.55
Abstract

We study decentralized online optimization of upper-linearizable payoffs over an action set under efficient separation access, with applications to online continuous diminishing-return (DR) submodular maximization. We propose Decentralized Barrier Follow-the-Regularized-Leader (Dec-BFTRL), and evaluate each agent's played action against the average of all local objectives. Each agent maps an internal iterate to a feasible action through an approximate gauge projection, communicates only a cumulative surrogate-gradient dual state, and invokes the local HybridNewton procedure to approximately minimize its post-communication BFTRL potential. For every agent, we achieve expected network-aggregate regret of $\widetilde O(\sqrt{T})$. Over $T$ rounds, each agent uses $T$ neighbor-mixing steps and $\widetilde O(T)$ separation-oracle calls. We give wrapper instantiations covering four up-concave or DR-submodular maximization problems.

0.21

Sharp Covering-Number Bounds for Feedforward ReLU Networks via Parameter Symmetries

cs.LG math.STstat.ML ·2026-11-11 ·Ruizhi Deng, Guohao Shen, Tangrui Xiong
theory similarity 0.55
Abstract

In neural networks, distinct parameter configurations often realize the same input--output mapping, indicating that nominal parameter counts can overstate functional complexity. In this paper, we characterize the parameter-space symmetries of ReLU and Leaky ReLU networks across fully connected, convolutional, and residual architectures. We prove that every quasianalytic parameter map preserving the network function is a composition of positive channel scaling and channel permutation. By quotienting out both symmetries, we derive uniform covering-number upper bounds in which positive scaling reduces the complexity exponent from the total parameter count $S$ to $S-H$ (with $H$ denoting the number of scalable channels) and permutation symmetry divides the bound by the factorial count $G$ of channel permutations. We establish matching lower bounds showing that the exponent $S-H$ is sharp in general and that the divisor $G$ is attained for one-hidden-layer and generically identifiable networks. We further extend these results to general continuous piecewise-linear activated networks. Finally, we apply our covering-number bounds to derive statistical risk guarantees for empirical risk minimization and capacity control beyond i.i.d.\ settings, and show how the underlying symmetries help explain weight rebalancing in function approximation and flat directions in empirical risk optimization.

0.16

Scalable Incremental Robustness Analysis of Neural Network Feedback Systems

cs.LG math.OCeess.SY ·2026-09-18 ·Zichen Wang, Peter Seiler, Geir Dullerud, Bin Hu
similarity 0.55
Abstract

Semidefinite programming (SDP) certificates for feedback systems containing deep neural networks (NNs) typically scale with the total number of neurons, whereas small-gain tests are scalable but can be highly conservative. This paper develops a unified and scalable framework for incremental robust stability and performance analysis of feedback interconnections involving high-dimensional NNs and unmodeled dynamics. By combining a structured decomposition of the full-order SDP condition with scalable Lipschitz constant estimation algorithms, we derive reduced verification conditions that certify incremental convergence and incremental $\ell_2$-gain bounds. The dimensions of the resulting control-analysis linear matrix inequalities (LMIs) depend only on the widths of the last two network layers and are \textit{independent of network depth}. The framework preserves the coupling between the plant and the NN, with the incremental small-gain condition recovered as a special case. To further reduce conservatism, we develop a multi-round alternating update scheme that iteratively refines the coupling variables while preserving scalability. Numerical experiments show that the proposed framework achieves state-of-the-art incremental $\ell_2$-gain bounds for large-scale NN feedback systems.

0.35