CS Paper Radar

General Frequency Principle for Linear Approximation Classes

Yujun Wang, Yiqi Gu

cs.LG math.NA ·submitted 2026-10-30 02:59 UTC · v1
theory

Abstract

In this paper, we study a general frequency principle under gradient descent training for the finite linear approximation classes. Under suitable assumptions, we prove that the learner amplitude relative to the target amplitude vanishes at high frequencies, uniformly throughout training. We also establish sufficient conditions for low-frequency priority, covering single-frequency targets trained separately and two frequency components trained in the same run. This general frequency principle does not imply that lower frequencies always converge faster. Instead, for two frequencies that satisfy certain comparison conditions, one component will take longer to learn than the other; and in common cases, the longer component has higher frequency. Numerical experiments with Legendre and trigonometric bases validate the theoretical results.

Closest papers

Nearest neighbours by meaning across the whole index.

Learning Fourier-Sparse Signals and Multiband Signals

cs.DS ·2026-09-02 ·Dongrun Cai, Xue Chen, Xiaowei Shao, Yile Wang
theory similarity 0.59
Abstract

A classical problem in continuous sparse Fourier transforms, which dates back to Prony's work in 1795, is to learn a $k$-Fourier-sparse signal $x(t):=\sum_{j=1}^k α_j e^{2 π\mathbf{i} f_j t}$ with arbitrary $k$ frequencies $f_1,\ldots,f_k \in \mathbb{R}$. We consider this learning problem within a fixed time window $t \in [-1,1]$ in the presence of bounded $L_2$ adversarial noise. Because the frequencies $f_1,\ldots,f_k$ can be arbitrarily close, accurately recovering individual frequencies is information-theoretically impossible even under exponentially small noise from Moitra (STOC 2015). This work therefore examines two fundamental questions: If an algorithm uses $\tilde{O}(k)$ frequencies to approximate $f_1,\ldots,f_k$ of $x$, what is the optimal covering error? And how can we efficiently learn the continuous signal $x(t)$ as a whole over the time window? Our main result is an algorithm running in time $\tilde{O}(k)$ that outputs a list $L$ of $\tilde{O}(k)$ frequencies with covering error $O(k)$ --- meaning that for most frequencies $f_j$ of $x$, $L$ guarantees $\min_{\tilde{f} \in L} |f_j-\tilde{f}| = O(k)$ --- and an reconstruction $\tilde{x}(t) \approx x(t)$ over $[-1,1]$. When constrained to outputting $\tilde{O}(k)$ frequencies, this $O(k)$ covering error is nearly optimal: Chen and Price (ICALP 2019) demonstrated two indistinguishable $k$-Fourier-sparse signals separated by distance $\tildeΩ(k^2)$ in the frequency domain, which implies an amortized lower bound of $\tildeΩ(k)$ per frequency. Furthermore, our sample and time complexities are nearly optimal, dramatically improving upon the previous best algorithm by Song, Sun, Weinstein, and Zhang (FOCS 2023), which required $\tilde{O}(k^4)$ samples and $\tilde{O}(k^{4ω})$ running time.

0.49

Learnable Spectral Activations

cs.LG cs.CV ·2026-10-05 ·Tamir Shor, Or Litany, Alex Bronstein
similarity 0.55
Abstract

Implicit neural representations (INRs) are shaped by the spectral structure induced by their input encodings and activation functions. Existing methods improve fitting primarily by modifying which frequencies are available to the network, through coordinate encodings or periodic nonlinearities. However, frequency access is not the only bottleneck: signals with localized or spatially varying structure require the network to efficiently compose frequencies into multi-harmonic internal responses. We introduce learnable spectral activations (LSA), which replace fixed neuron-level nonlinearities with a residual truncated Fourier series whose harmonic amplitudes are learned during training. LSA does not expand the asymptotic function class. Instead, it changes the factorization of the representation: linear weights select features while activation coefficients control spectral shaping, and the two are updated by separate gradients. Because the activation output is affine in the coefficients given fixed pre-activations, spectral tuning becomes a more direct subproblem compared to architectures where it is entangled with feature selection. Empirically, this factorization concentrates more target-signal energy in the leading eigenmodes of the neural tangent kernel, consistent with improved optimization behavior. Across audio, image, neural radiance field, and neural acoustic field tasks, LSA also improves reconstruction quality.

0.25

Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theory

cs.LG stat.MLmath.PR ·2026-10-06 ·Deborah Oliveira, Elliot Paquette
theory similarity 0.53
Abstract

Weak-to-strong generalization is the phenomenon where a strong student model trained with labels produced by a weak teacher model is able to generalize better than the teacher. In this paper, we study this phenomenon in two-layer random feature networks where the model strength is determined by its width. Using tools from random matrix theory, we derive deterministic equivalents for the population errors of an optimally trained teacher and a student trained with gradient flow. For ReLU activation and a pure spherical harmonic target, we obtain sharp asymptotics under a Gaussian universality assumption, showing a quadratic improvement: the student error scales as the square of the teacher error. These results attain the general lower bound of Medvedev at al (2025). We also analyze how the student behaves under more general stopping times and targets supported on multiple harmonic degrees, characterizing the regimes in which weak-to-strong generalization occurs and identifying the transition between quadratic, non-quadratic, and no improvement.

0.17

High-Order Sinusoidal Modeling for Precise Parameter Refinement

eess.SP eess.AS ·2026-10-15 ·Shaowen Chen, Tomoki Toda
unusual similarity 0.53
Abstract · This manuscript is currently under review for publication in the IEEE Transactions on Signal Processing. This work has been submitted to the…

Although the amplitude estimation of sinusoids is a linear and tractable problem when their frequencies are known, the accuracy of signal modeling is fundamentally limited by the inevitable frequency mismatches produced by existing methods. Quasi-harmonic modeling (QHM) and its extensions (aQHM and eaQHM) offer the ability to refine frequency estimates while extracting amplitudes; however, their extraction and correction capabilities remain limited. In this paper, we first revisit QHM methods from the perspective of frequency mismatches, and reveal that their performances are essentially constrained by the order of the model. Specifically, QHM methods are linear models and thus incapable of capturing higher-order frequency variations, leading to suboptimal modeling accuracy. Second, to address this limitation, we propose a higher-order modeling framework that enables more precise approximation of signal components. Eventually, we qualitatively characterize the optimization problem and employ the Levenberg--Marquardt (LM) algorithm for parameter estimation. As a result, the proposed method allows for significantly improved estimation of amplitudes, frequencies, and phases of sinusoidal components, enabling more accurate signal modeling. Experiments on both simulation study and speech signals demonstrate the superiority of the high-order model in frequency refinement and parameter extraction and its robustness under additive noise conditions.

0.24

Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework

cs.LG cs.AImath.SPstat.ML ·2026-10-15 ·Yanyang Xiao
similarity 0.53
Abstract · 15 pages, 0 figures. The main draft dates back to 2019, with a new abstract and introduction section

Neural networks often learn low-frequency components before high-frequency ones --- a phenomenon called spectral bias or termed F-principle. Many numerical studies have demonstrated this phenomenon, and theoretical studies have sought to explain it. However, several goals remain insufficiently addressed: (1) explaining spectral bias asymptotically, including its origin and why the preferred components are generally sinusoidal; (2) extending the explanation to general activation functions, such as sigmoid and ReLU; (3) making weak assumptions about the data distribution; (4) avoiding aliasing effects, such as those arising in finite-domain Fourier analysis; and (5) developing a theoretical framework applicable to arbitrary networks. We address these goals theoretically by analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix H, which arises when we expand the network output as a general Fourier series (GFS) in an orthogonal polynomial basis and express the gradient-descent dynamics of the GFS coefficients as ordinary differential equations. The graded structure of H enables an intuitive explanation of spectral bias, while orthogonal polynomial theory helps us analyze how learning bias depends on the activation function and, potentially, the network architecture.

0.23

Shallow neural network approximation in mixed Sobolev spaces

cs.LG math.NA ·2026-09-04 ·Yuwen Li, Guozhi Zhang
theory similarity 0.51
Abstract · 40 pages, 2 figures

We investigate the best $L_2$ approximation of mixed Sobolev spaces by shallow neural networks with $n$ neurons and general activation functions. We first establish an activation-independent Fourier-block principle: if an activation has univariate approximation order $ρ$ in the sense of the Fourier-block property, then the global approximation rate has algebraic order $\min\{α,ρ\}$ for target functions of mixed smoothness $α$, up to explicit logarithmic factors. To verify this property for concrete activations, we introduce a structured univariate approximation condition that implies the Fourier-block property with explicit parameters. For $\mathrm{ReLU}^k$, a matching algebraic lower bound identifies $\min\{α,k+1\}$ as the optimal algebraic approximation exponent in any dimension, up to logarithmic factors in the upper bound. The framework also yields the exponent $\min\{α,k+1\}$ for cardinal B-splines and soft-$\mathrm{ReLU}^k$, and the full mixed-smoothness exponent $α$ for ELU and cosine activations, again up to logarithmic~factors.

0.19

Minimax rates for learning spectral Barron functions by deep ReLU neural networks

cs.LG stat.MLmath.NA ·2026-09-30 ·Songqiu Ma, Yunfei Yang
theory similarity 0.51
Abstract

We study how well deep neural networks approximate and learn spectral Barron functions. Recent studies have shown that these function classes can be efficiently approximated by shallow neural networks without suffering from the curse of dimensionality. We complement these results by providing new approximation bounds for deep networks with ReLU activation and establishing the minimax rates for learning these function classes. Specifically, we show that $d$-dimensional spectral Barron functions with smoothness index $s>0$ can be approximated by deep ReLU neural networks with approximation rate $\widetilde{\mathcal{O}} (S^{-\frac{1}{2}-\frac{s}{d}})$, where $S$ denotes the number of nonzero parameters in the network. Using this approximation result, we further show that deep ReLU neural networks can learn spectral Barron functions in a fast rate $n^{-\frac{d+2s}{2d+2s}}$ with $n$ training samples. Finally, we prove that this convergence rate is minimax optimal up to logarithmic factors.

0.17

Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons

cs.LG stat.ML ·2026-09-28 ·Mana Sakai, Masaaki Imaizumi
theory similarity 0.49
Abstract

Hyperparameter transfer across model width can substantially reduce the cost of tuning large neural networks, but its behavior when the training horizon grows with width is not fully understood. Building on the framework of fast hyperparameter transfer (Ghosh et al., 2026), which formalizes when transfer is effective, we investigate conditions that ensure fast transfer in the growing-horizon regime. Specifically, we study learning-rate transfer in a shallow linear network with a single trainable hidden matrix, trained by full-batch gradient descent. Under additional spectral assumptions, our main results are threefold. (i) We prove fast learning-rate transfer as $n,T\to\infty$ whenever $T=o(\sqrt{n})$. (ii) We characterize the transfer rates through the finite-width perturbation scale, the first-order sensitivities of the loss and its learning-rate derivative to finite-width perturbations, and the local loss curvature. (iii) We derive limiting distributions for the optimal learning rate and optimized loss, governed by fluctuations associated with the extreme eigenvalues of the data Gram matrix. These results clarify how spectral structure and local loss sensitivities govern learning-rate transfer at growing horizons.

0.28