CS Paper Radar

Learnable Spectral Activations

Tamir Shor, Or Litany, Alex Bronstein

cs.LG cs.CV ·submitted 2026-10-05 21:29 UTC · v2

Abstract

Implicit neural representations (INRs) are shaped by the spectral structure induced by their input encodings and activation functions. Existing methods improve fitting primarily by modifying which frequencies are available to the network, through coordinate encodings or periodic nonlinearities. However, frequency access is not the only bottleneck: signals with localized or spatially varying structure require the network to efficiently compose frequencies into multi-harmonic internal responses. We introduce learnable spectral activations (LSA), which replace fixed neuron-level nonlinearities with a residual truncated Fourier series whose harmonic amplitudes are learned during training. LSA does not expand the asymptotic function class. Instead, it changes the factorization of the representation: linear weights select features while activation coefficients control spectral shaping, and the two are updated by separate gradients. Because the activation output is affine in the coefficients given fixed pre-activations, spectral tuning becomes a more direct subproblem compared to architectures where it is entangled with feature selection. Empirically, this factorization concentrates more target-signal energy in the leading eigenmodes of the neural tangent kernel, consistent with improved optimization behavior. Across audio, image, neural radiance field, and neural acoustic field tasks, LSA also improves reconstruction quality.

Closest papers

Nearest neighbours by meaning across the whole index.

CAFE+FNO: Fourier Kernel Generation via Multiplicative Feature Composition

cs.LG math.NA ·2026-10-07 ·Hyungjoon Juen, Minwoo Shin
code/data similarity 0.62
Abstract

The Fourier Neural Operator (FNO) learns solution operators of partial differential equations (PDEs) through Fourier-space kernel parameterization, but frequency truncation can limit the learning of high-frequency variations. AM-FNO and SirenFNO generate kernels for all grid modes from spectral coordinates using shared networks, making coordinate encoding and generator design important. Recent work on implicit neural representations (INRs) has proposed constructing frequency interactions through explicit feature composition rather than relying on subsequent MLPs to form them implicitly. Building on this approach, we propose CAFE+FNO, which incorporates Content-Aware Frequency Encoding+ (CAFE+) into Fourier kernel generation. CAFE+ combines Fourier--Chebyshev features through parallel affine branches and a Hadamard product, forming interactions within and across the two feature families. A kernel MLP maps the resulting representation of each normalized spectral coordinate to a complex channel-mixing matrix. Each layer shares its generator across all stored modes, making the number of trainable parameters independent of the number of modes for a fixed architecture. We compare CAFE+FNO with existing FNO variants on five PDE benchmarks and conduct ablation studies on basis configuration, multiplicative composition, and bandwidth learnability. Code and experimental configurations are available at https://github.com/fabsk101/CAFEPlusFNO.git.

0.45

When Does Frequency Decomposition Benefit Physics-Informed Neural Networks? A Preliminary Ablation Study

cs.LG cs.AI ·2026-08-24 ·Shubham Rai
benchmarkhot direction similarity 0.62
Abstract

Partial differential equations (PDEs) often have high-frequency and multi-scale features that neural networks struggle to approximate. Physics-Informed Neural Networks (PINNs) build the governing equations directly into training, but suffer from spectral bias: they learn low-frequency components faster than high-frequency ones. Techniques such as Fourier feature embeddings and sinusoidal activations address this, but most studies assume they help across the board without checking which spectral regimes actually benefit. We introduce a dual-branch, spectrally-gated architecture (DBSG-PINN) that splits low- and high-frequency components into separate subnetworks joined by an adaptive gate, and use it to run a partially controlled ablation of frequency decomposition and spectral routing. We test this on five one-dimensional benchmark PDEs, ranging from smooth, single-scale problems to oscillatory, multi-scale ones. Frequency decomposition helps most on the spectrally complex benchmarks, cutting relative $L_2$ error by up to $59.2\%$ on a multimodal wave problem, but gives little benefit on smoother PDEs. On one benchmark (1D Wave), it performs substantially worse than a simpler fixed-combination variant. The gate's benefit scales with how spectrally rich the target solution is: the full model's advantage over the ablations is largest on multi-scale benchmarks and smallest (or negative) on single-scale ones, consistent with the gate exploiting frequency structure rather than acting as noise,though we do not directly visualize or quantify its spatial activations in this study. All results come from a single training seed across five 1D benchmarks, so we present this as an exploratory study meant to raise questions rather than answer them, and outline the additional seeds and benchmarks needed to test whether the pattern holds.

0.32

HC-AINet: An Auditory-Inspired Network with Harmonic Compensation for Low-Complexity Speech Enhancement

eess.AS ·2026-10-22 ·Nan Li, Xiaoxiong Wu, Huawei Liu, Ziping Zhao
similarity 0.61
Abstract · 12 pages, 5 figures, 9 tables

Speech enhancement on resource-constrained devices requires noise suppression with limited computation and minimal loss of spectral detail. Auditory-inspired subband processing reduces computation, but its smooth gains have limited ability to distinguish harmonic peaks from neighboring noise. We propose a harmonic-compensated auditory-inspired network (HC-AINet) for low-complexity speech enhancement. A triangular equivalent rectangular bandwidth (ERB) filterbank supplies compact features to a lightweight network comprising inverted-residual units, temporal Conformer layers, and parallel frequency-gated convolutions. The network predicts global and harmonic gains together with fusion weights. A harmonic-location map derived from the noisy spectrum confines the harmonic contribution to selected bins on the original frequency grid, while the global branch controls the broader spectral envelope. The compensated magnitude, noisy phase, and a magnitude-derived minimum-phase cue jointly condition a compact phase estimator for waveform reconstruction. On VoiceBank-DEMAND, the 64-band model achieves a PESQ of 3.20 and a STOI of 0.95 with approximately 0.27 million parameters.

0.20

Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework

cs.LG cs.AImath.SPstat.ML ·2026-10-15 ·Yanyang Xiao
similarity 0.61
Abstract · 15 pages, 0 figures. The main draft dates back to 2019, with a new abstract and introduction section

Neural networks often learn low-frequency components before high-frequency ones --- a phenomenon called spectral bias or termed F-principle. Many numerical studies have demonstrated this phenomenon, and theoretical studies have sought to explain it. However, several goals remain insufficiently addressed: (1) explaining spectral bias asymptotically, including its origin and why the preferred components are generally sinusoidal; (2) extending the explanation to general activation functions, such as sigmoid and ReLU; (3) making weak assumptions about the data distribution; (4) avoiding aliasing effects, such as those arising in finite-domain Fourier analysis; and (5) developing a theoretical framework applicable to arbitrary networks. We address these goals theoretically by analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix H, which arises when we expand the network output as a general Fourier series (GFS) in an orthogonal polynomial basis and express the gradient-descent dynamics of the GFS coefficients as ordinary differential equations. The graded structure of H enables an intuitive explanation of spectral bias, while orthogonal polynomial theory helps us analyze how learning bias depends on the activation function and, potentially, the network architecture.

0.23

Wavelength Diversity Expands the Expressive Power of Optical Computing

cs.NE physics.opticsphysics.app-ph ·2026-10-16 ·Shiqi Chen, Che-Yung Shen, Parnian Ghapandar Kashani, Yongkang Cheng, Aydogan Ozcan
similarity 0.61
Abstract · 34 Pages, 10 Figures

Universal function approximation underpins neural networks' capacity to represent nonlinear relationships, but realizing this capability in optical hardware requires engineering the computational features encoded in light. Here, we show that wavelength diversity enhances optical function-approximation expressivity through wavelength-scaled Fourier features generated by a shared phase encoder, without increasing the number of input-encoded harmonics. Multiple wavelengths contribute to the representation of each output function rather than merely serving as independent computational channels. We establish theoretical scaling laws and approximation-error bounds, revealing that, for nondispersive encoders, the spectral endpoints determine the accessible computational bandwidth, while intermediate illumination wavelengths enrich the available harmonics within that bandwidth. Numerical studies demonstrate improved approximation of real- and complex-valued nonlinear functions, with a representative design achieving more than a 100-fold reduction in mean-squared error relative to tested monochromatic optical processors with the same number of trainable phase features. Experimentally, we validate enhanced approximation accuracy and demonstrate the simultaneous approximation of 4,136 distinct nonlinear functions using a compact, single-layer diffractive processor under five-wavelength illumination. These results establish wavelength diversity as an additional physical resource for scaling the expressive power of optical computing and enhancing the accuracy of parallel nonlinear function approximation.

0.24

Spectral Graph Neural Networks with Hermite Polynomials: A Comprehensive Study

cs.LG ·2026-09-24 ·Shuang Wu
similarity 0.60
Abstract

We study spectral graph neural networks built from Hermite polynomials and propose HermNet, a simple model that combines a nodewise predictor with normalized Hermite propagation. Its sparse recurrence requires neither eigendecomposition nor a learned basis. We distinguish the basic model from optional coordinate calibration, response normalization and Gaussian derivative regularization. Hermite and other complete polynomial bases span the same degree-bounded filter space, but their coordinates can produce different optimization behavior under limited training budgets. We analyze this behavior through spectral signal energy, label sampling, changes in learned features and the bias--variance trade-off of regularization. Controlled synthetic experiments identify a regime in which plain HermNet outperforms matched polynomial-basis alternatives, including with a jointly trained nonlinear predictor. Curvature regularization further improves HermNet when the same functional penalty is available to every comparator. Fixed-predictor controls support the advantage under short training budgets, but longer training removes the plain-model lead. Matched real-data comparisons show accuracy deficits, and architectural and numerical studies identify further limits. Together, the analysis and experiments clarify when Hermite propagation is useful and how calibration and regularization affect its performance.

0.28

Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles

cs.SD cs.AI ·2026-10-02 ·Junyoung Koh, Hao-Wen Dong
similarity 0.60
Abstract

Multi-pitch estimation in vocal ensembles is challenging because singers occupy overlapping pitch ranges and often sing at closely spaced fundamental frequencies, causing their harmonics to overlap in time-frequency representations. Existing models commonly use harmonic constant-Q transform (HCQT)-based representations to provide frequency-adaptive resolution, at the cost of expensive feature extraction when training mixtures are generated on the fly. We revisit this design and compare HCQT with a linear short-time Fourier transform (STFT), whose frequency bins are directly provided as model inputs. Despite its fixed frequency resolution and the absence of a pitch-aligned input grid, the linear STFT outperforms HCQT while substantially reducing feature-extraction cost. Further analysis shows that a longer analysis window or broader spectral coverage provides no additional improvement, while restricting the input to the predicted pitch range reduces the advantage of the linear STFT. These results suggest that finer frequency resolution does not necessarily improve vocal-ensemble MPE, and that shorter analysis windows can be more effective for time-varying vocal pitches.

0.17

Self-Supervised Source-Filter Model of Speech Production

eess.AS ·2026-10-13 ·Nani Meka, Raghavan Ramesh, Sri Rama Murty Kodukula
similarity 0.60
Abstract

Deep learning models of speech seldom represent the source-filter process by which speech is produced, relying instead on latent features with no physical correspondence to articulation or phonation. We propose SoFilNet, a self-supervised, differentiable Source-Filter Network that estimates the parameters of speech production directly from the raw waveform: a source, specified by fundamental frequency (F0), voicing and the aperiodicity of five frequency bands, and a filter, specified by ordered formant poles, additional poles that shape the rest of the spectral envelope, and antiresonance zeros. The model is trained only to reconstruct the input speech; no formant or pitch annotation is used. Since per-bin spectral losses favor blurred spectra wherever the model is uncertain, we train F0 and voicing against autocorrelation pseudo-labels refined with the model's own filter, aperiodicity against the periodic fraction of each band, and formant sharpness against spectral contrast. On VTR-FORMANTS, the estimated formants attain the lowest mean error among the reported methods, including supervised trackers, and on the unseen PTDB-TUG corpus the F0 frame error matches the best classical tracker and is lower than that of all learned trackers. The parameters are also consistent with speech production theory: aperiodicity rises with frequency and orders phone classes by the turbulence of their excitation, and the zeros deepen on nasal consonants, the lower one rising with place of articulation.

0.15