CS Paper Radar

A Weighted Kernel Method for Approximation that Adapts to Learned Multivariable Structure

John E. Darges, Laura Weidensager

cs.LG math.NA ·submitted 2026-09-15 04:02 UTC · v2

Abstract

Approximating the input-output behavior of a multivariable black-box function from limited data is challenging when blind to the importance of its inputs and their interactions. We introduce total sensitivity kernels (TSKs), a method based on families of weighted ANOVA kernels that learn and adapt to this multivariable structure. TSKs parameterize the weights on each multivariable component of the target function by factors for each input. We propose learning these factors directly from function evaluations by selecting the reproducing kernel Hilbert space (RKHS) in which the target function has minimum norm. Under suitable conditions, we show that this norm-minimization problem admits a unique solution, and we establish consistency of a finite-data formulation based on minimum-norm interpolation. The learned TSK factors characterize the participation of individual inputs across interactions and main effects, providing a kernel-dependent notion of input sensitivity related to total Sobol indices. Numerical experiments demonstrate that adapting the kernel to learned multivariable structure can substantially improve approximation accuracy over a standard product kernel.

Closest papers

Nearest neighbours by meaning across the whole index.

Kernel Methods for Learning Operators with Multiple Inputs and Outputs

cs.LG math.STstat.ML ·2026-08-12 ·Adrien Weihs, Chunyang Liao, Jingmin Sun, Hayden Schaeffer
similarity 0.69
Abstract

Learning mappings between infinite-dimensional objects is a central challenge in scientific machine learning. We introduce a general kernel-based encoder-decoder framework for operator learning that separates observation, representation, learning, and reconstruction. We develop this framework for multi-input, multi-output operator learning, where operators map between products of potentially distinct function spaces. Our approximation theory shows that, although the number of inputs and outputs can increase, the convergence rate is governed by the most challenging constituent approximation problem rather than the overall problem dimension. The framework leads to practical kernel methods with closed-form training and inference, combining mathematical tractability with computational efficiency. We further specialize the approach to multiple operator learning by introducing KernelMO, a family of kernel methods with complementary operator-valued and product-space formulations. Across five families of parametric partial differential equations, the proposed methods achieve competitive or state-of-the-art predictive accuracy while reducing training and inference costs relative to neural operator architectures and deep learning based models, offering an efficient and lightweight alternative.

0.04

Bayesian Optimization on Function Spaces via Sparse RKHS Manifolds

cs.LG stat.ML ·2026-10-05 ·Davide Sartor, Meghan E. Huber, Donghyun Kim, Nathan Wycoff
benchmark similarity 0.64
Abstract

Bayesian Optimization (BO) has become an established methodology for minimizing black-box functions of a vector input. Often, however, this parameter vector arises from the discretization of an inherently functional relationship. Several recent articles have considered the Functional Bayesian Optimization (FBO) setting, in which the variable to be optimized is not a member of a finite dimensional vector space, but rather an infinite dimensional function space. In this work, we propose $L^0$ Manifold Optimization (L0MO), a simple approach to FBO which searches the subset of a Reproducing Kernel Hilbert Space (RKHS) consisting of functions with a sparse representation in the kernel functions, optimizing both the kernel locations and their coefficients. We discuss in detail the relationship between our method and existing ones, providing a unifying lens through which to view prior works. To assess our method against the state of the art, we conduct an extensive computational study, and along the way develop a novel set of benchmark test functions which port standard finite-dimensional ones to the infinite dimensional domain. Our experiments demonstrate that, on balance, the proposed method achieves superior performance across a wide range of test benchmarks.

0.19

Trace-Norm Geometry of Composed Multitask Predictors

cs.LG ·2026-08-25 ·Mahdi Mohammadigohari
similarity 0.62
Abstract · Substantially revised and expanded. Added uniform guarantees for estimated feature Gram matrices, regularized fitting and generalization res…

Multitask predictors apply task-specific functions to a shared representation. Penalizing the representation and task functions separately can assign different costs to equivalent predictors and charge for directions the tasks do not use. We study the trace norm of the complete task predictors in a fixed input reproducing kernel Hilbert space, weighted by task sample sizes. For a finite-dimensional intermediate function space, this norm is the infimum of the product of the representation operator norm and task trace norm over invertible coordinate changes. Task inner products determine this norm; standard duality gives the exact empirical Rademacher complexity of its norm ball. We estimate the feature Gram matrix used in fitting. Under bounded leverage, one estimate controls the penalty for every task head and shared linear combination of fixed features, even when the Gram is singular and the coefficients are chosen after estimation. This yields comparisons of penalized empirical objectives and a generalization bound under an estimated norm constraint, with the stated loss and sampling assumptions. Controlled experiments show that signal alignment can favor either penalty. A SARCOS comparison finds close agreement between exact and estimated penalties on overlapping training and evaluation observations.

0.22

KernelOnet: An Interpretable Neural Operator Based on Kernel Functions

cs.LG ·2026-09-28 ·Yuan Guo, Hanshu Chen, Qiang Xi, Timon Rabczuk, Zhuojia Fu
similarity 0.60
Abstract

This paper proposes an interpretable neural operator framework, the Kernel Operator Network (KernelOnet), which incorporates kernel functions explicitly into the neural operator architecture, so that the operator structure matches the kernel-expansion form used in boundary-type kernel-expansion methods. Unlike traditional neural operators such as DeepONet, which learn basis functions implicitly through deep networks, KernelOnet replaces the trunk network with explicit kernels and offers three complementary kernels: a data-driven learnable kernel, in which a neural network parameterizes a radial basis function learned from data, and which for constant-coefficient linear problems can be regarded as a non-singular fundamental solution; a physics-informed kernel, which embeds physical information such as analytic fundamental solutions into the network structure, so that the expansion satisfies the governing equation automatically and can be trained without supervision on boundary conditions alone, with no interior solution data; and a hybrid kernel, which splits the solution, according to the linear principal part of the governing equation, into a homogeneous part spanned by analytic fundamental solutions and a source part carried by low-rank learned correction kernels, thereby balancing physical priors against data fitting on nonlinear problems lacking an analytic fundamental solution. On three benchmarks and one engineering problem in a shallow-water waveguide, KernelOnet attains high accuracy; where comparable with DeepONet, it is more accurate with fewer learnable parameters. Its unsupervised configuration needs no interior solution labels, and its per-query inference cost is far below that of per-instance solvers, offering an effective route to acoustic propagation in unbounded exterior domains that general-purpose neural operators struggle to handle.

0.12

Kernelized Parameter Sensitivity Atlases for Amortized Conditional Interpretability

cs.LG cs.CVstat.ML ·2026-10-08 ·Davide Wiest, Ahanaf Hasan Ariq, Abdul Monaf Chowdhury
code/data similarity 0.60
Abstract · 22 pages, 10 figures. Code: https://github.com/Axym-Labs/kpsa

Gradient-based attribution provides input-specific information about model parameters, but computing it via backpropagation for many samples is expensive, while static importance scores discard input dependence. We derive KPSA (Kernelized Parameter Sensitivity Atlas) from a block decomposition of the empirical neural tangent kernel. It measures normalized parameter-group sensitivity, embeds those sources in an explicit query space, and uses a lightweight kernel to estimate sensitivities for unseen inputs. We derive bounds that separate finite-source from irreducible representation error and give margin-based top-$k$ recovery. This paper studies two interpretability applications. Firstly, KPSA identifies parameter groups with high susceptibility to perturbations across vision, forecasting, and protein-modeling setups. Across a fivefold range of parameter budgets ($0.02$-$0.1\%$), its selected groups attain $39$-$85\%$ of the perturbation susceptibility of a first-order backpropagation oracle. In vision and forecasting, these parameter groups form sparse sensitivity circuits that extend across depth and vary systematically with the input. Secondly, we use KPSA on an activation-attribution problem for MLP features in transformers. It retains $81\%$ of the gap between a backpropagation oracle and a constant-sensitivity baseline on vision while increasing batched query-time throughput from 551 to 2,041 images/s and reducing peak GPU memory from 4.55 to 0.44GB. On the protein-modeling task, it recovers $68\%$ of this gap. An ablation of KPSA's kernel aggregation shows that nearest-neighbor retrieval in KPSA space can remain competitive. The empirics show that compact kernel atlases can store useful input-dependent parameter sensitivity while amortizing repeated gradient attribution, a step toward scalable conditional interpretability. Code can be found at https://github.com/Axym-Labs/kpsa .

0.35

Scalable Learning of Determinantal Point Processes with Kernel-Structured Interactions

cs.LG stat.ML ·2026-10-09 ·Takahiro Kawashima, Hideitsu Hino
hot direction similarity 0.60
Abstract

Determinantal point processes (DPPs) on a finite set are probability models that exhibit repulsion where inter-item similarities are encoded by a positive definite kernel matrix. Scaling DPP learning to large datasets typically requires imposing structural constraints on the kernel matrix, such as a low-rank-plus-diagonal form. However, structural assumptions introduce a severe trade-off between expressive power and computational scalability. Our study addresses this challenge by (a) modeling the full-rank interactions via a nonlinear kernel function and latent features, (b) designing a surrogate function for the maximum likelihood estimator, and (c) adopting stochastic optimization using random features of the nonlinear kernel. Numerical experiments on real-world datasets demonstrate that our method significantly improves this trade-off, achieving better generalization performance with remarkable parameter efficiency.

0.35

Learning Conditional Expectation Operators via Functional Newton Updates

cs.LG stat.ML ·2026-09-28 ·Thiago Ramos, Alek Fröhlich, Daniel Perazzo, Massimiliano Pontil
similarity 0.59
Abstract

We introduce the Functional Spectral-Newton Method (FSNM) for learning the leading singular structure of a conditional expectation operator without fixing a basis or reproducing kernel Hilbert space. FSNM fits a low-rank representation of the centered joint-to-product density ratio kernel by alternating functional Newton updates. Each update reduces to a preconditioned regression, which we approximate with vector-valued regression trees in a stagewise boosting procedure. At the population level, we establish descent and an $O(1/T)$ best-iterate block-stationarity rate under a relative weak-learner accuracy condition, and show that every nondegenerate local minimum over the full centered $L^2$ spaces is a globally optimal rank-$d$ approximation. Synthetic experiments show that FSNM recovers a low-rank density ratio and its leading spectral structure, and that the same learned kernel can answer multiple conditional queries without refitting.

0.23

Revisited Support Vector Machines: A Unified Joint Optimization Framework with Nyström Approximation for Scalable Adaptive Learning

cs.LG ·2026-10-11 ·Nassou Gongon Benjamin, Noulapeu Ngaffo, Patrick Njionou Sadjang
similarity 0.58
Abstract

The theoretical elegance of Support Vector Machines (SVMs), grounded in statistical learning theory, establishes them as a reference for robust classification. However, in the era of Big Data, classical SVMs face two major bottlenecks: the prohibitive cubic complexity $\mathcal{O}(N^3)$ of training and the rigidity of static kernels. Traditional methodologies treat kernel selection as a discrete, "brute-force" GridSearch problem, effectively decoupling the learning of data representation from the optimization of the decision boundary. This paper proposes a paradigm shift with a so-called "Revisited SVM", a unified framework that recasts the learning process as a continuous %bi-level joint optimization problem. We derive a novel objective function $J(α, θ)$ that allows for the simultaneous optimization of dual Lagrange multipliers and kernel hyper-parameters. To solve this challenging non-convex problem efficiently, we introduce a hybrid Block Coordinate Ascent (BCA) algorithm, tightly integrated with Nyström-based low-rank approximation, which drastically reduces computational complexity to $\mathcal{O}(Nm)$. Extensive experiments conducted across four heterogeneous domains (Bioinformatics, Finance, Cybersecurity, and Computer Vision) demonstrate that our approach not only matches but often exceeds exhaustive GridSearch accuracy, while delivering training speedups of up to 100x. Notably, on the complex NSL-KDD intrusion detection dataset, our adaptive kernel approach achieves 96.25\% accuracy compared to a mere 50\% for standard baselines, proving its unique ability to model complex non-linear manifolds in high-dimensional spaces. We validate our results using rigorous statistical tests and provide a theoretical convergence analysis of the proposed algorithm.

0.18