CS Paper Radar

Learning nonlinear interactions and cross-feature couplings with product-unit residual networks

Ziyuan Li, Uwe Jaekel, Babette Dellen

cs.LG cs.NE ·submitted 2026-11-13 12:00 UTC · v1
benchmarkhot direction

Abstract

Understanding nonlinear feature interactions is crucial in science and engineering, yet standard multilayer perceptrons (MLPs) often capture such interactions only implicitly, leading to entangled representations that can impair robustness and interpretability. We investigate complex-valued product-unit residual networks (CV-PURe) that integrate multiplicative product units with residual connections to explicitly model cross-feature couplings while stabilizing optimization. We conduct a systematic evaluation on an interaction-driven synthetic benchmark and real-world datasets and evaluate prediction together with input-level interaction recovery under a reproducible protocol. Across five data splits and five training seeds, CV-PURe achieves the lowest mean RMSE on four of six scientific benchmarks, reducing Friedman error by 53.9% relative to MLP. Its Friedman interaction maps assign 60.7% of total strength to the known interacting pair, compared with 31.0% for MLP. On a synthetic task containing third- and fourth-order products and 20 distractors, CV-PURe reduces RMSE from 0.1376 to 0.0797 relative to a complex-valued MLP and recovers all nine true pairwise projections in every training/sampling repetition. Bike Sharing experiments connect improved prediction to the contribution of temporal, calendar, and seasonal context through paired ablations of both PURe variants. Parkinsons Telemonitoring extends the context analysis to repeated measurements. Noise and learning-curve evaluations demonstrate sustained predictive gains under perturbed inputs and limited supervision, while inference measurements quantify computational cost. This article extends our ICCS 2026 paper, Modeling Nonlinear Feature Interactions with Product-Unit Residual Networks (doi:10.1007/978-3-032-29921-5_6).

Closest papers

Nearest neighbours by meaning across the whole index.

Beyond feature attribution: Recovering latent interaction structures in machine learning models via localized CBFI

cs.LG ·2026-10-26 ·Sejong Oh
similarity 0.61
Abstract

Background: Understanding individual predictions in complex tabular models remains challenging, as existing additive frameworks like SHAP and LIME often obscure structural logic by absorbing nonlinear interactions into individual feature scores. Exposing these latent interaction structures is essential for automated auditing in high-stakes environments where understanding the logic of model behavior is as essential as predictive accuracy. Methods: To address this limitation, we introduce the localized case-based feature importance (CBFI) framework. Unlike traditional numerical attribution, our method decomposes instance-specific predictions into structurally distinct components, effectively separating independent main effects (G1) from pure interactions (G4) to capture explicit feature synergies and provide a principled approach for diagnosing feature combinations. Results: Comprehensive experiments on wine quality classification and medical insurance regression demonstrate that localized CBFI successfully disentangles complex interactions hidden by additive models. Furthermore, fidelity tests and synthetic ground-truth benchmarks confirm that the identified interactions correspond precisely to the nonlinear structural dependency patterns embedded in complex architectures, including gradient boosting and deep neural networks, thereby successfully shifting the explainable AI paradigm from simple numerical attribution to the recovery of interaction-driven contributions.

0.10

Gradient Surgery for Physics-Informed Neural Networks

cs.LG ·2026-09-25 ·Thomas Borsani, Giuseppe Di Fatta
publishedbenchmark similarity 0.60
Abstract · Accepted at ACML 2026

Physics-Informed Neural Networks (PINNs) are trained by optimising a composite objective that combines data fitting with physics-based constraints, typically resulting in a highly imbalanced multi-task optimisation problem. Under these conditions, existing optimisation strategies are affected by conflicting task gradients, leading to slow convergence and unstable training, particularly for stiff and high-frequency partial differential equations. We analyse gradient conflicts throughout training of PINNs with standard optimiser and investigate Multi-Task Deep Learning (MTDL) optimisation methods. In our analysis across four benchmark problems we observed that PINN optimisation exhibits three distinct phases in which angle- and magnitude-based gradient conflicts alternate, with only one present at a time. Building on these observations, we propose PAM-GS, a physics-aware gradient surgery method that adaptively mitigates task interference during training according to the observed conflict types. Experiments on four representative PDE benchmarks demonstrate that PAM-GS combines competitive solution accuracy with consistently strong task-balanced performance, outperforming existing methods on most problems.

0.28

How Does Parameter Pruning Reshape DNN Representations? An Interaction-Driven Exploration

cs.LG ·2026-09-06 ·Fangbo Li, Junpeng Zhang, Qihan Ren, Quanshi Zhang
similarity 0.59
Abstract

This study focuses on the scientific problem of understanding internal factors that govern the diverse performance degradation of deep neural networks (DNNs) when different parameters are pruned. In order to explain why pruning certain parameters leads to significant performance degradation but pruning other parameters does not, we examine how the pruning operation affects the interaction patterns encoded by the DNN. We find that when we progressively increase the pruning ratio, the interaction patterns encoded by DNNs exhibit a distinct three-phase dynamics, \emph{i.e.}, model performance is not largely affected until the pruning operation begins to remove low-order interactions, and low-order interactions exhibit strong generalizability. Moreover, we find that the high sensitivity of DNN performance to the pruning of certain modules is attributed to whether the pruning operation removes generalizable low-order interaction patterns.

0.43

Shared Physics Responses Recover Hidden Rankings in Neural Operator Libraries

cs.LG math.NAphysics.comp-ph ·2026-08-20 ·Hanbing Liang, Fujun Liu
similarity 0.59
Abstract

Selecting the optimal neural-operator prediction during deployment is challenging when high-fidelity reference solutions are unavailable. We demonstrate that under a squared Hilbert-space loss, ranking a finite model library depends strictly on the low-dimensional span of candidate differences, allowing us to score all models simultaneously using a single anchor-based linearized response of the governing equation. This shared physical diagnostic accurately recovered over 99.6\% of pairwise preferences and 99.0\% of optimal checkpoints across diverse Fourier and convolutional operator libraries for fluid, reaction-diffusion, and wave dynamics. Furthermore, the corrected physical proxy frequently outperformed the best individual candidates, and we establish computable sufficient conditions that rigorously certify exact decisions for strongly monotone discretizations. By exploiting the local dynamical response rather than raw defect magnitude, this framework enables the reliable and highly efficient deployment of scientific surrogates without requiring ground-truth data.

0.26

Transferability of Learned States in Neural PDE Solvers

cs.LG ·2026-10-07 ·Shunye Wang, Haochen Wen, Shuo Li Liu, Xuanyi Wang, Lihao Liu +1 more
similarity 0.58
Abstract

Assessing useful reuse in neural PDE solvers is challenging: final accuracy can reflect source learning and target-time computation. Our reuse contract separates solution accuracy, learning contribution, and numerical utility through paired state comparisons, matched target information and budgets, and cost accounting. A literature audit extracts 18 version-specific protocol records from 12 papers, documenting retained states, target-time resources, and reported controls. For a fixed linear system and residual tolerance, we construct two initial guesses with identical solution-error, energy-error, and residual norms, reaching the same solution with different conjugate-gradient (CG) iteration counts. Across 240 source-training trajectories, two linear PDE families, Fourier neural operators and convolutional networks, a fixed predictor's benefit reverses across correction algorithms. Among pairs with both relative prediction errors less than or equal to 5 percent on 64 in-distribution tasks (63 by 63 interior grids), reductions in all three norms accompany more CG iterations, at mean taskwise rates of 23.5 percent and 23.9 percent in two libraries. Work-based selection saves 2.50-3.33 CG iterations on held-out in-distribution tasks; matched adaptation demonstrates finite-budget pretraining value. Independent batches confirm a 0.73 percent complete online saving for one physics-trained Fourier neural operator against zero-initialized Poisson-preconditioned CG. Reuse requires matched state comparisons and downstream computational evidence.

0.23

Learning to Compare Neural Representations from Parameterized Computation Graphs

cs.LG ·2026-11-19 ·Ya-Wei Eileen Lin, Daniel Herbst, Stefanie Jegelka
similarity 0.58
Abstract

Comparing internal representations across trained models is important for identifying corresponding layers and selecting components for model reuse. Yet, direct comparisons at inference require evaluating candidate models on known reference inputs and collecting their internal activations. In this paper, we study whether internal representation similarity measured across independently trained models can instead be learned to predict in unseen models from their parameterized computations. We introduce a framework that applies a graph metanetwork for mapping each internal site to an embedding derived from its upstream computations while accounting for hidden-coordinate permutation and internal attention-basis transformations. We analyze the structural properties of our estimators and derive conditional guarantees for layer retrieval and stitching candidate selection. Empirically, we find that the learned predictors support relation estimation and low-regret layer retrieval. They also select useful stitching connections while not being trained on stitching outcomes. This opens up a new way to study representation similarity directly from trained neural networks: representational relations learned across a collection of trained models can generalize to new computations, and that the predicted relations can guide candidate selection before direct measurement or adapter fitting for functional evaluations.

0.12

TabNSM: Neural Sparse Mixer for Tabular Regression

cs.LG cs.CE ·2026-08-18 ·Ali Eslamian, Qiang Cheng
similarity 0.58
Abstract

Large-scale, high-dimensional tabular regression remains challenging: tree-based models are robust but lack end-to-end representation learning, while deep models enable flexible feature learning but often incur costly interaction modeling and sensitivity to noisy or redundant features. We propose TabNSM, a scalable regression framework that extends our earlier sparse-attention and mixer architectures. At its core, the Adaptive Sparse Interaction Module (ASIM) integrates foreground feature discovery, sparse local interaction encoding, and Feature-Token Mixing, providing near-linear complexity under fixed sparse configurations. For regression, TabNSM introduces three complementary components: a Multi-Stage Regression Head for progressive prediction refinement; GridLoss, an ordinal-aware soft-binning objective that incorporates target structure into representation learning; and RISE (Reweighted Instance Sampling by Error), a difficulty-aware sampling strategy based on loss-quantile bins. Across nine real-world regression benchmarks, TabNSM delivers strong predictive performance and practical scalability, with particularly consistent gains on high-dimensional and heterogeneous datasets. These results demonstrate that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.

0.21

RetroMPA: A Molecular Property-Aware Auxiliary Framework for Enhancing Retrosynthesis Prediction

cs.LG cs.AI ·2026-08-17 ·Mianzhi Liu, Fan Xiao, Zhiliang Yu, Huayang Huang, Yuke Li +3 more
publishedcode/data similarity 0.58
Abstract · Accepted for publication in Journal of Chemical Information and Modeling

Retrosynthesis is a cornerstone of drug discovery and organic synthesis. While data-driven deep learning models have shown remarkable progress, they autonomously learn reaction patterns from extensive datasets with limited integration of established chemical knowledge as priors. To address this limitation, we introduce RetroMPA, a molecular property-aware, post-hoc enhancement module that injects chemical knowledge into the retrosynthesis pipeline. Rather than functioning as an independent SMILES sequence generator, RetroMPA is a broadly applicable, model-agnostic chemical filter designed to recalibrate and optimize the predictive pathways of existing algorithms. This plug-and-play framework integrates seamlessly with a range of data-driven retrosynthesis methods, enhancing outputs without modifying model architecture or requiring resource-intensive retraining. By leveraging a property-aware latent embedding space, RetroMPA consistently improves top-1 accuracy across eight representative retrosynthesis models by an average of 5.50% on USPTO-50K. Furthermore, we validate its scalability on the large-scale USPTO-Full dataset, achieving an average improvement of about 2.03% across both template-based and template-free architectures. Wet-lab experiments provide preliminary support for the practical utility of the framework. These syntheses confirmed viable, previously unreported substrate combinations for classic reaction paradigms---specifically, Suzuki-Miyaura coupling, Bucherer reaction, and Friedel-Crafts acylation---suggesting that RetroMPA can operate beyond mere data fitting. The code is open-sourced at https://github.com/MengzhouLu/RetroMPA.

0.59