CS Paper Radar

Robust LassoNet: Enhancing Feature Selection in Neural Networks via Robust Loss Functions

Daniela De Canditiis, Italia De Feis, Paola Stolfi

cs.LG stat.MLmath.PR ·submitted 2026-09-29 12:49 UTC · v2

Abstract

Feature selection in neural networks remains a challenging problem, particularly in the presence of noisy or contaminated data. LassoNet is a recent approach that addresses this issue by combining neural networks with hierarchical sparsity constraints, enabling simultaneous prediction and variable selection. However, its standard formulation relies on the mean squared error (MSE) loss, which is known to be highly sensitive to outliers. In this paper, we present Robust LassoNet, an extension of LassoNet that incorporates robust loss functions, such as Huber, Cauchy, Tukey's bisquare, and Nonnegative Garrote, to mitigate the effect of extreme observations. The proposed approach preserves the original optimization framework while improving stability under data contamination. Through experiments on synthetic and real datasets, we show that robust LassoNet significantly improves both predictive performance and feature selection accuracy in the presence of outliers or heavy tailed noise, while maintaining comparable performance in clean settings. These results highlight the importance of robustness in neural network based feature selection and suggest practical guidelines for choosing appropriate loss functions.

Closest papers

Nearest neighbours by meaning across the whole index.

Rethinking Optimization Methods for Hierarchical Feature Selection

cs.AI ·2026-11-06 ·Xinyuan Song, Zekun Cai, Liang Zhao
code/data similarity 0.61
Abstract · Under review

Rare-event forecasting often depends on interpretable conjunctions: a single keyword, location, or temporal cue may be weak, while their joint occurrence can identify a meaningful precursor. Hierarchical-Task Numerical conjunctive feature Learning (HTNL) selects Numerical Conjunctive Features (NCFs) through a structured sparsity objective. However, the resulting optimization problem is challenging, making optimization design critical to its practical performance. We give a unified diagnosis of four HTNL optimization families: original majorization-minimization (MM), variational quadratic form (VQF), alternating direction method of multipliers (ADMM), and weighted group lasso (WGL). Our analysis distinguishes three key factors that jointly determine HTNL optimization behavior. First, VQF and MM are algebraically equivalent under matched variational updates and optimize a squared-penalty formulation. Second, ADMM optimizes the unsquared objective through consensus splitting; we replace its truncated FISTA proximal solver with a scalar bisection update (ADMM-Bisect) for more accurate treatment of near-threshold blocks. Third, WGL targets the same unsquared objective, while all methods face rapidly decaying active-set admission scores. Across two synthetic datasets and 11 civil-unrest and flu-surveillance tasks, VQF achieves the highest mean AUC with the most compact non-collapsing supports. ADMM-Bisect reduces measured runtime, while calibrated WGL avoids the empty-support failures of its original scale. Synthetic results further show how proximal accuracy, pruning, and search budget affect candidate recovery, especially for longer conjunctions. Code will be made publicly available upon publication.

0.34

Subspace Learning with Outlier Exposure for Out-of-Distribution Detection

cs.CV ·2026-11-05 ·Hongxu Shen, Kun Fang, Jiahui Zheng, Jian Sun, Qinghua Tao
code/data similarity 0.60
Abstract

Out-of-Distribution (OoD) detection is pivotal for reliable deployments of Deep Neural Networks (DNNs), aiming to distinguish anomalous samples from the In-Distribution (InD) data in training. By introducing outliers as surrogate OoD data to finetune DNNs, Outlier Exposure (OE) has greatly advanced OoD detection in recent years. Existing OE-based research remains predominantly logit-centric, either by generating diverse outliers or by regularizing them with objectives operating at the prediction-output level. Nonetheless, these approaches leave much of the rich information in feature spaces untapped. In this work, we propose an OE-based detection framework, named RecOE, that learns appropriate feature subspaces from InD data and auxiliary outliers, such that the InD-OoD discrepancy can be effectively characterized through the learned feature subspaces. Specifically, the feature subspaces are optimized to capture the dominant variations of InD and outliers by minimizing reconstruction errors therein. Besides, a sample-level objective discourages reconstruction in the opposing subspace, while an orthogonality regularizer reduces subspace overlap, further promoting discriminative subspaces. At inference, the difference in reconstruction errors between the learned subspaces serves as the OoD detection score. Extensive experiments show the superiority of RecOE on multiple benchmarks, in which significant performance gains are achieved over recent strong baselines with only slight extra overhead from two small-size matrices of subspace bases. The source code is available at https://github.com/shenhongxu123/RecOE.

0.40

RobustLDS: Learning linear dynamical systems under adversarial corruptions

cs.LG stat.MLeess.SYmath.OC+1 ·2026-10-08 ·Aravinda Kanchana Ruwanpathirana, Hemant Tyagi
similarity 0.58
Abstract · 40 pages, 8 figures

We consider the problem of learning linear dynamical systems under adversarial contamination from a single trajectory of length $T$. While identification of linear dynamical systems itself is well-studied, the problem of robust system identification under adversarial contamination is relatively less explored. In this work, we study the setting where a fraction of the $T$ observations are contaminated by adversarial outliers. We propose different estimators based on relaxations of least-trimmed squares along with an alternating minimization algorithm. Furthermore, we also propose two estimators which exploit the group-sparsity (through penalization/hard-constraints) of the outliers. For the estimator with group-sparse penalty, we derive non-asymptotic error bounds which establish its robustness to outliers. We also show empirically that the proposed estimators work well in practice.

0.24

Sparsity-Adaptive Sharpness-Aware Minimization

cs.CV ·2026-09-13 ·Shiryu Ueno, Yoshikazu Hayashi, Kunihito Kato
published similarity 0.58
Abstract · IJCNN 2026

Deploying deep neural networks in real-world settings requires models that are both compact and robust to common corruptions. However, at deployment-relevant high sparsity, standard pruning pipelines often degrade corruption robustness, and existing sharpness-aware training/pruning approaches provide limited robustness gains. We address this issue by introducing Sparsity-Adaptive Sharpness-Aware Minimization (SA-SAM), which derives a sparsity-dependent SAM/ASAM perturbation radius by keeping the mean absolute perturbation (an $\ell_1$-based proxy) approximately invariant as sparsity increases. As a simple complementary option, we evaluate Magnitude-Weighted Hessian (MWH), derived from a second-order removal-path analysis, yielding an importance proportional to $\mathrm{Diag}(F)_i\,|w_i|$, where $\mathrm{Diag}(F)$ is the diagonal empirical Fisher used as a curvature proxy in our implementation. Across CIFAR-10-C, CIFAR-100-C, and ImageNet-100-C, our approach achieved stronger corruption robustness than the considered pruning baselines at 80--90\% sparsity, while preserving clean accuracy. We additionally quantify the robustness--throughput trade-off by reporting measured inference throughput under sparse execution at deployment-relevant sparsity levels.

0.34

Integrated Feature Selection for classification and Representation Learning with Group-Lasso Siamese Networks

cs.AI ·2026-10-29 ·Nisargo Ranjan Pal, Pradeep Singh
benchmark similarity 0.58
Abstract

The Least Absolute Shrinkage and Selection Operator (Lasso) has been widely employed for feature selection in linear models and has also been extended to multi-layer perceptron (MLP) networks for both feature and group feature selection. Group-Lasso has also been used in feature selection using feed-forward neural networks. To the best of our knowledge, this is the first study to incorporate the Group-Lasso principle into a Siamese network architecture for feature selection. Unlike existing approaches that primarily optimize classification accuracy, the proposed method aims to identify a compact subset of features that simultaneously preserves the intrinsic representation of the input data (the goal of Siamese network) and is expected to achieve high classification performance. Two variants of the proposed framework are developed: one based on the Euclidean distance and the other on a modified Jaccard similarity measure for real-valued vectors. The latter is specifically designed to improve feature selection for high?dimensional sparse data. The effectiveness of the proposed approach is demonstrated on 12 benchmark datasets with feature dimensions ranging from 6 to 7070 using eight different classifiers. Experimental results demonstrate that the proposed Siamese Lasso framework consistently identifies compact and discriminative feature subsets maintaining competitive or superior classification performance, although the loss function used explicitly did not consider any regularizer to learn classification. In addition, the quality of the embeddings produced by these networks is assessed using manifold learning techniques such as UMAP and t-SNE.

0.31

Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling

cs.LG stat.MLstat.ME ·2026-09-21 ·Xuhui Liu, Lili Zheng
similarity 0.57
Abstract

Black-box machine learning models increasingly deliver strong predictions, but extracting useful information from them, such as a set of important features, remains challenging. Existing model-agnostic methods primarily estimate feature importance or conduct inference on it rather than directly selecting features, whereas many feature selection methods are model-specific or rely on the model-X assumption. We introduce LOCO-guided Adaptive Minipatch Sampling (LAMPS), a model-agnostic ensemble framework that uses any black-box regression algorithm as its base learner to select features important for predicting the response. The base learner need only produce predictions and need not perform feature selection itself. LAMPS operates within a minipatch ensemble framework that subsamples both observations and features, allowing leave-one-covariate-out (LOCO) feature importance scores to be easily computed. It adaptively concentrates minipatch sampling on features with high LOCO scores while maintaining exploration. The resulting sampling probabilities rapidly separate signal from noise features after a few iterations, enabling selection through simple thresholding. We establish that LAMPS achieves exact feature selection in high-dimensional settings, provided that the base predictive models are sufficiently well trained on average. Extensive experiments on synthetic and real data show that LAMPS outperforms state-of-the-art feature selection methods, with particularly strong performance in the presence of correlated features.

0.34

Robust CurveMoE: Multi-Norm Adversarial Defense for Mixture-of-Experts Models via Mode Connectivity

cs.LG ·2026-08-26 ·Xu Zhang, Ren Wang
similarity 0.57
Abstract

Multi-norm adversarial defense aims to protect neural networks against perturbations defined by different norm constraints, but existing methods typically optimize competing robustness objectives within a single parameter configuration, leading to substantial training cost and unfavorable robustness trade-offs. We propose Robust CurveMoE, an efficient mixture-of-experts framework that connects models specialized for different perturbation norms through a low-loss path and exploits the complementary robustness profiles of models along this path. Robust CurveMoE derives clean and norm-specialized experts from robustness-constrained curve locations and selectively expertizes only influential layers, while sharing the remaining parameters across routing paths. To further reduce curve-construction cost, we introduce contribution-guided partial updating, which selects influential curve parameters using initialization-based gradient scores. We also theoretically bound the objective gap between partial and full curve optimization. Experiments on CIFAR-100 and ImageNet-100 with WideResNet and Vision Transformer architectures show that Robust CurveMoE consistently improves clean, norm-specific, and Union accuracy over MSD and ERMC. In particular, it improves Union accuracy by 2.37 and 2.13 percentage points over the strongest baseline on CIFAR-100 and ImageNet-100, respectively. Extensive ablations further validate the effectiveness of partial updating, selective expertization, and robustness-constrained expert selection.

0.19

High-probability guarantees for linear accessibility in feature superposition

cs.AI stat.MLcs.IRcs.LG+1 ·2026-09-09 ·Enrico Vompa
similarity 0.56
Abstract · preprint

Neural networks can leverage feature superposition to encode more concepts than dimensions, but cross-feature interference constrains the linear accessibility of simultaneously active features. By framing linear accessibility as a compressed sensing problem, we derive high-probability bounds for fixed supports under subgaussian noise, proving the sufficient dimension scales linearly ($d=O_{\varepsilon}(k \log m)$) rather than prior worst-case quadratic limits. We characterize the asymmetry between active and inactive interference and the trade-off between interference and observation-noise budgets. We then validate these bounds across system parameters through Gaussian-tail approximations. We also introduce IHT-SAE, which uses learned iterative refinement to improve feature recovery beyond the limits of linear availability. These results quantify the geometric constraints of the linear representation hypothesis, providing a framework for evaluating sparse autoencoders, compositional generalization, and neural interpretability.

0.18