CS Paper Radar

Sparse Regression Distilled from a Single Robust Fit

Wooyoung Shin, Seunghwan Park

cs.LG stat.MLstat.ME ·submitted 2026-09-20 23:30 UTC · v1

Abstract

Robust linear fits can resist response contamination yet remain too dense or unstable for useful global explanations. We propose penalized distillation, which fits a smoothly clipped absolute deviation (SCAD) estimator to a robust initial estimator's empirical fitted surface along a safeguarded coordinate-descent path and evaluates candidate states separately for fidelity, parsimony, perturbation stability, and held-out prediction. The new results attach to the states the algorithm actually computes. Conditional on a fixed uncontaminated design, deterministic bounds transfer response-replacement boundedness from the initial fit to every retained path state. Turning to fixed dimension, we characterize the oracle-support branch by its empirical-Gram projection and influence function, give conditions for covariance-weighted least-squares approximation equivalence, and establish a path-conditional generalized information criterion. By contrast, at large dimension-to-sample ratios the full-coordinate robust fit collapses without warning, and screening restores the construction. Under a sure-screening framework, the robustness bound and the support and selection guarantees transfer to the screened fit. Simulations separate robustness transfer from support recovery, efficiency, and computation across the dimension-to-sample ratio, with p up to 240, and the signal density, which isolates what the sparse stage adds once the screen over-selects. In a duplicate-grouped superconductivity study, the distilled estimator remains predictively stable under prespecified training-response shifts but retains 66.8--68.8 of 81 slopes. Stronger sparsification reduces the model to 12.6--14.0 slopes only at visible fidelity and prediction cost. Distillation therefore preserves predictive stability on these data without substantiating a compact coordinate-level explanation.

Comment
68 pages, including appendices; 8 figures

Closest papers

Nearest neighbours by meaning across the whole index.

Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectification

cs.LG cs.AI ·2026-09-17 ·Zhilong Zheng, Letian Tao, Yujie Yang, Wei Xiong, Kehua Sheng +5 more
similarity 0.58
Abstract

Fine-tuning foundation models on new tasks inevitably suffer from catastrophic forgetting. While existing works attempt to mitigate this on the basis of parameter-efficient fine-tuning methods, they adopted an overly restrictive Subspace Orthogonality condition. In this paper, we introduce a purely post-hoc and tuning-agnostic weight rectification framework that achieves Parameter Space Orthogonality, which is the necessary and sufficient condition for preserving historical performance to the first order. By projecting parameter updates into the JAcobian NUll Space (JANUS), our method significantly recovers compromised historical knowledge without interfering with the underlying fine-tuning process. To overcome the local validity of the Jacobian approximation, we further propose a Multi-step Adaptive Rectification mechanism that utilizes the JANUS shift to dynamically verify the valid trust region and adjust step sizes. Coupled with our proposed ghost projection, ghost orientation comparison, and sequence-level singular value decomposition compression techniques, JANUS also achieves great temporal and spatial efficiency. Experiments demonstrate that JANUS seamlessly integrates with various fine-tuning methods, significantly mitigating the stability-plasticity dilemma by recovering historical knowledge while preserving downstream task adaptation.

0.26

Adaptive Regularization for Random Features: A Neighboring Early-Stopping Rule with Oracle-Rate Guarantees

cs.LG stat.ML ·2026-08-26 ·Caixing Wang, Zhibo Chen, Yue Wang
hot direction similarity 0.56
Abstract · 31 pages, 10 figures, 3 tables

Random feature methods provide a scalable approximation to kernel ridge regression (KRR), but the regularization parameter that yields the oracle learning rate depends on unknown smoothness and capacity parameters. In this work, we propose a neighboring early-stopping rule for adaptive regularization in KRR with random features (KRR-RF). The method uses a grid that is uniform in inverse regularization and compares only adjacent estimators, reducing the number of discrepancy comparisons relative to standard all-pairs Lepskii-type procedures. Both the neighboring discrepancy and its empirical complexity term can be computed directly in the random feature space, without constructing the exact kernel Gram matrix. We establish a high-probability comparison bound for neighboring KRR-RF estimators and show that, under standard source and capacity conditions together with suitable grid and random feature budget conditions, the selected estimator attains the oracle polynomial learning rate up to logarithmic factors. The result allows the regularization parameter to be selected without prior knowledge of the source and capacity exponents and covers both well-specified and partially misspecified regimes. Our analysis is based on an empirical random feature effective dimension that connects the observable stopping threshold with the population complexity of the random feature model. Simulation and real-data experiments illustrate the prediction performance and computational behavior of the proposed method in comparison with standard tuning procedures.

0.33

A Unified Descriptive-Complexity Framework for Model Selection under Correlated Designs

cs.LG stat.ML ·2026-08-27 ·Yanhang Zhang, Wei Liu, Yuhong Yang
benchmark similarity 0.56
Abstract · 73 pages, 14 figures, 4 tables

Model selection becomes particularly challenging under strong predictor dependence and model-class uncertainty, especially when there are exponentially many models. We propose a Descriptive-Complexity Information Criterion (DCIC) that regularizes large candidate model collections through Kraft-admissible code lengths. Under sub-Weibull noise, we establish selection consistency through approximation-error separation without relying on RIP-type conditions, together with nonasymptotic oracle risk bounds that remain valid under model misspecification. The same coding principle places heterogeneous classes on a common complexity scale at a small additional class-identification cost. This extension yields class--model recovery under suitable identifiability conditions and risk adaptation across classes. We further develop a complexity-guided search path that makes the computation--statistics trade-off explicit. Large penalties yield polynomial-size retained search regions with high probability, whereas smaller penalties sharpen the oracle risk benchmark. Numerical experiments illustrate stable support recovery and favorable estimation performance under strong dependence and model-class uncertainty.

0.27

Kernelized Parameter Sensitivity Atlases for Amortized Conditional Interpretability

cs.LG cs.CVstat.ML ·2026-10-08 ·Davide Wiest, Ahanaf Hasan Ariq, Abdul Monaf Chowdhury
code/data similarity 0.55
Abstract · 22 pages, 10 figures. Code: https://github.com/Axym-Labs/kpsa

Gradient-based attribution provides input-specific information about model parameters, but computing it via backpropagation for many samples is expensive, while static importance scores discard input dependence. We derive KPSA (Kernelized Parameter Sensitivity Atlas) from a block decomposition of the empirical neural tangent kernel. It measures normalized parameter-group sensitivity, embeds those sources in an explicit query space, and uses a lightweight kernel to estimate sensitivities for unseen inputs. We derive bounds that separate finite-source from irreducible representation error and give margin-based top-$k$ recovery. This paper studies two interpretability applications. Firstly, KPSA identifies parameter groups with high susceptibility to perturbations across vision, forecasting, and protein-modeling setups. Across a fivefold range of parameter budgets ($0.02$-$0.1\%$), its selected groups attain $39$-$85\%$ of the perturbation susceptibility of a first-order backpropagation oracle. In vision and forecasting, these parameter groups form sparse sensitivity circuits that extend across depth and vary systematically with the input. Secondly, we use KPSA on an activation-attribution problem for MLP features in transformers. It retains $81\%$ of the gap between a backpropagation oracle and a constant-sensitivity baseline on vision while increasing batched query-time throughput from 551 to 2,041 images/s and reducing peak GPU memory from 4.55 to 0.44GB. On the protein-modeling task, it recovers $68\%$ of this gap. An ablation of KPSA's kernel aggregation shows that nearest-neighbor retrieval in KPSA space can remain competitive. The empirics show that compact kernel atlases can store useful input-dependent parameter sensitivity while amortizing repeated gradient attribution, a step toward scalable conditional interpretability. Code can be found at https://github.com/Axym-Labs/kpsa .

0.35

Robust Data-Driven Output-Feedback Reference Governor with Quantified Model Uncertainty

eess.SY ·2026-11-16 ·Tam W. Nguyen
benchmark similarity 0.55
Abstract · 45 pages, 4 figures, 2 tables

This paper constructs a data-driven reference governor (DDRG) from input--output data and supplied deterministic disturbance bounds. A data-consistent model set quantifies the uncertainty remaining in each prediction direction. With exact output measurements, the DDRG propagates output-history uncertainty while retaining exact applied-reference histories and converts the resulting tubes into constraint margins. Under a common contraction condition in a selected history norm, explicit terminal feasibility inequalities certify infinite-horizon continuation under a held reference and yield a closed invariant admissible set. A scalar update selects the largest certified step toward the requested reference, with recursive feasibility and constraint satisfaction from certified initialization. A separate bounded-measurement-noise extension derives the errors-in-variables model set and the additional physical-output margin through a measured-history certificate. On the coupled benchmark, longer nested records reduce one-step prediction disagreement at the probe regressor, and propagation accounts for most of the gap between the tube and true-model envelopes at the reported constraint row and prediction step.

0.28

Coefficient Calibration as Selection Pressure in Symbolic Regression

cs.LG ·2026-10-07 ·Mattia Billa, Veronica Guidetti, Federica Mandreoli
similarity 0.55
Abstract · 18 pages, 8 figures, 7 tables

In memetic symbolic regression, candidate structures are compared after coefficient calibration, so the calibration protocol itself contributes to evolutionary selection. Standard centralized calibration evaluates each structure at its pooled-sample optimum, ignoring how stable this calibration is under covariate shifts, and can thus favor structures whose fit relies on sample-specific coefficients. We propose Dirichlet-Sinkhorn Constant Averaging (DSCA), a calibration strategy that partitions the optimization data into equally sized subsets with different covariate distributions, calibrates each candidate independently on every partition, and evaluates it at the mean of the resulting parameters. We show that the excess loss of DSCA relative to centralized calibration vanishes at the population level for correctly specified, identifiable expressions, whereas under misspecification it persists when partition-specific calibrations do not aggregate to the pooled optimum. On synthetic benchmarks and ten real-world datasets, DSCA improves functional recovery and the accuracy-complexity trade-off over centralized Broyden-Fletcher-Goldfarb-Shanno and Levenberg-Marquardt calibration, under selection by negative log-likelihood and by the Akaike and Bayesian information criteria. Mechanism analyses associate the DSCA excess loss with the generalization gap and show that the effect is not reproduced by repeated centralized fitting. These results indicate that controlled heterogeneous calibration provides a complementary source of selection pressure in symbolic-regression search.

0.28

Screening Collapse under Recursive Synthetic Data

cs.LG stat.ML ·2026-10-11 ·Jia Zhang, Xin Chen
similarity 0.55
Abstract · 20 pages, 5 figures

Recursive synthetic-data training can degrade a fitted model, but its effects on the variable-screening stage that precedes model fitting are less understood. When each generation labels data for the next, a marginal screener ranks variables against the current pseudo-label target rather than the original response. Across generations, this target drift can erode or reverse the active--inactive score margin and cause screening to discard variables that clean-label screening would retain. We call this process \emph{screening collapse}. In a Gaussian linear model with minimum-norm recursive generators, we establish finite-sample screening bounds, characterize a late-generation information-loss limit, and derive a minimax impossibility result. Screening collapse can precede and compound ordinary model collapse: omitted variables incur a covariance-weighted loss that no refit restricted to the selected variables can eliminate, while recursive labels further degrade estimation on the retained variables. We also examine two approaches to mitigating screening collapse. Fixed-fraction real-label retention preserves a clean-label component but leaves resolution fixed, whereas data accumulation can avoid screening collapse if the pooled target preserves a positive margin and the growing sample resolves it. Numerical experiments on gene-expression data support the theoretical findings.

0.23

Robust Local Optimization Done Right

cs.CV cs.MS ·2026-10-05 ·James Pritts, Kevin Köser
unusual similarity 0.54
Abstract

RANSAC scoring and local optimization (LO) impose different robustness requirements, motivating the separation of hypothesis selection from refinement. We systematically isolate the effects of robust-loss shape, incorrectly specified inlier scales, and optimization strategy on essential matrix, fundamental matrix, and homography estimation. A profile-marginal score marginalizes the nuisance inlier scale and selects an inlier partition, from which we estimate the scale that sets the LO loss width; this makes LO robust to an inlier scale specified too large, whereas one specified too small degrades selection itself. Refinement needs gradient from correspondences the seed currently rejects: optimizers that reweight from current residuals stay pinned to their seed, whereas methods with broad basins recover strongly perturbed seeds yet degrade accurate score-selected hypotheses, so basin size alone is insufficient to assess RANSAC LO. Joint half-quadratic optimization balances the two and is the most consistent strategy across model classes. An optimizer matched to the profile-marginal score, which never decreases it, does not reach the best accuracy, challenging the prescription that scoring and refinement objectives should match. Composed from these findings, our RANSAC reduces the median essential-matrix pose error of a state-of-the-art RANSAC on PhotoTourism from 2.23 degrees to 1.58 degrees with a correctly specified inlier scale and from 38 degrees to 6.2 degrees when it is grossly misspecified (128x too large).

0.23