CS Paper Radar

Truncated Out-of-Bag Probability Gain: A Noise-Robust Criterion for Random-Forest Wrapper Feature Selection

Sejong Oh

cs.LG ·submitted 2026-10-21 04:35 UTC · v1
benchmark

Abstract

Random-Forest wrapper feature selection can use out-of-bag (OOB) predictions as a built-in validation signal, but the criteria that decide each forward step accumulate OOB estimation noise. We propose TOP-Gain (Truncated Out-of-bag Probability Gain), which scores a candidate feature by the mean sample-level increase in the OOB probability of the true class, with decreases truncated at zero. Under a simple noise model, truncation removes about two-thirds of the noise variance contributed by unaffected samples while preserving the signal, and it makes the ranking of candidate features markedly more reproducible. In a pre-registered benchmark on 26 OpenML-CC18 datasets, TOP-Gain achieved the best mean rank in area under the ROC curve (ROC-AUC), accuracy and Macro-F1 among 15 methods; it was significantly better than 13 of 14 baselines and matched the accuracy of VSURF with 21 times less single-core computation. Within an identical wrapper, it selected significantly more stable feature sets than net-gain, Brier, log-loss and accuracy criteria without loss of accuracy. A confirmatory study on 24 further datasets replicated the advantage over all baselines. The accuracy gains were broad rather than tied to specific dataset characteristics, and, as pre-registered, the stability gain grew as relevant features became sparser

Closest papers

Nearest neighbours by meaning across the whole index.

ARISE: An adaptive residual-informed stability ensemble for feature selection in small-sample biomedical omics

cs.LG stat.ML ·2026-08-14 ·Zardad Khan, Amjad Ali, Naz Gul, Saeed Aldahmani
similarity 0.61
Abstract · 30 pages, 6 figures

Objective: Small-sample molecular classification requires feature selectors that identify predictive, stable, and nonredundant subsets for binary and multiclass outcomes. We propose ARISE (Adaptive Residual-Informed Stability Ensemble), which integrates complementary relevance signals, class-balanced stability assessment, residual-informed redundancy control, and multiclass pairwise coverage. Methods: ARISE combines seven percentile-normalized relevance components through 15 predefined profiles, adaptively weighted by nested inner cross-validation. It was evaluated on five molecular datasets, eight feature-set sizes, three fixed classifiers (k-nearest neighbours, support vector machine, and random forest), and six filter comparators. Generalization was estimated by five-fold outer cross-validation repeated 50 times using balanced accuracy, macro-F1, and Cohen's kappa. Results: Across 210,000 held-out assessments, ARISE ranked first in all 15 dataset-metric combinations. Equal-dataset means were 0.793 for balanced accuracy, 0.776 for macro-F1, and 0.725 for kappa, exceeding the strongest aggregate comparator by 0.022, 0.023, and 0.028, respectively. Performance remained strong across compact feature sets, although the optimal budget differed by dataset. Conclusion: ARISE provides a transparent, adaptive framework that jointly addresses relevance, stability, redundancy, and multiclass discrimination. Its consistent results across datasets, classifiers, metrics, and feature-set sizes support further evaluation for small-sample molecular classification.

0.10

JevForest: Path Voting for Budgeted Feature Acquisition

cs.LG stat.ML ·2026-10-07 ·Yu Yan
similarity 0.60
Abstract

Choosing which information to observe is central to prediction under limited observation budgets. We study JevForest, a feature acquisition policy that aggregates path-dependent proposals from bootstrapped trees, weights them by global training information gain, and predicts from the acquired values with a shared masked classifier. An online implementation queries Jev for semantic answers selected by this policy. On small balanced held-out samples, four-question forest acquisition achieves accuracy $0.729$ on AG News ($n=48$), compared with $0.667$ for a static gain ranking and $0.583$ for random ordering. On TREC ($n=24$), the ordering reverses: forest accuracy is $0.667$, compared with $0.750$ and $0.833$. Asking all eight questions in one batch yields higher accuracy at lower measured cost and latency than four sequential forest queries; direct Jev classification matches the batch accuracy while costing less. Offline MiniBooNE experiments yield accuracy $0.845\pm0.010$ at ten features and $0.885\pm0.008$ at forty features over three jointly varying data and forest seeds (mean $\pm$ sample standard deviation). A companion Newton boosting implementation provides preliminary full-feature synthetic results. These exploratory findings establish a working Jev acquisition workflow but do not support a general advantage for path voting: its value depends on the task, predictor, and the distinction between question budgets and actual query costs.

0.25

Global-Local and Window-Concatenated Functional Random Forests: Early and Late Fusion of Window-Specific Functional Principal Components

cs.LG stat.MLstat.ME ·2026-11-11 ·Fabrizio Maturo, Annamaria Porreca
similarity 0.59
Abstract · 36 pages, 12 figures, 9 tables

Functional random forests built on a single functional principal component decomposition of the whole domain may dilute features confined to short intervals, whose location is rarely known in advance. We propose two random forests based on window-specific functional principal components. The global-local functional random forest (GL-FRF) is a late-fusion method that stacks a tuned full-curve forest and one forest per window with penalised convex weights; it contains the full-curve forest as a feasible solution, and its weights indicate which parts of the domain drive the prediction. The window-concatenated functional random forest (WC-FRF) is an early-fusion method that trains a single tuned forest on the concatenated window-specific scores. On twenty UCR datasets, GL-FRF improved on a tuned full-curve forest with the same number of trees on 16, with Brier score reductions of up to 0.21, significantly after Holm adjustment. No method was best on more than six datasets. WC-FRF was more accurate than GL-FRF on 13 datasets but failed when the signal required the full curve, whereas GL-FRF had the smallest mean and maximum excess over the best method across all 32 simulated and real settings, which makes it the safer choice when the location of the signal is unknown.

0.33

Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling

cs.LG stat.MLstat.ME ·2026-09-18 ·Quoc Viet Le, Joonha Park
similarity 0.58
Abstract · 29 pages (10 main text, 19 pages appendix), 21 tables, 3 algorithms. No figures

We revisit Breiman's observation that reducing inter-tree correlation without weakening individual trees can improve random forests. Building on this principle, we introduce two variants: Dirichlet-Multinomial Bagging Random Forest (DM) and Dirichlet-Weighted Random Forest (DW). Both modulate sample reweighting via a concentration parameter $α>0$. We provide a simple theoretical criterion that clarifies when these variants behave indistinguishably from standard random forests, and we use it to guide a lightweight tuning strategy. In a controlled evaluation on public classification benchmarks, DM and DW are consistently competitive and often stronger than other random-forest (RF) baselines, with negligible additional runtime.

0.33

Model-Agnostic Feature Selection via LOCO-Guided Adaptive Minipatch Sampling

cs.LG stat.MLstat.ME ·2026-09-21 ·Xuhui Liu, Lili Zheng
similarity 0.57
Abstract

Black-box machine learning models increasingly deliver strong predictions, but extracting useful information from them, such as a set of important features, remains challenging. Existing model-agnostic methods primarily estimate feature importance or conduct inference on it rather than directly selecting features, whereas many feature selection methods are model-specific or rely on the model-X assumption. We introduce LOCO-guided Adaptive Minipatch Sampling (LAMPS), a model-agnostic ensemble framework that uses any black-box regression algorithm as its base learner to select features important for predicting the response. The base learner need only produce predictions and need not perform feature selection itself. LAMPS operates within a minipatch ensemble framework that subsamples both observations and features, allowing leave-one-covariate-out (LOCO) feature importance scores to be easily computed. It adaptively concentrates minipatch sampling on features with high LOCO scores while maintaining exploration. The resulting sampling probabilities rapidly separate signal from noise features after a few iterations, enabling selection through simple thresholding. We establish that LAMPS achieves exact feature selection in high-dimensional settings, provided that the base predictive models are sufficiently well trained on average. Extensive experiments on synthetic and real data show that LAMPS outperforms state-of-the-art feature selection methods, with particularly strong performance in the presence of correlated features.

0.34

Rank-1 Weight-Corrected Trainable Weight Averaging for Out-of-Distribution Detection

cs.CV cs.AIcs.LG ·2026-10-24 ·Akito Shinohara, Hiroaki Aizawa
similarity 0.56
Abstract · Accepted to ACCV2026 Workshop on Subspaces in the Era of Foundation Models

Out-of-distribution (OOD) detection identifies inputs outside the training distribution. Rank-based analyses such as RankFeat and RankWeight suggest that OOD overconfidence is closely related to dominant low-rank components in feature and weight spaces. Since weight averaging operates directly in parameter space and improves model generalization, we consider it for OOD detection from a dominant-component perspective. Trainable Weight Averaging (TWA) assigns a single coefficient to each checkpoint, limiting independent control of dominant weight components. We extend TWA with leading rank-one weight components extracted by Singular Value Decomposition as correction directions, whose contributions are learned through the in-distribution classification objective. On ResNet50, our method improves over fine-tuning and standard TWA across the evaluated datasets with MSP and RankWeight, both with and without Outlier Exposure. Mahalanobis-based evaluation yields mixed results, indicating that the benefit depends on the OOD scoring protocol. Target-layer analyses with MSP and Energy favor correction near the classifier.

0.40

Big Models, Small Gains: Frozen Forecasters as Classifiers on MONSTER

cs.LG cs.AI ·2026-11-04 ·Rishit Jain, Rishit Kar, Chetashri Bhadane, Nilesh Patil
benchmarkcode/data similarity 0.56
Abstract · Asian Conference on Machine Learning, FMLTS Workshop, 2026

Forecasting foundation models are increasingly used as frozen feature extractors for time series classification, but it remains unclear whether their pretrained representations offer an advantage over strong classifiers that do not require pretraining. This raises an important question: do these representations provide useful information beyond what the model architecture alone can capture, particularly when labelled data are scarce? We present a controlled benchmarking of two forecasting models, Reverso and Chronos-Bolt, on six large datasets from the MONSTER benchmark, using 1%, 10%, and 100% of the available training labels. We compare their frozen representations against four established time series classification methods and random-weight versions of the same forecasting models. Our results show that pretraining improves classification accuracy over random-weight counterparts on five of the six datasets, with gains of up to 23 percentage points. However, the frozen representations do not reliably outperform the strongest non-pretrained baseline in any setting: they perform worse in 15 of 17 settings and tie in the remaining two. Further experiments show that restoring amplitude information recovers part of the performance gap, while fine-tuning and forecast-error features do not close it. These findings highlight the importance of strong task specific baselines and random weight controls when evaluating forecasting foundation models for classification. Our code is available at https://github.com/Rachet2006/frozen-forecasters-monster.

0.31

Validation-Frontier Representation Selection under Constrained Observation

cs.AI ·2026-08-15 ·Wesley Shu
benchmark similarity 0.56
Abstract

AI systems deployed outside clean benchmark settings often rely on observations that are incomplete, unstable, costly, or degraded by monitoring failures. This paper studies representation selection under constrained observation: choosing a state representation when raw accuracy is not the only operational criterion. We propose a validation-frontier selector that combines balanced accuracy with penalties for feature cost, overfit gap, and validation-test instability. In a focused public-tabular benchmark using three scikit-learn datasets, five observation regimes, 45 matched task cells, 720 candidate actions, and 405 representation rows, the adaptive selector improves frontier score over full trace features by 0.025801 while reducing mean feature count by 22.733. Balanced-accuracy difference is small and not statistically significant. A broader offline stress test gives mixed results. The supported claim is therefore bounded: adaptive representation selection can improve a constrained-observation robustness-efficiency frontier in matched benchmark settings, but does not universally dominate trace baselines.

0.04