CS Paper Radar

Validation-Frontier Representation Selection under Constrained Observation

Wesley Shu

cs.AI ·submitted 2026-08-15 07:40 UTC · v1
benchmark

Abstract

AI systems deployed outside clean benchmark settings often rely on observations that are incomplete, unstable, costly, or degraded by monitoring failures. This paper studies representation selection under constrained observation: choosing a state representation when raw accuracy is not the only operational criterion. We propose a validation-frontier selector that combines balanced accuracy with penalties for feature cost, overfit gap, and validation-test instability. In a focused public-tabular benchmark using three scikit-learn datasets, five observation regimes, 45 matched task cells, 720 candidate actions, and 405 representation rows, the adaptive selector improves frontier score over full trace features by 0.025801 while reducing mean feature count by 22.733. Balanced-accuracy difference is small and not statistically significant. A broader offline stress test gives mixed results. The supported claim is therefore bounded: adaptive representation selection can improve a constrained-observation robustness-efficiency frontier in matched benchmark settings, but does not universally dominate trace baselines.

Closest papers

Nearest neighbours by meaning across the whole index.

Adaptive Ensemble Selection for Noisy Labels on Tabular Data

cs.LG ·2026-09-26 ·Faizaan Ali, Inwon Kang, Oshani Seneviratne
benchmark similarity 0.63
Abstract

Incorrect or corrupted labels in tabular datasets can significantly degrade supervised learning performance, particularly when mislabeling is subtle and not easily detectable from feature space alone. In the context of automated or AI-augmented data science workflows, robust detection of such label noise is critical for building reliable models. We propose a data-centric reasoning module for AI data science systems that automatically diagnoses dataset quality and selects appropriate cleaning strategies. Given a dataset, a meta-model predicts weights over a diverse set of detectors, including confidence-based, neighborhood-based, and distributional methods. Across benchmark datasets with controlled noise, our approach achieves performance comparable to a Confident Learning baseline on average, with dataset-dependent gains and losses, particularly in heterogeneous regimes. We further show that detector effectiveness is systematically linked to dataset properties. These results demonstrate the value of descriptor-driven, data-centric ensembling as a component of AI-assisted data-science pipelines for robust dataset assessment and model reliability.

0.23

Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection

cs.CL cs.AIcs.LG ·2026-08-20 ·Atsuyuki Miyai, Kiyoharu Aizawa, Toshihiko Yamasaki
code/datahighly cited similarity 0.60
Abstract · v2: Fix typo in v1, Github: https://github.com/Agent4Science-UTokyo/Task-CoEvolve

We present a novel approach to efficient LLM harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose $\textbf{Task-CoEvolve}$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms subset-based baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.

0.64

SelectArena: A Modern Benchmark for Feature Selection on Real-world Tabular Data

cs.LG cs.AI ·2026-10-09 ·Bastian Schäfer, Dominika Matus, Lennart Purucker, Ricardo Knauer, Erik Rodner +3 more
benchmark similarity 0.60
Abstract

Selecting a subset of features (columns) from a tabular dataset is a ubiquitous challenge in real-world applications. Industry experts seek to minimize model training costs, climate scientists require an interpretable subset of factors to understand extreme weather events, and clinicians need to choose among diagnostic tests due to financial and time constraints. When looking at the scientific literature to determine which feature selection method to use, practitioners across domains face an unrepresentative landscape of benchmarking. Despite extensive research, existing feature selection benchmarks provide limited guidance because they rely on outdated models, ad hoc evaluation protocols, or artificially generated data that do not reflect real-world problems. To close the gap between scientific benchmarking and real-world applications, we introduce SelectArena, a modern benchmark for feature selection. We compare 15 widely used feature selection methods on 71 manually curated real-world tabular datasets, covering classification and regression problems, using standardized evaluation protocols and state-of-the-art tabular (foundation) models. Our results reveal clear trade-offs across predictive performance, runtime, stability, and validity. Tree-based methods, CART and RFImportance, along with F-Test are Pareto-optimal when taking into account predictive performance and runtime. F-Test, information-theory-, and distance-based methods lead on stability and validity, with F-Test emerging as a balanced choice across our metrics. Search-based methods are slow, unstable and achieve low performance. SelectArena turns these findings into practical guidance and, as an extensible platform, supports future feature selection research.

0.21

When Do LLMs Help with Feature Selection? A Systematic Empirical Study

cs.LG cs.AI ·2026-10-20 ·Fateme Memar, Dongjie Wang, Arvin Agah
similarity 0.59
Abstract · 8 pages, 6 figures, 2 tables

Large language models (LLMs) can leverage semantic feature names alongside summary statistics for tabular feature selection, yet it remains unclear when such information improves subset selection relative to established statistical methods. We investigate this question within an 18-dataset framework comprising 10 datasets for Group Relative Policy Optimization (GRPO) training, 2 for checkpoint validation, and 6 held-out evaluation datasets. We compare five classical baselines, frozen Qwen2.5-7B and Llama-3.3-70B selectors, and two GRPO-adapted Qwen2.5-7B models, reporting classification ROC-AUC and regression R^2 separately and using dataset-wise ranks for cross-task comparison. Across five seeded evaluation partitions, classical selectors generally achieve higher predictive performance than the tested 7B LLM/RL selectors, including under a pre-specified ~33% feature budget with strict train-only classical fitting, while the 7B selectors retain compact subsets. A cardinality-matched semantic sensitivity experiment shows that, for Zero-shot Qwen-7B, informative feature names improve regression performance over anonymized and shuffled alternatives (+0.1964 and +0.1606 mean R^2), whereas aggregate classification differences are smaller (+0.0159 and +0.0084 ROC-AUC) and dataset-dependent. These results characterize when semantic information benefits LLM-based feature selection and show that classical selectors remain strong baselines for predictive tabular feature selection.

0.30

Cross-Scenario Industrial Defect Detection via Domain-Robust Training and Metric-Aligned Selection

cs.CV ·2026-10-14 ·Dikai Li, Weijia Wang, Yuan-Gen Wang
top venuecode/data similarity 0.59
Abstract · This paper has been accepted for presentation in IEEE ICME 2026

Automated defect detection of industrial boards must remain highly reliable under cross-scenario variations in lighting, surface contaminants, and production line conditions. This makes cross-scenario defect detection a domain generalization problem, wherein models are required to detect, localize, and classify defects across unseen scenarios while minimizing false alarms on clean boards. For the IDA 2026 Track 1 challenge, we propose a cross-scenario defect detection method that includes four complementary modules: cross-scenario robust training, small-defect-aware learning, score calibration and inference threshold selection, and metric-aligned checkpoint selection. These four modules respectively introduce cross-scenario robust normalization and DSU-style statistic perturbation with progressive random convolution, NWD and parameter-free spatial attention, the VFL learning objective, and a custom SWAD variant driven by the Track 1-oriented proxy. Experimental results demonstrate that our model achieves a Track 1 score of 0.564 with a screening score of 0.696 on the test dataset on a single RTX 4090 GPU. The code is available at https://github.com/GZHU-DVL/Dikai.

0.40

Online Supervised Dimension Reduction with Random Features: Diagnostics and Computational Trade-offs

cs.LG stat.MLstat.CO ·2026-09-17 ·Zhenlin Yao, Wei Xiong
similarity 0.59
Abstract · 41 pages, 4 figures, 18 tables; includes core supplementary material

Accurate optimization of a supervised spectral objective need not produce an accurate population subspace or a better predictive representation. We investigate these distinctions for Online Kernel Supervised Principal Component Analysis (OKSPCA), which combines a centered cross-moment in finite random-feature coordinates with an Adam-style orthonormal basis update for an established objective. Fixed-map consistency, concentration and perturbation results describe the estimator and its exact subspace; same-target comparisons then assess the practical iterate separately. Across six predictive benchmarks, performance depends on the declared pipeline: replacing the tracker with the exact empirical target leaves the two regression deficits largely unchanged. Direct classification-rank models capture nearly all terminal objective energy on average, but a saved intermediate state exhibits substantial geometric deviation; a controlled sample-size study further separates empirical accuracy from population recovery. In distinct numerical-service workloads, exact on-request computation is faster in the tested classification settings, whereas Adam saves time relative to the tested full thin-SVD service for some dense wider-regression requests, alongside persistent geometric error. These diagnostics limit explanations based solely on terminal optimization accuracy and distinguish numerical cost from quality, rank coverage and freshness; they establish neither practical-tracker convergence nor predictive or deployment benefits from basis availability.

0.27

CHILD: Human-in-the-Loop OOD Detection for Safe Clinical Deployment

cs.CV ·2026-09-07 ·Jinlun Ye, Kaiyue Lu, Runhe Lai, Xinhua Lu, Jia-Xin Zhuang +1 more
top venuecode/data similarity 0.58
Abstract · 11 pages. Accepted at MICCAI 2026

Out-of-distribution (OOD) detection is critical for safe deployment of medical AI systems. Recently, test-time adaptation (TTA) has emerged as a new paradigm for OOD detection, automatically adjusting detector behavior during deployment. However, such automatic adaptation mechanisms may raise safety concerns in safety-critical clinical environments. While physician oversight can mitigate these risks, it is resource-intensive and must be judiciously allocated. To reconcile safety with efficiency, we propose CHILD, a training-free framework designed to enhance streaming OOD detection via sparse human feedback. Operating under strict budget constraints, CHILD employs an adaptive risk-aware sample selection mechanism to pinpoint only the most decision-uncertain samples for review. Crucially, it maximizes the utility of this sparse feedback through a retrieval-based score calibration module, which refines model predictions using a compact feature cache without any parameter updates. Extensive experiments on four medical benchmarks demonstrate that CHILD turns limited supervision into significant reliability gains: with a sparse feedback budget of only 5%, it reduces the average FPR95 from 72.63% to 60.26% and improves AUROC from 75.53% to 81.85%, consistently outperforming state-of-the-art baselines. Our code is publicly available at https://github.com/figec/CHILD.

0.66

Calibrated Functional Prediction from Overlapping Nonlinear Observations

cs.LG stat.ML ·2026-11-16 ·Peida Li
benchmark similarity 0.58
Abstract · 34 pages, 8 figures

We study prediction of nonlinear audit functionals from scalar observations of a shared hidden table state. For Schur-rational operators, local Gaussian projections yield a common sparse kernel for overlapping tasks, repeated row identities, and cross-row dependence. Shared-state residual samples extend this kernel through a positive-semidefinite feature representation. Positive matrix integration and sample-value enclosures connect the original inputs to the actual saved hybrid prediction. A sorted calibration-role perturbation bound then certifies its complete finite-pool pinball risk. Independent complete-loss selection chooses the ridge, score scale, and residual weight, while task-rank calibration supplies an upper endpoint. Evaluation uses twenty new reference contexts per dataset at two task sizes, giving 400 conditions across ten benchmark populations. The mean dataset complete-loss ratios are 0.9977 to the degree-two kernel, 0.9961 to matched complete-MC shrinkage, and 0.9911 to the retrospective minimum over 174 public configurations. The last ratio has a paired reference-bootstrap interval of [0.9898,0.9927]. On 40 profiles at coefficient tolerance $10^{-7}$, Gamma quadrature has a geometric mean cold-time ratio of .0646 to strengthened Arb/Petras integration.

0.30