CS Paper Radar

Feature Selection for Tabular Foundation Models: Exploring Feature Utility in Low-Data Molecular Prediction

Yiqi Yao, Miquel Duran-Frigola

cs.LG ·submitted 2026-09-25 04:13 UTC · v3
code/data

Abstract

Choosing features is difficult with few labeled molecules, even for a tabular foundation model that learns in context. We study this problem with TabPFN for low-data molecular property prediction, separating input width, feature meaning, and predictive usefulness while keeping contexts, query molecules, and predictor settings matched. On ADME/Tox assays from Therapeutics Data Commons, endpoint-matched assay predictions improve a compact RDKit descriptor baseline, and a provenance audit shows that most audited query molecules already occur in the predictors' training sources. Across 77 FS-Mol development tasks and 157 held-out test assays drawn from ChEMBL, broad RDKit descriptor and fingerprint families provide a strong default, rank-averaging TabPFN with support-fitted fingerprint forests improves upon both models, and learned weighting of family predictions adds further gains at a larger prediction budget. A utility-trained selector builds a traceable 16-column input from 238 sourceassay predictors, improving upon the baseline while trailing wider representations and equalwidth structural SVD coordinates. Named outputs identify what each feature predicts and where it comes from. Their usefulness for a new assay is assessed alongside the other input features, and paired removal and replacement tests measure what each selected output contributes. Together, these studies offer a protocol for evaluating feature selection, representation choice, and data provenance with a fixed tabular foundation model for few-shot molecular property prediction.

Comment
34 pages, 13 figures, 21 tables, 11 appendices. v3: revised and expanded with held-out FS-Mol test evaluations. Code: https://github.com/ersilia-os/chemicl

Closest papers

Nearest neighbours by meaning across the whole index.

TabOmics: A Small, Open Tabular Foundation Model for Wide Omics Data

cs.LG ·2026-11-11 ·Andrea López-García, Luis Bote-Curiel, Francisco Javier Gimeno-Blanes, José Luis Rojo-Álvarez, Sergio Muñoz-Romero
code/data similarity 0.75
Abstract

Omics studies often produce tables with thousands to tens of thousands of molecular features and few samples. Tabular foundation models use diverse strategies to predict labels for such tables, with some relying on in-context learning, while approaches to wide tables may involve subsampling, grouping, or compressing features. We present TabOmics, a small, open family of tabular foundation models with 7.3M parameters, pretrained exclusively on synthetic tables. TabOmics processes all features retained after preprocessing jointly and derives predictions, attention-derived feature-importance scores, and test sample embeddings from the same forward pass. On 46 classification tasks from three benchmarks of high-dimensional, low-sample-size tables, TabOmics v2.5 with eight ensemble members achieves the highest mean area under the receiver operating characteristic curve (AUROC) among the evaluated models (0.943), outperforming TabPFN-3.5 with eight members on 32 tasks despite having about 30 times fewer parameters. With a single member, TabOmics v2.5 reaches a mean AUROC of 0.941, versus 0.942 for TabPFN-3.5 with eight members, with no significant paired difference under the reported task-level analysis. On synthetic tables, its feature-importance scores achieve feature recovery similar to that of TabPFN-Wide, and its predictions remain above chance at 30,000 features. On breast cancer tasks, the top ranked genes include PAM50 genes, which define the subtype labels, indicating concordance with the label definition rather than independent biological validation. The code, model weights, training recipes, and experimental artifacts are publicly available.

0.23

Molecular Property Prediction under Structural Shift with Tabular Foundation Models

cs.LG ·2026-09-30 ·Jinmo Lee, Dooho Lee, Minho Jeong, Jaemin Yoo
code/data similarity 0.72
Abstract

Predicting molecular properties for compounds that differ structurally from labeled training molecules is important for drug discovery and materials design. Tabular foundation models (TFMs) offer a promising approach through in-context learning, but their performance under structural shifts and the value of molecular comparisons in this setting remain underexplored. We study structural generalization in molecular property prediction and introduce MolPAIR (Molecular Pair-Augmented In-context Refinement), a framework that combines molecule-level and molecular-pair contexts without task-specific parameter updates. A global tabular foundation model (TFM) first predicts a query's property from labeled molecular examples. A second frozen TFM predicts differences in prediction errors between the query and labeled reference molecules, using these comparisons to refine the initial prediction. Across 58 MoleculeACE and Polaris tasks, CheMeleon representations combined with TabPFN-3 already outperform each evaluated baseline on a majority of tasks. MOLPAIR further improves this predictor on 46 of 58 tasks, with gains across four molecular representations and three TFM backbones. These results show that explicit molecular comparisons can strengthen tabular in-context learning for structural generalization while keeping the molecular encoder and pretrained model weights fixed. The code and datasets are available at https://github.com/nums-ai/MolPAIR.

0.24

Interpretable by Design Tabular Prediction via In-Context Local Linear Learning

cs.LG ·2026-10-29 ·Ran Eisenberg, Steven Berger, Ofir Lindenbaum
benchmark similarity 0.69
Abstract

In-context learning (ICL) lets tabular foundation models make predictions without dataset-specific training or fine-tuning. These models achieve strong predictive accuracy, but interpreting how individual features contribute to the prediction remains difficult. In biology, for example, feature contributions can help identify biomarkers or risk genes. Post-hoc interpretability methods assign feature importance scores, but these scores do not indicate what the model actually uses and can lead to explanation errors. We introduce TICL$^3$ (Tabular In-Context Local Linear Learning), an interpretable-by-design predictor that uses a tabular foundation model to guide a simple linear prediction rule. TICL$^3$ first learns a shared linear model to capture dataset-level feature effects. Then, for each query, it retrieves similar training samples using representations from the foundation model and fits a second linear model to the residuals in that neighborhood. The shared model captures broad effects, while the local model corrects errors among similar samples. Their combined coefficients determine each feature contribution to the final query-specific linear score, so the same model makes and explains the prediction. Across 386 classification benchmark-dataset evaluations, TICL$^3$ trails TabICLv2 by 1.1 percentage points and outperforms CatBoost in 266 evaluations. It improves average support F1 over MAPLE by 4.7 points and achieves the best performance on both real-data selected-feature utility metrics.

0.12

TabBioMed: A Large-Scale Benchmark for Biomedical Tabular Learning

cs.LG ·2026-11-18 ·Pau Mateo-Bernadó, Saivenkata Nagavyjayanthi Polapragada, David Bonet, Pol Arbiol Rakuljic, Laia Mogas Pladevall +6 more
publishedbenchmarkcode/data similarity 0.69
Abstract · 87 pages, 43 figures, including appendices and the NeurIPS paper checklist

Recent advances in machine learning have accelerated biomedical prediction, but evaluation on structured biomedical data remains fragmented, especially for datasets with high dimensionality, large sample size, missingness, and class imbalance. We introduce TabBioMed, a large-scale benchmark of 94 curated public biomedical tabular datasets spanning electronic health records, drug discovery, genomics, transcriptomics, proteomics, metabolomics, single-cell omics, and systems biology. TabBioMed unifies these datasets through a standardized, dataset-aware preprocessing and evaluation framework, enabling reproducible comparison across classification, regression, and multi-target tasks. We benchmark 23 models across classical baselines, gradient-boosted trees, neural tabular architectures, and tabular foundation models. Foundation models deliver the strongest family-level predictive performance. However, their gains are regime-dependent: tree-based models and tuned multilayer perceptrons remain highly competitive, often dominating under inference-time or training-cost constraints. Overall, TabBioMed shows that biomedical tabular learning is strongly context-dependent, with optimal model choice depending on data regime, task type, and computational budget. We release TabBioMed with curated datasets and splits, preprocessing code, baselines, and results, providing a reproducible benchmark for practical model selection and future methods development in biomedical machine learning, available at https://github.com/AI-sandbox/TabBioMed.

0.44

GeneICL: A Tabular Foundation Model for Bulk Transcriptomics

cs.LG ·2026-10-06 ·Michael Bohl, Alexander Theus, David Wissel, Valentina Boeva
similarity 0.66
Abstract

Gene expression is widely measured in biomedicine, yet clinical outcome prediction remains challenging due to high dimensionality, strong feature correlations, and limited labeled data. Large self-supervised transcriptomic foundation models often fail to outperform simple supervised baselines. Tabular foundation models offer an alternative through in-context learning, but are typically pretrained on generic synthetic data rather than transcriptomic structure. We ask whether transcriptomics-aware pretraining, rather than scale, is the missing ingredient. Towards this end, we introduce GeneICL, a 4.2M-parameter tabular foundation model combining a semi-synthetic pretraining prior built from measured bulk expression profiles with a parameter-efficient recurrent architecture. We further enable right-censored survival prediction via a training-free reduction to regression using Cox partial-likelihood residuals. We evaluate GeneICL on 80 clinical outcome-prediction tasks spanning classification, regression, and survival. Tabular foundation models consistently outperform self-supervised transcriptomic models, while GeneICL achieves the best overall rank among evaluated foundation models and tuned baselines. GeneICL does so with up to 387$\times$ fewer parameters, no gradient updates at inference, and predictions within seconds on a laptop CPU.

0.44

A Systematic Comparison of In-Context Learning Strategies for Peptide Property Prediction in Low-Data Regimes

cs.LG physics.chem-ph ·2026-11-03 ·Jiantao Zhao, Andrei Voinea, Mathijs F. J. Mabesoone
similarity 0.66
Abstract · 23 pages, 6 figures

Peptide sequence-property relationships are often highly non-linear, but can be learned with machine learning models and exploited for the design of new materials and bioactive compounds. Various methods based on transformer architectures have become available for molecular property predictions. These foundation models promise straightforward inference and prediction of properties of new materials through in-context learning. Benchmarks of individual model architectures exist, but a systematic comparison of various foundation models against classical baselines on similar datasets is lacking. Here, we provide a systematic evaluation of the predictive performance of various in-context learning strategies. We compare several large language models, tabular foundation models, and typed decision models against an XGBoost baseline. We evaluate predictive performance on binary and multiclass classification tasks on pH-dependent surface activ-ity, IR spectral signatures, self-assembly, and anti-cancer activities. Our results show that ICL is competitive, yet does not significantly outperform classical baselines, and performance varies between models and tasks. Our findings further suggest that language models use both in context examples and knowledge acquired during pretraining to infer properties of unseen samples. This finding indicates that pretrained models can leverage the existing literature for predictions, but may also bias their predictions towards established sequence-property relationships.

0.18

ScreenShot: A Foundation Model for Few-Shot Combination Drug Screening

cs.LG ·2026-08-12 ·Antoine de Mathelin, Christopher Tosh, Wesley Tansey
code/data similarity 0.66
Abstract

Treating patients with combinations of drugs reduces the risk of resistance to any individual drug. Finding effective combinations is difficult because the large search space makes combinatorial screens prohibitively expensive, time consuming, and often technically infeasible. Predictive models can fill this gap, yet existing methods typically require molecular profiling of each sample and per-cohort training, limiting their applicability when time and tissue are scarce. To address this challenge, we introduce ScreenShot, a hierarchical transformer pretrained on 40 drug screening datasets covering 3,700 drugs and 6,000 biological samples, whose architecture mirrors the nested structure of screening data. Given a few-shot context of observations from a new patient, ScreenShot predicts the response of the sample to combination therapies through in-context learning, operating directly on functional measurements with no fine-tuning and no molecular profiling. On four held-out datasets, ScreenShot outperforms all baselines in both prediction accuracy and identification of selectively effective treatments. ScreenShot's internal representations are directly useful for experimental design: we use them to drive a weighted k-means++ active learning strategy that selects which experiments to run, achieving the same hit detection as uniform screening with a third of the budget. Source code and interactive dashboard: https://github.com/tansey-lab/screenshot.

0.24

TopU-LBVS: A Realistic Multi Target Benchmark for Ligand Based Virtual Screening

cs.LG cs.AI ·2026-09-24 ·Surbhi Kumar, Yuhe Zhou, Varun Shiralkar, Niu Huang, Baris Coskunuzer
benchmarkcode/data similarity 0.66
Abstract · 75 pages

Ligand-based virtual screening (LBVS) is a practical first-pass tool in early-stage drug discovery, but existing benchmarks can overestimate performance through random negatives, easy decoys, limited target coverage, and non-standardized evaluation protocols. We introduce TopU-LBVS, a multi-target benchmark for LBVS under hard-negative screening conditions. Starting from curated ChEMBL~35 bioactivity data, TopU-LBVS covers 93 protein targets across 7 protein classes and constructs target-specific screening libraries with property-matched, structurally similar decoys at a fixed 1:40 active-to-decoy ratio. Libraries contain roughly 400 to 10,000 compounds and are designed to reduce simple physicochemical and nearest-neighbor fingerprint shortcuts. TopU-LBVS provides three fixed protocols. TopU-LBVS-full evaluates ChEMBL$^\ast \rightarrow$ TopU generalization across all 93 targets. TopU-LBVS-low evaluates low-data TopU $\rightarrow$ TopU learning within the hard-negative distribution. TopU-LBVS-mini provides a compact seven-target protocol with a paired random-decoy control that changes only the test decoys, enabling low-cost development and direct measurement of the gap between random ChEMBL$^\ast$ and TopU decoys. Across ten reference baselines spanning fingerprint methods, molecular GNNs, fingerprint hybrids, and modern molecular models, performance under random-decoy evaluation degrades sharply under hard-negative screening. We release data, fixed splits, evaluation code, and baseline implementations for reproducible comparison of future LBVS and molecular representation learning methods. Code and data are available at https://github.com/topu-benchmark/topu-lbvs and https://huggingface.co/datasets/topu-benchmark/topu-lbvs.

0.38