CS Paper Radar

Calibrated Uncertainty Quantification on TDC ADMET: What Can Be Resolved Across 22 Datasets?

Laurits Fredsgaard, Michael Riis Andersen, Mikkel N. Schmidt

cs.LG ·submitted 2026-11-06 12:14 UTC · v1
benchmark

Abstract

Absorption, distribution, metabolism, excretion, and toxicity (ADMET) models often rely on small, heterogeneous datasets, yet their predictions may determine which compounds proceed to experimental testing. Useful evaluations must therefore assess both predictive performance and the quality of reported probabilities or intervals. We ask which common uncertainty-quantification methods perform best on the 22 Therapeutics Data Commons (TDC) ADMET datasets and what this benchmark can resolve. We compare eleven calibrated pipelines using nested five-fold cross-validation across ten chemical partitions and five model seeds. Ensemble Monte Carlo (MC) dropout has the best average rank for TDC performance, Brier score, and 90% interval score. For TDC performance, however, it is not separated from the other ensemble pipelines; for Brier and interval score, LightGBM is also not ruled out as best. LightGBM wins more individual datasets than any other method for TDC performance and Brier score, but performs poorly on several endpoints. Smaller evaluations can identify a different leader and often fail to recover the broader comparison. These results make LightGBM and an ensemble neural model complementary baselines. Comparative claims should be checked across multiple chemical partitions using statistical analyses matched to the question being asked.

Comment
33 pages. Preprint of a paper accepted for poster presentation at the AIDaR Workshop at NeurIPS 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Predicting Dose-Response Success from Primary Screening Data

cs.LG q-bio.BM ·2026-08-27 ·Sean Lim
similarity 0.62
Abstract · Format changed. Some information added

High-throughput drug screening uses inexpensive primary screens to select compounds for more costly follow-up dose-response experiments, which estimate compound potency (pXC50) by determining the concentration required to produce a specified biological effect. However, many promoted compounds fail to yield a reportable potency, even when activity is subsequently confirmed. We investigated whether inexpensive three-point primary-screening data could predict whether a more costly 11-point dose-response experiment would yield a reportable pXC50. Using a dataset of 32,971 compound-assay pairs with three-point primary-screening and 11-point dose-response measurements, we trained random-forest models to predict successful potency quantification. Our model achieved an AUROC of 0.939 +/- 0.002 and generalized to previously unseen chemical scaffolds and assay-mechanism families. We found that primary-screen response features were more predictive of dose-response quantifiability than molecular structure-based features. Successful potency quantification was also strongly associated with primary-screen response amplitude. Across eight chronological test releases, the model retrospectively retained 90% of reportable potencies while selecting only 55.7% of historically promoted compound-assay pairs for follow-up profiling. These findings suggest that predicting potency quantifiability from primary-screening data could improve the efficiency of compound prioritization and reduce unnecessary dose-response experiments.

0.17

D2ML: Models for Direct-to-Biology Small Molecule Screening

cs.LG q-bio.BM ·2026-10-29 ·AkshatKumar Nigam, Neri Amara, Hillary Dequina, Dyana Kenanova, Seth Malmersjo +5 more
published similarity 0.60
Abstract · NeurIPS 2026 Workshop on AI for Drug Discovery (AI4DD)

Direct-to-biology (D2B) screening enables rapid testing of newly synthesized molecules without purification. However, uncertain concentrations in crude mixtures and limited purified-compound measurements complicate the use of these data in machine learning. Here we show that models tailored to the structure of D2B data can improve prediction and guide the discovery of more potent and selective compounds. Our assay-aware models predict activity from single-point screens, relative potency across targets from crude dose-response assays, and absolute potency from purified-compound measurements. For crude assays, potency ratios between targets tested from the same well can retain selectivity information despite uncertain concentrations. A joint model that shares information across stages and balances their contributions to training improved predictions over separate neural networks. The largest gain was in crude dose-response selectivity prediction, where rank correlation increased from 0.27 to 0.40. In a separate experiment, independently trained models guided the selection of 84 new compounds to improve ROS1 potency while limiting activity against the anti-target TRKB. Synthesis and testing identified compounds that extended the measured Pareto front, achieving better combinations of potency and selectivity than those in the training data. The most selective new compound achieved a ROS1 IC50 of 14.7 nM and 233-fold selectivity over TRKB. By comparison, the most selective training compound had a ROS1 IC50 of 20.3 nM and 79-fold selectivity. ATLAS provides the experimental setting for this approach, supporting synthesis and screening of 1,000-10,000 model-selected compounds in two- to three-week cycles. Together, these results show how assay-aware learning can use the full D2B assay sequence to guide multi-objective molecular optimization.

0.16

Do LLMs Know When They Are Wrong About Molecules? In-Context Property Prediction with Self-Reported Uncertainty

cs.LG ·2026-10-30 ·Matthias Busch, Marius Tacke, Tamadur Albaraghtheh, Sviatlana V. Lamaka, Mikhail L. Zheludkevich +3 more
similarity 0.60
Abstract

Large language models (LLMs) are used in many domains, but molecular property prediction with them is still little explored. In this work we evaluate the capability of 7 frontier LLMs to predict molecular properties together with a self-reported uncertainty from in-context examples, and compare them with classical baselines trained on the same molecules. We use 4 datasets and screen each for contamination of the training data of the LLMs: three are clean, and the fourth, ESOL, is memorised by most of the models and serves as a comparison. On the clean datasets every LLM improves with the context, at 30 examples the most accurate one beats every baseline. The requested reasoning effort changes accuracy little. The self-reported uncertainty ranks the prediction errors weakly, with Spearman rank correlations of up to 0.46, about the level reached by the uncertainties of the baselines. Combining uncertainty and prediction, on lipophilicity the most certain half of the best LLM is the most accurate at small context sizes; on BACE and antiviral, and on lipophilicity at large context sizes, the best LLMs are on par with the best baseline. On the contaminated ESOL dataset the LLMs show no improvement with the context size, as expected for a memorised dataset, and their stated uncertainty is smaller on the molecules whose value they recall.

0.25

RangeFinder: Predicting Ligand Property Ranges from Protein Pockets

cs.LG q-bio.BM ·2026-11-02 ·Alan Nadelsticher Ruvalcaba, Chris Alvin, Supratik Mukhopadhyay, Adam Bess
publishedcode/data similarity 0.59
Abstract · Accepted at the AI for Drug Discovery workshop --- NeurIPS 2026

Identifying ligands for a protein target requires searching a vast chemical space for a comparatively small set of compatible molecules. Existing machine learning approaches primarily address this problem by generating candidate ligands or scoring individual protein-ligand pairs, but provide limited information about the broader chemical space compatible with a binding pocket. We introduce RangeFinder, an SE(3)-equivariant transformer pocket-level model, trained and evaluated on BioLiP2 data, that predicts ranges over nine physicochemical ligand properties directly from pocket structure, chemistry, and evolutionary context. Using an intersection-over-union objective and evaluation metric (range-IoU), RangeFinder achieves a range-IoU of $0.464 \pm 0.002$, outperforming all deployable baselines while requiring only $12.2$ ms per pocket. We also quantify our method's ability to reduce the ligand search space while retaining compatible binders; in this regard, RangeFinder achieves a Reduction-Recall F-Score of $0.543 \pm 0.011$ --- once again outperforming all deployable baselines. Ablation experiments indicate that RangeFinder's performance depends more strongly on the density and quality of ligand supervision than on any individual input modality or specialized training strategy. Together, these results position RangeFinder as an inexpensive and effective molecular filtering prior, with demonstrated potential in narrowing the ligand search space of target pockets before docking, generation, or synthesis. The code for this project can be found at: https://github.com/alannadels/RangeFinder.

0.45

Towards Out-of-Distribution Uncertainty Quantification for Machine-Learned Interatomic Potentials via Pre-Training

cs.LG physics.chem-ph ·2026-10-30 ·Stefan P. Schmid, Tobias Setzer, Maximilian Scheurer
hot direction similarity 0.59
Abstract

Informative uncertainty quantification (UQ) is essential for the practical deployment of machine-learned interatomic potentials (MLIPs), as it provides insight into their reliability across chemical space. Although numerous post-hoc uncertainty estimation methods for MLIPs have recently been proposed, most have been trained in low-data regimes, restricting their applicability to narrow domains of interest. Inspired by the emergence of foundational MLIPs, we propose pre-training uncertainty estimators on large, diverse datasets to enable broader applicability. We pre-train several UQ models on large materials science datasets to quantify the uncertainty of different MLIPs, and evaluate their performance on both in-distribution and out-of-domain (OOD) adsorption-energy predictions. We find that pre-trained uncertainty estimators provide viable zero-shot uncertainty estimates across multiple datasets, while fine-tuning consistently yields positive correlations between predicted uncertainty and prediction error, even on the most challenging datasets. These results lay the groundwork for foundational UQ models for MLIPs, ultimately facilitating their deployment in practical settings by providing users with a priori information about model reliability and applicability.

0.26

TrialDesignBench: Benchmarking Language Models for Phase-I Trial Design Against Clinical Knowledge

cs.AI cs.CLstat.AP ·2026-11-09 ·Yihan Hank Tang, Peng Yang, Jiayi Xin, Tianqi Shang, Xuyang Chen +3 more
publishedbenchmarkcode/data similarity 0.58
Abstract · 13 pages, 3 figures, 6 tables. Presented as a poster at the NeurIPS 2026 Workshop on World Models for High-Stakes Health (WMHS). Code and da…

AI is used in much of the clinical trial pipeline. However, selecting the statistical design of a trial, a crucial step that determines how drug doses are tested on humans and evaluated, has seen almost no AI progress. LLMs appear to be a natural fit given their strengths in biomedical knowledge and statistical reasoning. Nevertheless, there is currently no appropriate way to evaluate such abilities because recent regulatory and methodological shifts, most notably FDA's Project Optimus, have redefined what constitutes a reasonable design and rendered many previous designs obsolete. To lay the groundwork for AI-assisted trial design, we introduce TrialDesignBench, a benchmark of 617 Phase-I/I-II oncology dose-finding trials, each paired with an expert-validated reference design that is consistent with the latest FDA regulatory initiatives. This reference standard is produced through a formalization of the expert design process. We evaluate five frontier LLMs, none of which matches the reference standard. Our ablation experiments trace this performance gap to two main causes, inference of latent, unrecorded design determinants from trial sources (supplying them raises class-wise balanced accuracy by 0.22) and structured decision-tree reasoning (+0.24). TrialDesignBench offers a reproducible, contemporary-standard objective for evaluating and enhancing AI-assisted trial design.

0.45

Conformal Prediction for Molecular Properties under Label Shift

cs.LG ·2026-08-18 ·Hyeonsu Lee, Juyeon Kim, Erkhembayar Jadamba, Seungjin Choi, Hyunjin Shin
published similarity 0.58
Abstract · NeurIPS 2025 Workshop on Reliable ML from Unreliable Data

Drug discovery and development underpins healthcare but remains costly and failure-prone. A critical bottleneck lies in predicting molecular properties such as solubility, potency, and toxicity, which directly determine whether a candidate can advance from preclinical to clinical trials. Artificial Intelligence (AI) has accelerated this process, yet its reliability is often undermined by distribution shift, as experimental conditions frequently diverge from training data. In addition, conventional point predictions provide only single-value estimates, offering limited guidance for high-stakes experimental design. We address these challenges with a conformal prediction framework tailored to label shift. By weighting conformal scores using marginal label probability ratios, our method produces statistically rigorous prediction intervals without retraining. This enables robust uncertainty quantification even when property distributions drift, directly tackling one of the most pervasive obstacles to applying AI in real-world drug development. By moving beyond accuracy alone to provide actionable confidence measures, our approach enhances the trustworthiness of AI-driven predictions. This further aligns predictive modeling with regulatory demands for transparency and uncertainty reporting and ultimately supports more reliable decision-making in billion-dollar development pipelines.

0.43

Uncertainty Identifies Difficult Samples Across Methods: A Multi-Task Study on a Heterogeneous Skin Lesion Dataset

cs.CV cs.LG ·2026-08-14 ·Leon Koole, Jiapan Guo, Matias Valdenegro-Toro
published similarity 0.58
Abstract · 12 pages, 9 figures, UNSURE 2026 @ MICCAI camera ready

Skin lesion classifiers can be confidently wrong on the cases that matter most, so knowing when a prediction should not be trusted is clinically as useful as the prediction. We study uncertainty quantification on a dataset pooled from many ISIC sources, with a shared backbone and two jointly learned heads: a binary malignant versus non-malignant head and a five-class diagnostic head. Five UQ methods (MC Dropout, DropConnect, Flipout, Deep Ensembles, DUQ) are compared on accuracy, calibration, uncertainty decomposition, and risk-coverage. Difficulty is largely method-agnostic: even methods with narrow entropy distributions rank the same samples as hard (per-sample entropy correlations of $0.54$ to $0.91$). The choice of method matters more for calibration and uncertainty decomposition, where Deep Ensembles is the clear winner, than for finding difficult cases. The ranking is also good enough that deferring the most uncertain cases removes a disproportionate share of errors, supporting uncertainty-based selective referral, evaluated here in-distribution only.

0.33