CS Paper Radar

Correctness and Predictive Uncertainty in Bayesian TSK Fuzzy Regression: A Reproducible Evaluation on Tabular and Building-Meter Data

Zhiren Xiao

cs.LG stat.ML ·submitted 2026-11-01 02:35 UTC · v1
unusual

Abstract

Uncertainty estimates from fuzzy regression are informative only when the implementation is correct and the comparison protocol makes the sources of variation visible. We revisit a first-order Takagi--Sugeno--Kang (TSK) regression implementation and rebuild its dense, conjugate Bayesian, BIC-based, and rule-level spike-and-slab estimators using corrected code. The corrected pipeline freezes the Gaussian membership partition at training time, uses a scaled conjugate prior, evaluates firing strengths in the log domain, and reports direct posterior-predictive quantiles for Bayesian methods. Across 30 outer 80/20 splits on three targets from two UCI regression datasets, corrected TSK-LS obtains $R^2=0.957$, $0.923$, and $0.786$ on Energy-Heating, Energy-Cooling, and Concrete, respectively; rule-level Gibbs sampling with Bayesian model averaging obtains $0.957$, $0.922$, and $0.784$. We then evaluate the same workflow on a bounded chronological sample of the Building Data Genome 2 meter data that covers 158 buildings across 17 sites. In a common-unit evaluation, every method is scored on the same 47,400 test rows and every trained model uses 94,800 train--validation rows. Persistence has RMSE $27.710$, compared with $29.083$ for both TSK-LS and Bayesian TSK; the pre-specified building-level mean RMSE difference between Bayesian TSK and persistence is $+0.0447$ with a 95% site-cluster bootstrap interval of $[-1.5536,1.5665]$. The Bayesian TSK 90% intervals have PICP $0.980$ and MPIW $116.469$. Taken together, the evidence supports a narrower conclusion: spike-and-slab model averaging is a reproducible implementation option, but it does not provide a demonstrated accuracy or calibration advantage across the tested tabular and building-meter settings.

Comment
21 pages, 3 figures. Supplementary material is included as an ancillary PDF

Closest papers

Nearest neighbours by meaning across the whole index.

Benchmarking uncertainty quantification for engineering regression under graded distribution shift

cs.LG ·2026-10-26 ·Ahmad Lawal, Rahim Pasbanigoloojeh, Asmau Lawal Bilbis
similarity 0.54
Abstract

Machine learning models in engineering are often used beyond their training conditions, where reliable uncertainty quantification (UQ) matters most. Existing comparisons of regression UQ methods rely on random splits, which assess calibration only in distribution. We introduce a three-rung shift ladder (in distribution, soft shift and hard shift) built along physically meaningful shift variables, and evaluate 12 methods from five families on seven public engineering datasets: five extrapolation settings, an unseen turbofan fault mode and a winter air-quality sensor deployment. All tuning uses source data only. In distribution, conformal methods cover at least 0.946 of test points on every dataset. Under hard shift, mean coverage falls to 0.40-0.75 across the 11 methods that always return an interval, and no systematic difference in method ranks is detected (Friedman p = 0.63). Weighted conformalised quantile regression stays near nominal on the points it answers, but abstains on 85\% of hard-shift points on average and covers only 0.67 of the few Concrete points it answers. Source-validation gains predict improvement in distribution and under soft shift, but not under hard shift (Spearman's rho = 0.04). Engineering evaluations should split calibration data by deployment unit, report subgroup coverage and sharpness, and treat abstention as an explicit operational outcome.

0.22

Possibility-based Inference: A Deterministic Framework for Robust and Reliable Uncertainty Quantification in Decision-Making

cs.LG ·2026-10-29 ·Seyedfakhredin Musavishavazi, Maryam Alipour
similarity 0.53
Abstract

Decision-making under uncertainty remains a key issue in Machine Learning (ML), particularly in the deployment of safety-critical systems, where reliable quantification of system behavior is essential for evaluating behavioral impacts and strengthening confidence in ML models. This contribution, influenced by fuzzy set theory and grounded in Bayesian statistics, addresses challenges related to data precision and the performance-accuracy trade-off in Bayesian inference (BI). To this end, we introduce a possibility-based contrastive grade of prediction as an alternative quantification method to existing latent posterior-based approaches for parameter estimation (approximation) and model selection. Unlike conventional Bayesian inference, which represents uncertainty within the probabilistic framework, the proposed model adopts a possibility-based representation that allows uncertain and potentially conflicting information to be characterized more flexibly. Rather than focusing on isolated aspects of a problem, the resulting mathematical framework provides a systematic approach to data mining while mitigating overly confident, yet implausible predictions generated by highly specialized deep learning architectures.

0.22

Predictive Uncertainty for Neural CAE Surrogates

cs.LG cs.AImath.PRphysics.comp-ph ·2026-09-21 ·Kaustubh Tangsali, Mohammad Amin Nabian, Kelvin Lee, Carmelo Gonzales, Sanjay Choudhry
similarity 0.53
Abstract

Neural surrogates can substantially accelerate computer-aided engineering (CAE) workflows, but their use in design requires uncertainty estimates that remain meaningful across varying geometries, spatial prediction fields, and engineering quantities of interest. We investigate how established uncertainty quantification (UQ) approaches behave when adapted to geometry-conditioned neural surrogates. We compare one closed-form and two sampling-based approaches-a Gaussian process (GP)-based method, concrete Monte Carlo (MC) dropout, and deep ensembles-and evaluate them on three large, industry-relevant CAE datasets for external aerodynamics and crash dynamics. We examine whether predicted uncertainties have credible magnitudes, identify locations with larger prediction errors, respond to unfamiliar inputs, and remain informative for derived engineering quantities. On the DrivAerStar dataset, where all three methods are compared, each generally assigns higher uncertainty to locations with larger prediction errors, and validation-based rescaling brings interval coverage close to nominal on a disjoint in-distribution test set. Results on AirFRANS and automotive crash also show useful error ranking and interval estimates, but the relative performance of the methods changes with the dataset and evaluation criterion. UQ methods and evaluation metrics should therefore be selected based on the intended downstream CAE decision.

0.17

Uncertainty Quantification of Next Generation Reservoir Computing with Applications to Memory-Driven Dynamical Systems

cs.LG stat.ML ·2026-09-26 ·Livia Popa, Sumanta Basu, Martin T. Wells
similarity 0.52
Abstract

Nonlinear dynamical systems with memory arise across science and engineering, yet uncertainty quantification for efficient forecasting methods such as Next Generation Reservoir Computing (NGRC) remains underdeveloped. We study Bayesian ridge and conformal prediction intervals for NGRC and characterize when their uncertainty estimates agree or differ. In low dimensions, their asymptotic widths are governed by different summaries of the residual distribution, so agreement depends on residual shape rather than dimensionality alone. In high dimensions, regularization introduces a further tradeoff between estimation variance, shrinkage bias, and posterior uncertainty, leading to an explicit transition between regimes where Bayesian intervals are wider or narrower than conformal intervals. We extend these results to quadratic NGRC feature maps and give sufficient conditions for transferring the analysis to temporally dependent forecast windows. Simulations and real-data experiments support the theoretical predictions and illustrate how residual distribution, regularization, dimensionality, and distribution shift affect interval calibration and efficiency. These results provide a principled framework for choosing and interpreting uncertainty quantification methods in reservoir-based forecasting.

0.32

Calibrated Uncertainty Quantification on TDC ADMET: What Can Be Resolved Across 22 Datasets?

cs.LG ·2026-11-06 ·Laurits Fredsgaard, Michael Riis Andersen, Mikkel N. Schmidt
benchmark similarity 0.50
Abstract · 33 pages. Preprint of a paper accepted for poster presentation at the AIDaR Workshop at NeurIPS 2026

Absorption, distribution, metabolism, excretion, and toxicity (ADMET) models often rely on small, heterogeneous datasets, yet their predictions may determine which compounds proceed to experimental testing. Useful evaluations must therefore assess both predictive performance and the quality of reported probabilities or intervals. We ask which common uncertainty-quantification methods perform best on the 22 Therapeutics Data Commons (TDC) ADMET datasets and what this benchmark can resolve. We compare eleven calibrated pipelines using nested five-fold cross-validation across ten chemical partitions and five model seeds. Ensemble Monte Carlo (MC) dropout has the best average rank for TDC performance, Brier score, and 90% interval score. For TDC performance, however, it is not separated from the other ensemble pipelines; for Brier and interval score, LightGBM is also not ruled out as best. LightGBM wins more individual datasets than any other method for TDC performance and Brier score, but performs poorly on several endpoints. Smaller evaluations can identify a different leader and often fail to recover the broader comparison. These results make LightGBM and an ensemble neural model complementary baselines. Comparative claims should be checked across multiple chemical partitions using statistical analyses matched to the question being asked.

0.25

Information-Based Calibration of Uncertainty Quantification in Product-of-Experts Gaussian Process Models

cs.LG ·2026-08-29 ·Yean Hoon Ong, Paolo Barucca, Wei Pan, Jun Wang
top venue similarity 0.50
Abstract · Published in the Journal of Artificial Intelligence Research, Volume 86 (2026)

Gaussian process (GP) regression with a single global GP (GP-glo) incurs cubic computational cost, limiting scalability to large datasets. Product-of-experts GP models (GP-pro), which combine local GP models to capture global correlations, alleviate this computational burden. However, training local experts on disjoint data subsets can lead to overestimated posterior variances. We propose GP-pro-c, a product-of-experts GP model that calibrates these variances using an information-based method. The method exploits the monotonicity and submodularity of information gain in GPs to define a calibration ratio that reduces the posterior variance of individual local GP models. We evaluate GP-pro-c using negative log-likelihood (NLL), root mean squared error (RMSE), and expected normalised calibration error (ENCE). Experiments on four synthetic functions and six regression datasets show that GP-pro-c achieves average reductions of 2.3% in NLL and 12.0% in ENCE compared with the uncalibrated GP-pro model. The proposed method mitigates posterior variance overestimation while maintaining predictive accuracy and reducing computational complexity. GP-pro-c provides a promising approach for uncertainty estimation in scalable GP models and may serve as a useful surrogate model for Bayesian optimisation with high-dimensional and large-scale data.

0.63

Bayesian Optimisation Using Product-of-Experts Gaussian Process Models with Uncertainty Calibration

cs.LG ·2026-09-14 ·Yean Hoon Ong
similarity 0.50
Abstract

Bayesian optimisation (BO) typically relies on a single global Gaussian process (GP) model as its surrogate model. However, GP regression has cubic computational complexity in the number of training data points, limiting its applicability to large-scale optimisation problems. The product-of-experts Gaussian process model with uncertainty calibration (GP-pro-c) mitigates this limitation by combining multiple local GP experts, enabling improved uncertainty quantification, reduced computational cost, and preservation of global correlations. Despite these desirable properties, the use of GP-pro-c in BO has not been thoroughly studied. This paper introduces BO-pro-c, a Bayesian optimisation algorithm that uses GP-pro-c as its surrogate model, and evaluate its performance across a diverse range of BO settings. Experimental results suggest that BO-pro-c maintains competitive optimisation performance while achieving a 0.9% reduction in simple regret and a 39.4% reduction in computational overhead relative to a BO algorithm based on a single global GP model.

0.40

TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy

cs.LG ·2026-10-09 ·Tianqi Zhao, Jia Guo, Yan Shuo Tan, Qiong Zhang
benchmarkcode/data similarity 0.49
Abstract · 53 pages, including supplementary materials. Code: https://github.com/Tianqi-Zhao/TabReliBench

Tabular foundation models (TFMs) are increasingly popular for their strong predictive performance. Many also provide posterior predictive distributions (PPDs), whose reliability remains less studied. We assess reliability through point prediction, distributional quality, marginal and subgroup calibration, and interval efficiency. We then introduce TabReliBench, a large classification and regression benchmark for these comparisons and dataset-level analyses. We find that TFMs' rankings change between TabArena and our benchmark and across reliability metrics. Regression TFMs usually achieve near-nominal marginal coverage and retain narrow intervals, but this favorable marginal behavior does not imply valid conditional coverage. Dataset-level analyses show that classification difficulty often accompanies poor calibration, whereas regression predictive performance is not a reliable indicator of calibration. Ensembling improves predictive performance but does not consistently improve calibration of PPDs; TFMs also benefit little from conformal post-processing, which does not reliably resolve subgroup gaps. Reliable TFM selection therefore requires multidimensional evaluation beyond aggregate rankings. Code is available at https://github.com/Tianqi-Zhao/TabReliBench.

0.60