CS Paper Radar

When Prediction Error Is Not Enough: Evaluating Nuisance-Function Prediction for Causal Estimation

Cong Cao

cs.AI cs.LGstat.ME ·submitted 2026-08-30 20:01 UTC · v3

Abstract

Prediction error is widely used to evaluate nuisance-function estimators in causal inference, but its relationship with the performance of the resulting causal estimator is not well understood. We studied this question in a partially linear model using Monte Carlo simulations, comparing ordinary least squares (OLS), generalized additive models (GAMs), XGBoost, and Double Machine Learning with XGBoost (DML-XGBoost) with independent and with clustered observations. We evaluated nuisance-function prediction error, bias, RMSE, and 95% confidence interval coverage, and derived the second-order remainder through which errors in the exposure and outcome nuisance functions jointly determine the bias of the residualized estimator. When all methods estimated the causal effect from the same training sample, DML-XGBoost had the lowest RMSE and the highest coverage among the non-oracle methods, although XGBoost fitted to the full training sample had lower out-of-sample prediction error for both nuisance functions. GAMs had the lowest prediction error of all methods but were biased, with coverage of 0.55 with independent observations. The remainder reproduced the average bias of OLS, GAMs, and DML-XGBoost, but only half to three-quarters of the bias of XGBoost, whose residuals were formed on the data used to fit it. A joint-error measure based on the absolute cross-product of the two nuisance errors was only weakly associated with causal bias. Prediction error is useful for assessing nuisance-function estimation, but it should not be treated as a direct measure of the quality of the resulting causal estimator.

Comment
10 pages, 2 figures, 1 table

Closest papers

Nearest neighbours by meaning across the whole index.

Does Predictive Performance Translate to Causal Performance? Evaluating tabular foundation models as nuisance learners for causal effect estimation

cs.AI stat.APcs.LGstat.ME+1 ·2026-11-08 ·Andrew R Weckstein, Issa J Dahabreh, Marinka Zitnik, Sonia Hernández-Díaz, Sebastian Schneeweiss
similarity 0.66
Abstract

Tabular foundation models based on prior-data fitted networks (PFNs) achieve strong predictive performance, but their value for causal inference is unclear. We evaluated whether PFNs, which predict through in-context learning, could match or outperform conventional nuisance learners in supporting accurate causal estimation and reliable uncertainty quantification (inference). Learners modeled propensity scores and outcome regressions within doubly robust estimators of average causal effects, with and without cross-fitting, across synthetic and semi-synthetic settings with varying confounding complexity and effect heterogeneity. Under simple parametric data-generating processes, the PFNs were competitive with correctly specified parametric regressions and ensembles containing them. In more complex settings, they outperformed all conventional learners evaluated, including richly specified Super Learner ensembles, for both accuracy and uncertainty quantification. As sample size increased, only the two PFN models (TabPFN and Mitra) consistently approached nominal confidence interval coverage, whereas under-coverage persisted for conventional learners. In some settings, PFNs achieved near-nominal coverage and well-calibrated standard errors even without cross-fitting. At the smallest sample size (n=200) and in the higher-dimensional semi-synthetic settings, PFNs and other flexible learners exhibited poor coverage without cross-fitting. For PFNs, this appeared to be driven by propensity score overfitting consistent with in-sample memorization of treatment labels. These findings highlight the promise of prior-data fitted networks as in-context nuisance learners for causal estimation, while underscoring the need for further evaluation in challenging settings.

0.17

Residual-Based Causal Discovery via Pretrained In-Context Quantile Regression

cs.LG stat.ML ·2026-10-21 ·Daniel Klippert, Quang Nhat Huynh, Alexander Marx
similarity 0.62
Abstract

Residual-based causal discovery assumes that each variable is generated from its parents and an independent noise source. It estimates this noise and tests its independence from candidate causes. For identifiable functions such as additive-noise models (ANMs) and the broader class of location-scale noise models (LSNMs), this principle identifies the causal direction, as only the causal model admits this independence. Existing LSNM methods rely on parametric assumptions to obtain standardized residuals, risking model misspecification. In contrast, the RESIT-framework allows for nonparametric estimation of residuals to iteratively identify sink nodes, but was proposed for ANMs. To close this gap, we propose Loscar, an extension of RESIT to LSNMs that estimates the conditional mean and scale via quantile regression, enabling pretrained tabular foundation models as plug-in estimators. We compare it with a feasible generalized least squares (FGLS) variant, which is consistent given suitable basis functions. Across bivariate and multivariate causal-discovery benchmarks, in-context quantile regression substantially outperforms FGLS with flexible function classes, particularly for multivariate causes.

0.04

MediPFN: Amortized In-Context Nuisance Learning and Inference for Causal Mediation Analysis

cs.LG stat.MLstat.ME ·2026-11-09 ·Shi Bo, Jiaming Hu
similarity 0.62
Abstract · 30 pages, 7 figures

Causal mediation analysis decomposes a treatment effect into indirect pathways through mediators and direct pathways through other processes. Tabular foundation models perform classification and regression through in-context learning, providing estimates of conditional means and propensity scores used in causal mediation analysis. Constructing confidence intervals from these estimates requires accounting for uncertainty introduced by fitting and combining several nuisance functions. We introduce MediPFN, an amortized inference framework that adapts a pretrained tabular foundation model to nuisance prediction and combines its predictions through a cross-fitted influence-function estimator. A separate head learns quantiles of the estimator's standardized error from simulated tables with known indirect effects. Supervised fine-tuning, symmetrization, and independent calibration then produce a confidence interval. We establish coverage bounds for the resulting intervals under standard assumptions. Across experiments, MediPFN attains coverage closer to the nominal level than several comparators. It also reduces the computational cost of uncertainty quantification, running 59.1 times faster than bootstrap in matched comparisons.

0.18

Unit-level aggregation breaks coverage in causal foundation models

cs.LG stat.ML ·2026-10-16 ·Gabe Schulman, Lucius E. J. Bynum, Yoav Wald
code/data similarity 0.60
Abstract · 23 pages, 6 figures, 8 tables. Code: https://github.com/gabeschulman-nyu/subset-cate-coverage

Causal foundation models (CFMs) show promising results for treatment-effect estimation, with unit-level estimates that are competitive on standard benchmarks. Many causal analyses, however, inform decisions through effects averaged over groups of units, such as the full sample, the treated units, or a subgroup, and for such decisions a CFM is useful only if its credible intervals are calibrated. We examine these intervals and find that their coverage degrades, both in simple synthetic settings and on standard benchmarks. On ACIC 2016, for example, temperature-calibrated 95% intervals from CausalPFN fall below nominal coverage beyond groups of 30 out of 4802 units, and intervals for the full-sample average contain the true value in only 20% of datasets. We trace this problem to how these models are trained and how their intervals are formed. They output a predictive distribution for one unit at a time and, at inference time, combine the unit-level variances as if the errors on different units were independent. We show formally that posterior uncertainty over the data-generating process makes these errors correlated. As a result, the combined variance is too small by a factor that grows with the size of the group. Empirically, this factor predicts where coverage fails and by how much. We then show that two neural process models, which make joint predictions for a group of units, account for this correlation and substantially improve coverage on synthetic tasks and, as heads on a frozen CausalPFN, on all six benchmarks.

0.30

Proximal Causal Learning under Unmeasured Confounding

cs.LG ·2026-10-03 ·Ying Tang, Yi Wang
similarity 0.60
Abstract · 14 pages,4 figures

Estimating treatment effects from observational data typically relies on the No Unmeasured Confounding Assumption (NUCA), which rarely holds in practice. Proximal causal learning (PCL) addresses unmeasured confounding via proxy variables, yet existing methods require the proxy variables to be pre-specified. Thus, we propose PCL-U, a framework that learns proxy variables directly from observed covariates. PCL-U uses neural encoders to decompose covariates into treatment-inducing, outcome-inducing, and shared proxies, guided by minimax mutual information objectives, and obtains causal estimates through a practical moment-based risk function. Experiments on benchmarks show that PCL-U matches or outperforms existing baselines. Besides, there are two types of synthetic datasets with varying dimensions and confounding strengths that illustrate that our method maintains stable estimation accuracy.

0.14

What You Observe Determines How You Identify Causal Effects: Evaluating Causal Models across Observational Views

cs.LG ·2026-09-29 ·Heejin Jung, Gyeongdeok Seo, Hoyoon Byun, Joseph Lee, Kyungwoo Song
benchmark similarity 0.60
Abstract · 9 pages, 8 figures

Causal foundation models (CFMs) pre-trained on data generated from various structural causal models (SCMs) have been proposed for estimating causal effects from observational data. However, differences in pre-training environments and evaluation protocols make it difficult to assess how their performance depends on the information available for causal identification. To enable controlled comparisons, we introduce CausalIDView, a multi-view benchmark that holds fixed SCM realization and target estimand while varying only the observational view available to the estimator. Each observational view corresponds to a distinct identification regime under the benchmark's maintained causal assumptions. Across these matched views, no CFM consistently performs best and model rankings vary substantially. Under controlled structural changes, CFMs exhibit model-specific failures to maintain stable estimates when true effects are unchanged and to track genuine effect changes. We also examine whether combining explicit identification with strong predictive estimation is effective. A modular approach that pairs a predictive tabular foundation model with regime-specific identification procedures is competitive with CFMs and outperforms several of them. These findings motivate cross-regime comparisons to assess the empirical value of CFMs.

0.17

Efficient and Accurate Causal Validation of Training Data Attribution

cs.LG stat.MEstat.ML ·2026-11-14 ·Simon Vary, Alireza Mousavi-Hosseini, Roger Grosse
similarity 0.58
Abstract · 20 pages, 9 figures

Causal validation of training data attribution (TDA) methods is costly, since evaluation involves retraining a model on many different subsets of the original training set. For example, the Linear Datamodeling Score (LDS), which has become a standard metric for evaluating TDA methods, measures the correlation between TDA predictions and expected retrained model outputs. In this paper, we reduce the amount of computation needed to estimate the LDS along three axes: the number of sampled training subsets, the number of retraining seeds per subset, and the number of models in the attribution ensemble. Each of these factors induces a form of bias in the LDS estimation, and removing these sources of bias significantly reduces the number of retrainings needed to match the same accuracy in LDS estimation. On ResNet-9/CIFAR-10, finite-subset debiasing achieves $3.2\%$ relative RMSE with as few as $4$ subsets, and $32$ subsets match the accuracy of ordinary Spearman using $96$ subsets, measured against a 300-subset reference. On ResNet-9/CIFAR-10, BERT/QNLI and GPT-2/WikiText-2, estimating the training-noise variance allows us to reproduce the standard five-seed LDS at the same accuracy with $2$--$5\times$ fewer retrainings.

0.29

Data-informed Adaptation of Foundation Models for Causal Effect Estimation

cs.LG cs.AI ·2026-11-08 ·Chunwei Ma, Russell D Wolfinger
benchmark similarity 0.57
Abstract · Preprint

Due to the fundamental problem of causal inference -- the impossibility of observing counterfactual outcomes -- there is always a gap between the training goal (outcome, propensity) and the inference target (CATE, ATE). Traditional causal effect estimation revolves around setting assumptions, selecting identification strategies, and choosing causal estimators, while more recently, Causal Foundation Models (CFMs), built on Prior-Data-Fitted Networks (PFNs), ease this process by pretraining on massive synthetic prior datasets. Despite their success, such amortization makes causal inference less transparent, obscuring which assumptions the model implicitly relies on for a given dataset. The high diversity of real-world data-generating processes (DGPs) and Structural Causal Models (SCMs) makes it hard to know, a priori, which causal inference strategy is compatible with a given dataset. In this work, we focus on this compatibility between an observational dataset and method-specific assumptions, and find that geometric characteristics of data embeddings -- specifically, the variogram -- provide a useful signal of whether a given causal estimation strategy is appropriate. We develop MoE-Learner, a mixture-of-causal-experts framework that embeds a dataset under multiple assumption-specific views and uses the variogram as a counterfactual-free router to assess each expert's validity. Rather than eliminating causal assumptions, this shifts their validation from a fixed, a priori choice to a criterion that can be checked directly from data geometry. MoE-Learner achieves the best average rank on benchmark datasets (IHDP, ACIC, Lalonde). Being orthogonal to (causal) foundation models, our method builds a bridge from data characteristics to causal estimation strategy rather than proposing another CFM.

0.06