CS Paper Radar

When Does Combining Software-Effort Estimates Help? Evidence from Four Public Project Datasets

Azzeddine Ihsine, Sara Ihsine

cs.SE ·submitted 2026-11-03 12:03 UTC · v1

Abstract

Combining estimates can reduce individual prediction errors, but an ensemble may also dilute an unusually accurate estimator or repeat errors shared by its members. We study these competing effects on four archival software-effort datasets (Kitchenham: 145 projects, Desharnais: 77, China: 499, Maxwell: 63) with 100 repeated 70/30 training and test partitions per dataset. Three conventional models (parametric regression, analogy, and random forest) are combined with each other and, in Kitchenham, with a recorded human estimate. Out-of-fold calibration errors drive five combination rules: equal weighting, inverse-MSE weighting, a nonnegative simplex combination, signed Bates-Granger weighting, and Bates-Granger with covariance shrinkage. All preprocessing is learned within training folds. Model-error correlations range from 0.54 to 0.97. In Kitchenham, the human estimate dominates: equal weighting raises mean test log-MAE from 0.239 to 0.455, a loss that remains significant under a corrected resampled t-test with Holm adjustment, whereas the simplex rule limits the increase to 0.250. In Desharnais, China, and Maxwell, every rule lowers mean log-MAE (by 1.3 to 8.6%) and BG with shrinkage beats the calibration-selected individual on 75, 83, and 83 of 100 splits, but none of these gains is statistically distinguishable from zero once the overlap between repeated splits is accounted for. On a completion-year split of Desharnais, equal weighting becomes slightly worse and inverse-MSE is near the individual baseline, while simplex and shrinkage retain mean gains of about 5%. Combining correlated effort estimates is therefore a conditional decision: the harm of naive pooling next to a dominant source is clear, while the benefit elsewhere is small and uncertain.

Comment
14 pages, 2 figures, 9 tables. Public aggregate results available on Zenodo: doi:10.5281/zenodo.23913209

Closest papers

Nearest neighbours by meaning across the whole index.

Governed Human-AI Prioritization Under Uncertainty: Adaptive Estimation and Dependency-Constrained Portfolio Selection

cs.SE ·2026-09-09 ·Azzeddine Ihsine, Sara Ihsine
similarity 0.57
Abstract · 18 pages, 7 figures. Substantially revised version following public PREreview feedback, with 30-seed robustness analysis, correlated estimat…

AI-native software engineering increasingly combines human judgment, historical analogy, parametric estimation, and AI-generated forecasts in the same prioritization decision. We study five quantitative operators used in the D-POAF decision practice and stress-test them under uncertainty, estimator dependence, drift, shared context error, dependencies, and limited capacity. Robustness extensions use 30 deterministic seeds. Moderate scoring changes preserve broad order while materially changing funded-set membership. Under independent estimator errors, inverse-MSE aggregation reaches MAE 0.615, a 42.2% reduction relative to the best individual estimator. The gain falls to 4.4% at error correlation rho = 0.6 and becomes negative at rho = 0.8, while calibration-set regression remains slightly better than the best individual and approaches a test-set oracle. Under estimator drift, four-Wave adaptive weighting reduces average RMSE by 4.64% (95% CI [4.50%, 4.78%]). Model-collective divergence achieves ROC-AUC 0.908 with independent channels and remains above 0.878 with shared noise up to 0.8. We separate budget-constrained portfolio selection from ODP's native dependency-aware sequencing role. When ODP distance is repurposed as a portfolio-selection heuristic, value-to-effort is stronger in the tested generator. For a fixed selected set, ascending ERS/BVS is optimal for an ERS-weighted completion objective without dependencies and reaches mean efficiency 0.982 versus the exact precedence-constrained optimum at dependency intensity lambda = 1.0. Scaling to 100 Feature Blocks, ODP has the highest mean early-value AUC among the transparent sequencing baselines, with a small margin over PVS ordering. These results characterize both the validity region of evidence-weighted human-AI prioritization and the boundary between selection and sequencing objectives.

0.54

Efficient and Accurate Causal Validation of Training Data Attribution

cs.LG stat.MEstat.ML ·2026-11-14 ·Simon Vary, Alireza Mousavi-Hosseini, Roger Grosse
similarity 0.56
Abstract · 20 pages, 9 figures

Causal validation of training data attribution (TDA) methods is costly, since evaluation involves retraining a model on many different subsets of the original training set. For example, the Linear Datamodeling Score (LDS), which has become a standard metric for evaluating TDA methods, measures the correlation between TDA predictions and expected retrained model outputs. In this paper, we reduce the amount of computation needed to estimate the LDS along three axes: the number of sampled training subsets, the number of retraining seeds per subset, and the number of models in the attribution ensemble. Each of these factors induces a form of bias in the LDS estimation, and removing these sources of bias significantly reduces the number of retrainings needed to match the same accuracy in LDS estimation. On ResNet-9/CIFAR-10, finite-subset debiasing achieves $3.2\%$ relative RMSE with as few as $4$ subsets, and $32$ subsets match the accuracy of ordinary Spearman using $96$ subsets, measured against a 300-subset reference. On ResNet-9/CIFAR-10, BERT/QNLI and GPT-2/WikiText-2, estimating the training-noise variance allows us to reproduce the standard five-seed LDS at the same accuracy with $2$--$5\times$ fewer retrainings.

0.29

Balance of Benchmarks: Semantic Density Reweighting for Task-Conditioned Model Comparison

cs.AI cs.CLcs.LG ·2026-08-30 ·Jhen-Ke Lin, Hong-Yun Lin
benchmark similarity 0.55
Abstract · 65 pages including references and appendices. Expanded evaluation with WildScores, a collection of 148 developer-reported benchmarks

Model comparison increasingly relies on large collections of publicly reported benchmark scores, yet common aggregation strategies trade off evidence coverage against control over capability weighting. Manually curated suites leave potentially informative evaluations unused, while uniform averaging retains them but gives greater influence to capabilities that happen to be benchmarked more densely. We introduce Balance of Benchmarks (BoB), a framework that retains eligible benchmark evidence while adapting its influence for task-conditioned model comparison using only public aggregate scores. BoB combines semantic density weighting, score equating across benchmarks of different difficulty, and task-relevant residual pooling. We evaluate it on 605 configurations across 14 Artificial Analysis benchmarks and on WildScores, a collection of 148 developer-reported benchmarks evaluated with held-out source-lineage families. On WildScores, BoB-Support raises family-mean Spearman correlation from 0.764 under uniform standardized averaging to 0.823, reduces MAE from 6.19 to 5.10 normalized score points, and increases three-model shortlist hit rate from 65.3% to 72.6%. BoB-Constant reaches a Spearman correlation of 0.831 and a hit rate of 74.6%. Separately, density weighting reduces average ranking changes when benchmarks are repeated, including as paraphrased copies. BoB-Support also reduces retrospective three-model shortlist regret from 2.08 to 1.67 normalized score points. BoB makes benchmark inclusion, redundancy, and task relevance explicit and testable measurement choices, allowing existing benchmark evidence to be used more fully while moderating the influence of benchmark proliferation.

0.24

Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers

cs.LG cs.CL ·2026-10-16 ·Srinivasan Iyengar
similarity 0.54
Abstract · 27 pages (18 main + appendix), 14 figures, 11 tables. Code will be released

A common way to make language models and agents more accurate is to spend more compute at inference time: generate many candidate solutions and submit the one a verifier prefers. Success then requires that some candidate is correct and that the verifier picks it. The first grows predictably with the number of samples; the second does not, because real verifiers (unit tests, majority votes, LLM judges, reward models) accept wrong answers, reject right ones, and make correlated mistakes. With a fixed budget, a practitioner must decide how many samples to draw, which models to draw them from, and how much to spend on verification, and today these choices are made by rules of thumb. We introduce SPLIT, which makes them from a small labelled pilot. It rests on an exact formula for the probability of submitting a correct answer under accept/reject and top-score selection that accounts for varying task difficulty, blind spots shared across models, and correlated repeated verifier calls. SPLIT fits the formula by empirical Bayes and uses it to plan budgets on unseen tasks. Across seven benchmarks spanning math, science QA, code and software-engineering agents, 26 models, 175 agent systems and six kinds of real verifiers, all from public releases, SPLIT lands within 0.02-0.8 percentage points of the best allocation in hindsight, while every fixed rule we test loses at least 2.5 points in some setting. Regret depends only on how well SPLIT ranks configurations, and it stays near-optimal under compute, price and latency costs and online. The formula also explains when asking more LLM judges lowers precision. As a forecaster far beyond the pilot, SPLIT beats power-law fitting when the verifier caps success, and we report where it loses.

0.56

Small area learning

cs.LG stat.MEmath.ST ·2026-10-11 ·Katarzyna Reluga, Nicola Salvati, Setareh Ranjbar, Nikos Tzavidis, Mark van der Laan
similarity 0.54
Abstract · 53 pages, 1 figure, 10 tables

Small area estimation (SAE) traditionally borrows strength across domains through a prespecified working model. We introduce small area learning (SAL), a learner-agnostic framework that formulates SAE as a multi-target semi-supervised learning problem and treats outcome prediction as a nuisance task. Areas may differ through unrestricted covariate shift and structured conditional shift, with information shared through common and area-specific components. We develop standard and cross-fitted estimators for low-complexity and flexible learners, unifying synthetic, model-based, and fully residual-corrected estimation. For area means, we establish consistency and asymptotic normality and provide variance estimators and confidence intervals. We extend the theory to settings with only aggregated population covariates and characterize the resulting aggregation bias. SAL reveals a bias--variance trade-off: partial residual correction can reduce mean squared error, whereas full correction yields estimators robust to working-model misspecification, potentially at higher variance. Simulations yield RMSE reductions of 34\%--49\% relative to direct estimation, with the best configuration varying across settings. An empirical application favours a cross-fitted estimator using a flexible learner under the chosen validation criterion. The numerical studies also illustrate how shrinkage can produce overly optimistic confidence intervals at the expense of strong modeling assumptions.

0.38

How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats

cs.CL cs.AIcs.HCstat.ME ·2026-09-20 ·Ian Arawjo
code/data similarity 0.54
Abstract · 39 pages, 20 figures

Researchers across academia increasingly base significance claims on LLM judge scores and small-sample AI evaluations. Yet without well-calibrated confidence intervals (CIs), hypothesis tests, and judge-bias corrections, such claims are unreliable. We address these issues in several contributions. First, we find that running statistics over raw LLM judge scores leads to inflated false positives: counterintuitively, for many inter-rater agreement metrics, false positive risk peaks at "almost perfect" human-LLM agreement. To help researchers understand how to run statistics over LLM judges responsibly, we present guidance and tooling for the statistical analysis of mixed human-AI judge designs, and implement nine hypothesis tests via prediction-powered inference (PPI), including the first known PPI corrections for four rank-based tests (Wilcoxon signed-rank, Mann-Whitney U, and omnibus variants). To keep PPI++ stable with small human-labeled calibration sets, we introduce bootstrap-adaptive power tuning, which shrinks the estimated weight toward a target estimated from the labeled data, and accounts for that weight's own sampling variance. Second, through Monte Carlo simulations, we derive recommendations for what CI, p-value, and FWER correction methods to use for small-sample AI evaluations (N<100), and warn researchers against bootstrap CIs. We package these recommendations into evalstats, an open-source Python package that selects calibrated methods automatically, and demonstrate it in three scenarios, including one where a real LLM judge validated at "substantial agreement" would have led a researcher to publish a spurious finding. evalstats is publicly available at https://github.com/ianarawjo/evalstats.

0.43

Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts

cs.LG ·2026-09-27 ·Cheng Peng, Ruixi Luo, Zhi Chen, Wei Tang
similarity 0.53
Abstract

Large language models (LLMs) are increasingly used to forecast real-world events, but access to the strongest individual forecaster may be costly or otherwise constrained. We study weak-to-strong forecast aggregation: can individually weaker LLM forecasters be aggregated to outperform a stronger forecaster? Using ForecastBench (Karger et al., 2025), we evaluate 70 LLM forecasters across 16 comparison groups, each with more than 1,000 shared subquestions, yielding 1,121 weaker-model pairs. Within each group, we identify the strongest individual by test Brier score and evaluate aggregates composed exclusively of weaker forecasters, with aggregation weights learned on separate training data. We find substantial evidence of weak-to-strong improvement. Learned linear pooling identifies a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16 groups. We also find that these improvements do not rely on having a near-best constituent and are generally accompanied by good calibration. Additional analyses show that adding more models does not consistently improve performance, and competitive weaker-model aggregates also remain available under practical constraints.

0.40

SteelAgent: LLM-Orchestrated, Physics-Informed Steel Property Prediction with Audited Generalization and Calibrated Uncertainty

cs.LG cond-mat.mtrl-sci ·2026-11-03 ·Aleksandr Volkov, Roman Sultimov, Mikhail Kuzin, Yury Maximov
top venue similarity 0.53
Abstract · 10 pages, 3 tables. A shorter version of this work was published in the Proceedings of the Thirty-Fifth International Joint Conference on Ar…

Steel property models can perform well on random test splits even when they extrapolate poorly to unfamiliar alloys. SteelAgent couples metallurgical descriptors and machine-learning models with a language-model interface for property prediction, uncertainty reporting and inverse design. We examine how the choice of held-out unit changes the conclusions drawn from SteelBench tensile-strength evaluations. For gradient boosting, pooled out-of-fold $R^2$ fell from 0.859 on a random split to 0.113 when steel families were held out. Empirical coverage of nominal 90% prediction intervals ranged from 91.0% under random splitting to 37.6% on held-out sources. The effect of physics-derived features depended on the evaluation. They increased gradient-boosting $R^2$ by 0.2534 in a 17-family holdout analysis and decreased it by 0.358 in a separate four-grade laboratory analysis. We report group-level errors and a synthetic regression experiment illustrating how calibration can deteriorate on held-out groups. These results motivate reporting the evaluation population alongside a property estimate or compliance score.

0.46