CS Paper Radar

Robustness Evaluation of GPU Cloud Pricing Benchmarks under Incomplete Observations: A Controlled Missingness Study

Jooya Wong

cs.AI ·submitted 2026-11-13 06:18 UTC · v1
benchmark

Abstract

Daily GPU rental-price benchmarks can fail when marketplace quotes are missing. This article stress-tests a fixed equal-weight median rule under four imposed masking mechanisms and four benchmark-level recovery methods. Across 5,800 Monte Carlo realisations in 27 scenarios on 190 held-out days, whole-source outages reduce publication coverage to 0.77 while preserving zero conditional error on published dates; composition-changing loss retains 0.86 coverage but produces 7.74 log points of error. A separate past-only stratum-level ablation reduces error by 4.15 to 6.70 log points in five composition-changing cases, using a richer information set. An observable-signal withholding rule improves the primary error-coverage criterion in 0 of 10 cases. Results concern observed asking-price measurements, not transaction prices or identified real-world missingness mechanisms.

Comment
5 pages, 3 figures, 7 tables

Closest papers

Nearest neighbours by meaning across the whole index.

KernelMutate: Mutation-Guided Validation of Generated GPU Kernels

cs.SE ·2026-11-11 ·Ziyuan Li, Zihe Song, Shuxiao Xie, Mingfang Ji, Xiaofang Qi +2 more
hot direction similarity 0.56
Abstract · 29 pages, including an appendix with supplementary evidence

Generated GPU implementations are commonly accepted after a few differential tests, although replacement obligations may remain unchecked. We present KernelMutate, an empirical study of validation reliability using controlled faults, released implementations, and replacement tests. A five-draw anchor accepts 548 of 838 CUDA candidates. The full suite finds native FP32 witnesses for 136 accepted candidates and dtype-only findings for another 362, mostly low-precision execution limitations without established publisher guarantees. Source-guided testing identifies the full suite's 692 candidates at 26.1% lower charged cost, but yields only dtype evidence for 16 accepted candidates with native witnesses in the full suite. Independently observed context and boundary findings cover the same candidate set. Task-held-out template matching reveals differences in ranking and observation coverage: global top-eight recommendations match more probes than class-conditioned lists (51 versus 48 of 75), while conditional analyses restrict the apparent advantage at smaller limits. These template-matching results are offline measurements, not deployed-suite gains. Independent controls and replacement cases expose numerical-policy, binding, and custom-execution boundaries. Validation therefore needs to report witness domains and observation coverage alongside candidate yield and cost.

0.41

Grid-Oriented Probabilistic Load Forecasting for AI Data Centers

eess.SY cs.LG ·2026-11-06 ·Qingyi Yao, Ding Lin, Tianqiao Zhao, Meng Yue, Jianhui Wang
benchmark similarity 0.54
Abstract

Probabilistic load forecasts for AI data centers need to represent both uncertainty under changing workloads and temporal dependence to support grid operations. This paper develops an open, grid-oriented benchmark using measured GPU power from a 448-GPU AI cluster. Forecasts are evaluated through four stages: probabilistic prediction, online uncertainty calibration, temporal scenario generation, and horizon-wide reserve sizing. Following a change in cluster utilization, a quantile LightGBM forecast achieves only 63.4% coverage for a nominal 90% interval, while online regime-conditioned calibration increases coverage to 87.3%. Even with well-calibrated marginal forecasts, independent sampling produces unrealistic ramp distributions and excessive reserve procurement. Reconstructing temporal dependence from historical forecast errors reduces the discrepancy in ramp statistics and lowers realized reserve cost by 7.4-10.6% across three forecasting tasks. These findings show that marginal forecast quality alone is insufficient for operational decisions and motivate joint evaluation of calibration, temporal structure, and decision outcomes.

0.22

Stress-Testing Efficient Responsible-AI Evaluation: When Compute Savings Change Benchmark Conclusions

cs.LG ·2026-08-31 ·Ahmed El Kady, Aravind Narayanan, Rehana Riaz, Yani Ioannou, Shaina Raza
benchmarkcode/data similarity 0.54
Abstract

Efficient evaluation changes the protocol used to support claims about model behavior, yet it is rarely tested whether those claims remain stable after the evaluation itself is made cheaper. We stress-test conclusion robustness in responsible-AI benchmarking by evaluating three dense and mixture-of-experts models on BBQ and BBQ-V under seven conditions spanning batching, quantization, benchmark reduction, and their combinations. Rather than treating preserved aggregate accuracy as sufficient, we compare accuracy, bias severity and prevalence, reasoning quality, subgroup behavior, subset-membership stability, runtime, and measured GPU energy against a full-benchmark BF16 baseline. Larger batching keeps accuracy within 0.35 percentage points of baseline and produces comparatively small subgroup changes, while reducing energy in five of six model--dataset settings. INT8 largely preserves quality but uses 1.79--4.26$\times$ baseline energy. INT4 causes larger, model- and context-dependent changes. Reduced benchmarks provide the most consistent savings, but very small subsets are substantially more sensitive to which items are retained. Efficient evaluation should therefore be treated as a measurement intervention whose validity must be checked across the conclusions the benchmark is intended to support. Our project website is https://vectorinstitute.github.io/sustainable-rai-evaluation/ and the code is available at https://github.com/VectorInstitute/sustainable-rai-evaluation.

0.52

Pricing Time, Not Just Tokens: Latency-Aware Mechanism Design for LLM Inference

cs.GT ·2026-09-30 ·Ian McDougall, Karthikeyan Sankaralingam
top venuetheory similarity 0.53
Abstract · 28 pages, 1 figure. Paper appears in the proceedings of the 27th ACM Conference on Economics & Computation (EC'26)

The economic theory of LLM pricing treats tokens as a homogeneous commodity considering aggregate token count as the main features buyers and sellers consider. We model inference as a service market where buyers have three-dimensional private information - willingness-to-pay, task volume, and time preference - and utility depends on latency slack alongside token quantities. Our main result is a separation theorem: discrete hardware tiers induce endogenous self-selection on time preferences, reducing three-dimensional screening to standard one-dimensional screening within each tier. We derive the cost structure from GPU inference physics - compute-bound prefill and bandwidth-bound decode - and characterize optimal tiered mechanisms via virtual-value techniques. Optimal per-task prices are volume-independent, providing theoretical grounding for flat per-token API pricing. We verify the mechanism empirically by calibrating to 8-GPU clusters of H100 and B200 hardware. The separation theorem holds in 83% of 105 tested configurations overall, rising to 96% at economically relevant WTP scales. A seller adopting two-tier pricing under the optimal mechanism captures 26-66% higher profit than the best single-tier alternative, with gains driven by efficient cross-tier allocation in regimes where hardware costs are a significant fraction of per-request value.

0.68

BitIR: Cross-Architecture Fault Injection for Resilience Analysis of Heterogeneous GPU Applications

cs.DC ·2026-10-02 ·Maisy Dunlavy, Michael Papka, Zhiling Lan
benchmark similarity 0.52
Abstract

Modern GPU-based HPC systems rely on heterogeneous vendor stacks, yet resilience studies are largely limited to single architectures, leaving it unclear how faults behave across different GPU backends. We present \emph{BitIR}, a cross-architecture fault injection framework that injects deterministic single-bit faults at the LLVM IR level, ensuring semantically equivalent perturbations prior to backend lowering and enabling direct cross-vendor comparison across NVIDIA, Intel, and AMD GPUs. We evaluate BitIR on three production supercomputers -- Polaris (ALCF, NVIDIA A100), Aurora (ALCF, Intel GPU Max 1550), and Frontier (OLCF, AMD Instinct MI250X) -- representing the full spectrum of current leadership-class GPU architectures. Using representative heterogeneous benchmarks, we conduct large-scale injection campaigns across all three systems and classify outcomes into masked results, silent data corruptions (SDCs), and failures. Our results show that identical faults produce markedly different behaviors across vendors: Intel most often masks faults but exhibits more hangs when faults escape masking, NVIDIA exposes more detectable hard failures, and AMD alternates between SDC-dominant and failure-dominant behavior depending on the benchmark and fault site. These findings demonstrate that resilience is not backend-invariant, underscoring the need for backend-aware and workload-aware fault mitigation.

0.14

Across-Design Uncertainty in Short Pricing Panels: Inference and Identification

cs.LG econ.EM ·2026-08-21 ·Pedro Cadahia Delgado
unusual similarity 0.52
Abstract · 38 pages, 4 figures

Short observational pricing panels often contain many data points but very few actual price changes. This paper shows that this sparsity creates a hidden source of error that standard statistical methods miss. When estimating price effects, most of the uncertainty does not come from sample size within a panel, but from the specific history of price movements observed. Standard confidence intervals fail because they only measure variation within the panel, ignoring this broader design-level error. Using simulations, we find that this cross-design variation accounts for most of the estimation error, causing standard methods to significantly understate uncertainty. First, we show that cross-design error decreases predictably as the total volume of price variation increases. Second, adding more data from regions that share the same price trends does not fix the issue; true precision improves only when combining data across units with independent price trajectories. Third, applying a simple variance-component adjustment across independently priced units restores accurate statistical coverage. We confirm these findings in real-world store scanner data, showing that products and pricing zones behave as if they have far fewer independent price movements than their raw counts suggest. Ultimately, reliable inference in passive pricing data requires genuine, independent variation, which can be achieved through controlled regional price testing.

0.55

One Global Beam Across Many GPUs: High-Throughput Beam Search at Billion-Record Frontier Scale

cs.DC ·2026-10-05 ·Ivan Litvak
theory similarity 0.52
Abstract · 18 pages, 15 figures

Beam search repeatedly makes many children, removes duplicates, and keeps the best $B$. We show how many GPUs can perform these steps as one search even when the retained set does not fit on one device. States and candidates stay on the GPUs; the CPU receives only small control and ancestry records. We prove that an abstract distributed pipeline returns the monolithic reduced-key top-$B$ for the same complete candidates, integer scores, Hash128 equivalence, and a fully specified physical-layout tie order, provided that its implementation performs one global reduction per key and race-free routing. A static audit of the historical one-T4 pin leaves cross-buffer uniqueness unresolved and identifies a separate multi-rank scatter risk; neither is a reproduced failure, and other revisions inherit neither defects nor correctness without comparison. On eight H200 GPUs, a Cube4 run completed saturated depth 8 at $B_{\rm eff}=2{,}900{,}361{,}216$ in $931.266$ s: $69{,}608{,}669{,}184$ nominal parent--generator pairs, or a derived $74.746$ million pairs/s. A separate two-T4 Megaminx run yielded a derived $30.274$ million nominal parent--generator pairs/s at $B_{\rm eff}=82{,}837{,}504$. Those tasks and hardware differ, so they do not establish strong scaling. Separately, on one eight-RTX-3060 host, fixed-count eight-GPU speedup was $7.487$ under a common execution profile and $5.605$ under selected stable profiles; weak actual-work throughput gain was $6.113$ and $7.295$. These profile-sensitive ratios use each series' own one-GPU baseline.

0.21

ePACT: Energy-Performance-Aware Commitment Tracking for LLM Serving

cs.DC ·2026-10-01 ·You Peng, Youhe Jiang, Chen Wang, Binhang Yuan
similarity 0.52
Abstract

Reducing LLM serving energy does not by itself guarantee lower deployment cost when electricity procurement exposes operators to unfavorable deviations from preset commitments. We study hourly commitments with positive, potentially asymmetric costs for overuse and underuse, and formulate energy-Performance-Aware Commitment Tracking: minimize deviation costs subject to request-level service requirements. We implement ePACT, a two-level controller that adjusts serving capacity and GPU clocks as requests arrive. A global planner updates interval energy targets from measured consumption and the remaining hourly commitment. A local decision maker predicts candidate configurations' energy and completion times, checks predicted deadline misses, and selects among admitted configurations by asymmetric target-deviation cost, with a service-first fallback. Coarse-to-fine action search runs asynchronously with serving. We evaluate ePACT through single-hour comparisons, controller ablations, and full-day trace simulations for H20 and H200 GPU pools. In the 24-hour simulations, ePACT reduces the asymmetric deviation cost by $73.8\%$ and $75.7\%$ relative to vLLM while retaining near-vLLM SLO attainment. Mean absolute hourly deviations are $2.16\%$ and $2.31\%$, respectively.

0.38