CS Paper Radar

Handling Missing Data in Performance Portability Studies

Ami Marowka

cs.DC ·submitted 2026-10-04 23:17 UTC · v1

Abstract

Missing data is a common challenge in many real-world datasets, often leading to biased results or reduced accuracy. Missing data is a pervasive problem in performance efficiency datasets, particularly in high-performance computing (HPC) performance portability studies. Performance portability scores rely on complete sets of performance efficiencies as inputs to their respective metrics; however, missing values can arise due to incomplete benchmarking, hardware constraints, or implementation gaps. Many established performance portability metrics lack mechanisms for handling missing inputs, which can result in incomplete analyses or the inability to compute scores altogether. In this study, we evaluate a range of imputation methods and algorithms for addressing missing performance efficiencies. We present illustrative examples for each approach and apply them to real-world HPC performance portability datasets. Our results highlight the trade-offs between accuracy, robustness, and computational cost, offering practical recommendations for researchers seeking to mitigate the impact of missing data in performance portability assessments.

Comment
14 pages, 3 figuers, 3 tabels, conference paper

Closest papers

Nearest neighbours by meaning across the whole index.

When Does Task-Aware Imputation Matter? An Analysis of Robustness Under Train-Test Mismatch

cs.LG ·2026-10-14 ·Maria V. Salvador, Ana F. Sequeira, Jaime S. Cardoso
similarity 0.63
Abstract

Missing data is a pervasive challenge in machine learning, often degrading predictive performance and complicating model deployment. Most existing approaches treat imputation as a preprocessing step independent of the downstream task, a separation that becomes problematic under train-test mismatches in missingness mechanisms or rates. We introduce a unified framework that treats missing entries as variables in a task-conditioned optimization problem. We instantiate this framework in two settings: an amortized approach for scenarios with missing data at both training and test time, and a latent optimization approach for settings where only training data is incomplete. The evaluation is conducted on 98 tabular datasets, and robustness under train-test mismatches is assessed on 36 datasets. Results reveal that reconstruction error constitutes a poor proxy for predictive utility (Spearman rho approximately 0.16 to 0.32), and that task-aware imputation becomes critical in scenarios where classical methods fail to recover complete-data performance, achieving win rates of up to 80% in challenging train-test mismatch scenarios.

0.17

OmniSynth: A Meta-Evaluation Framework for Benchmarking the Quality of Missingness-Inclusive Synthetic Health Data

cs.DB ·2026-11-09 ·Annabel Zhu, Océane Doremus, Ariel Guerra-Adames, Marta Avalos-Fernandez, Emmanuel Lagarde +2 more
similarity 0.58
Abstract · Accepted to the BIBM'26 ARRL-Health'26 Workshop

Synthetic data are an emerging solution to the reproducibility crisis in machine learning for health, driven by data privacy restrictions surrounding electronic health records. Yet no consensus exists on how to evaluate synthetic health data quality, existing metrics functionally overlook missingness despite its prevalence in clinical data, and benchmarking frameworks have never been empirically validated. We introduce OmniSynth, a modular Python benchmarking suite that evaluates synthetic tabular data across fidelity, missingness similarity, utility, and privacy - the first framework to explicitly evaluate missingness as a distinct dimension of synthetic data fidelity. OmniSynth further proposes a meta-evaluation procedure that stress-tests evaluation performance under controlled degradation scenarios, providing a systematic characterization of how an evaluation framework behaves across datasets, sample sizes, and variable types. Applying this procedure to OmniSynth in two health datasets (Pima Indians Diabetes; MIMIC-IV-ED), we found that scores responded monotonically to degradation, three of four novel missingness metrics captured induced missing data patterns, and individual metric contributions to the scoring system were empirically justified.

0.24

Adaptation Fidelity of SPEC CPU2026

cs.PF ·2026-08-27 ·Doa'a Al-Otoom, Mahesh Madhav
publishedcode/data similarity 0.57
Abstract · 5 pages, 2 figures, Presented at the 2026 IEEE International Symposium on Workload Characterization (IISWC 2026), Boulder, Colorado

Standardized benchmarks are often criticized for not being "real workloads," but this critique is rarely backed by data. This paper provides the first systematic, quantitative analysis of the "fidelity gap" between the SPEC CPU2026 suite and its original, upstream open-source counterparts. We compile both the SPEC benchmarks and their upstream applications and execute them with official input workloads under two scenarios: a single-copy latency run and a 192-copy throughput run. Our findings show that most benchmarks exhibit high fidelity in single-copy runs, while a few outliers reveal the impact of SPEC's adaptation process. The multi-copy results further highlight the necessity of this adaptation: several benchmarks become significantly more efficient than their upstream versions under heavy load, underscoring the importance of I/O reduction. This work offers data-driven validation of SPEC's methodology, showing that the fidelity gap is not a flaw but a quantifiable consequence of enforcing portability, determinism, and CPU-centric measurement.

0.41

Local Zero-Shot Latency Prediction for Low-End GPUs

cs.DC cs.PF ·2026-11-02 ·Arno Geimer, Beltran Fiz, Dimitris Stripelis
similarity 0.55
Abstract

Quantifying execution latency is an important consideration across Machine Learning. Existing latency-prediction approaches typically train ML models on traces collected from workload execution on target hardware. Collection of representative traces across many consumer GPUs is expensive both in organizing the devices and in manpower, causing existing work to mainly focus on a limited set of high-end GPUs available in cloud computing. To offset this cost, we show that a local host GPU can be configured to reproduce the execution performance of less powerful target GPUs. Through selectively modifying device parameters, we obtain locally measured latencies that closely approximate those observed on the target hardware. We reformulate the latency-prediction task: instead of fitting a prediction model to a dataset of execution traces, we identify local configurations that best reproduce target devices' performance, enabling zero-shot prediction. We evaluate our approach against real measurements from 60 computer-vision models executed on 21 consumer GPU models. Our method achieves a mean absolute percentage error (MAPE) of 14.21%, compared with more than 25% for comparable baselines, reducing relative prediction error by 43.6%. Predicted and measured latencies exhibit an average Spearman correlation of 0.961 across training tasks, demonstrating that the method also accurately preserves performance rankings across devices.

0.20

RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems

cs.PF cs.AIcs.ARcs.IR ·2026-09-04 ·Zlatan Feric, Amir Taherin, Bin Ren, Yanzhi Wang, Jennifer Dy +1 more
publishedcode/data similarity 0.53
Abstract · 18 pages, 9 figures

We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU environments. RAGMark evaluates diverse RAG components, including retrievers, vector databases, prompt-processing methods, and generator models, while collecting detailed per-stage metrics such as latency, GPU utilization, memory consumption, power usage, time to first token (TTFT), throughput, and answer quality. The framework is highly extensible, separating RAG stages, timing, and resource monitoring into modular components, and is designed to efficiently sweep large configuration spaces while minimizing repeated model and database initialization overhead. Using RAGMark, we characterize five RAG workloads on open-domain QA datasets across varying retrieval depths, model scales, reranking, compression methods, and vector database configurations. We show that while autoregressive generation dominates latency in naive pipelines, context-reduction techniques shift bottlenecks across compute, memory bandwidth, and preprocessing stages. Reranking and compression produce compounding benefits: reranking reduces compression workload itself, while both jointly reduce prefill and KV-cache traversal costs, lowering energy consumption by up to 66%. We further observe strong cross-stage interactions, where small upstream context reductions cascade through downstream latency, memory traffic, and energy consumption. The RAGMark source code is publicly available at: https://github.com/zferic/RAGMark.

0.73

Hybrid Methods for Robust Tabular Data Imputation

cs.LG ·2026-09-30 ·Jinwei Li, Michelle Bruch, Daniel Tenbrinck
benchmark similarity 0.53
Abstract · 35 pages, 8 figures

Missing data are a fundamental challenge in statistical analysis and machine learning, as the choice of imputation method substantially impacts downstream inference. In this work, we propose two hybrid imputation methods called NuclearForest and SoftForest, which combine nuclear-norm-based low-rank initialization using Singular Value Thresholding (SVT) and SoftImpute, respectively, with a non-iterative Random Forest refinement. For the SVT-based component, we further introduce an adaptive step-size rule, prove adaptive step-size bounds, and establish convergence for the corresponding zero-initialized iteration. The low-rank initialization provides a structured warm start that captures the global covariance patterns in the data, while the subsequent Random Forest step recovers residual nonlinear signals encoding local dependencies. We conduct an extensive benchmark on diverse datasets from different application domains, comparing the proposed methods with seven established imputation methods under the Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR) mechanisms across varying missingness rates. Our results demonstrate that NuclearForest and SoftForest match or exceed the imputation fidelity of state-of-the-art iterative methods such as MissForest, while significantly reducing computational cost. In particular, they achieve speedups of approximately 5.81 times and 9.52 times over MissForest by replacing iterative cycles with a single refinement step. Our approach effectively exploits the low-rank structure of real-world tabular data and accommodates mixed-type variables, providing an efficient and robust solution for data imputation in bioinformatics, economics, and beyond.

0.29

Tools-CC-Bench: a Benchmark Suite for Collective Communication with Compression in HPC and AI Workloads

cs.DC ·2026-09-08 ·Haozhe Fan, Wei Wang, Xingchen Liu, Man Liu, Xingjian Tian +10 more
publishedbenchmark similarity 0.53
Abstract · 12 pages, 10 figures, accepted by conference IISWC 2026

Distributed HPC and LLM workloads increasingly require efficient communication for scalability, yet growing data movement has become a major performance bottleneck. Communication compression can reduce this overhead and complement execution-level optimizations, but its benefits remain difficult to assess because existing benchmarks lack support for diverse backends, realistic datasets, application-specific accuracy metrics, and overlap-induced resource contention. We present CC-Bench, a lightweight, extensible, and application-oriented benchmark suite for evaluating communication compression under realistic execution conditions. CC-Bench uses declarative application-environment modeling to decouple profiling logic from communication libraries, datasets, and fidelity metrics, enabling portable cross-library evaluation. It further combines function-level interception and hardware counter monitoring to characterize per-phase latency, hardware utilization, numerical fidelity, and computation interference. With representative datasets from HPC and LLM workloads, CC-Bench evaluates three compression-enabled communication libraries on CPU and GPU clusters, revealing accuracy-performance trade-offs and bottlenecks to guide practical deployment and optimization.

0.54

Productivity in HPC: A Critical Review from SLOC and Performance Portability to AI-Assisted Programming

cs.DC ·2026-10-05 ·Ami Marowka
similarity 0.53
Abstract · 39 pages, 2 figuers, 2 tables

Productivity has long been recognized as a central concern in high-performance computing (HPC), yet it still lacks a workable formal definition and a broadly accepted metric for quantified assessment. Recent attempts to incorporate productivity into performance-portability analysis have introduced code-divergence, convergence, and semantic measures, but have not established a convincing formal relationship between productivity and performance portability. More recent language and compiler systems, including JACC, Mojo, JLIR, and Accl++, pursue productive performance portability through higher-level abstractions, multi-level intermediate representations, automatic kernel generation, and runtime compilation. These systems demonstrate substantial engineering progress, while productivity is still commonly represented through programmability, ecosystem reuse, code size, or reduced implementation burden rather than direct measurement of human effort. This paper presents a critical review of HPC productivity measurement, tracing its development from source lines of code (SLOC), static complexity, and empirical usability studies to development-time models and semantic approaches. We also examine how automated migration and artificial-intelligence(AI)-assisted programming weaken the correspondence between code transformation and human effort, shifting effort toward verification, correction, and performance tuning. The reviewed methods are compared in terms of what they measure, directness, objectivity, reproducibility, practicality, and their relationship to performance and portability. We conclude by identifying the principal limitations and open challenges for quantified HPC productivity assessment.

0.34