CS Paper Radar

Adaptation Fidelity of SPEC CPU2026

Doa'a Al-Otoom, Mahesh Madhav

cs.PF ·submitted 2026-08-27 20:56 UTC · v1
publishedcode/data

Abstract

Standardized benchmarks are often criticized for not being "real workloads," but this critique is rarely backed by data. This paper provides the first systematic, quantitative analysis of the "fidelity gap" between the SPEC CPU2026 suite and its original, upstream open-source counterparts. We compile both the SPEC benchmarks and their upstream applications and execute them with official input workloads under two scenarios: a single-copy latency run and a 192-copy throughput run. Our findings show that most benchmarks exhibit high fidelity in single-copy runs, while a few outliers reveal the impact of SPEC's adaptation process. The multi-copy results further highlight the necessity of this adaptation: several benchmarks become significantly more efficient than their upstream versions under heavy load, underscoring the importance of I/O reduction. This work offers data-driven validation of SPEC's methodology, showing that the fidelity gap is not a flaw but a quantifiable consequence of enforcing portability, determinism, and CPU-centric measurement.

Comment
5 pages, 2 figures, Presented at the 2026 IEEE International Symposium on Workload Characterization (IISWC 2026), Boulder, Colorado

Closest papers

Nearest neighbours by meaning across the whole index.

SPEC CPU2026 Under the Microscope: Programs, Phases, and Samples

cs.AR cs.PF ·2026-11-12 ·Peijun Li, Pei-Hung Lin, Tianming Cui, Caiwen Ding, Yang Katie Zhao +1 more
benchmark similarity 0.79
Abstract

SPEC CPU2026, the latest generation of the SPEC CPU benchmarks, is set to become the standard benchmark suite in computer architecture research, where new designs are often evaluated through sampled simulation with tools such as SimPoint. Existing characterizations of the suite focus on whole-run averages, which reveal neither how behavior changes within a run nor how well sampling captures it. We characterize CPU2026 on real hardware at three levels: programs, phases, and samples. At the program level, we report whole-run metrics and top-down bottleneck classes for all four suites and examine how they change with workload size, compiler version, and platform, across machines with three different instruction set architectures: x86, Arm, and RISC-V. Notably, changing only the compiler version can halve or nearly double the IPC of individual benchmarks. At the phase level, we track hardware counters every 10 ms over all 90 SPECrate inputs, identify recurring patterns of time-varying behavior, and divide the inputs into three groups: steady, multi-phase, and bursty. More than half of the inputs show time-varying behavior, and IPC within a single input can vary by more than an order of magnitude. At the sample level, we validate SimPoint against hardware measurements of every interval it can select. SimPoint is accurate on steady inputs, but its error reaches 43% on the others, and changing only the random seed spreads the error of one input over a range of 58 percentage points. Keeping SimPoint's clusters but choosing each representative by measured CPI brings every input within 2.6%, so the error comes from the choice of representatives, not from the clustering. We conclude with practical recommendations on sampling strategy for CPU2026, which give researchers a deeper understanding of the suite and show where additional simulation effort is needed to keep sampled results accurate.

0.20

Performance Characterization of SPEC CPU 2026 on AMD EPYC 9755 Processor

cs.AR cs.PF ·2026-09-01 ·Kunal Kashyap, Rajiv Ramanathan, Shayantika Bhattacharya
publishedbenchmark similarity 0.75
Abstract · 12 pages, 5 figures, 9 tables. Accepted at the 2026 IEEE International Symposium on Workload Characterization (IISWC). Best Paper Award Nomi…

SPEC CPU 2026 is the first major update to the industry-standard CPU benchmark suite since 2017. This paper presents the first microarchitecture based performance characterization of the new suite, conducted on AMD EPYC "Zen 5", also the first SPEC CPU characterization study on this microarchitecture. Using a multi-lens methodology spanning pipeline efficiency, control flow behavior, cache hierarchy pressure, and instruction mix, we analyze both SPECrate and SPECspeed suites. We introduce scale analysis, comparing single-copy to full-system behavior to expose system-level bottlenecks invisible to conventional characterization. Our analysis reveals substantial behavioral diversity across the suite, and the multi-lens analysis identifies three distinct behavioral clusters: frontend control-flow-dominated workloads that stress branch predictor throughput rather than accuracy, high-efficiency compute workloads that suffer SMT contention at scale, and memory bandwidth-bound workloads with poor L3 filtering even at single-copy. Scale-dependent effects, including SMT dispatch contention causing throughput reduction and L3 capacity interference, emerge only at full system utilization. This work establishes an empirical foundation for architectural research and workload-driven design decisions targeting next-generation datacenter processors.

0.18

Rolling Round-Robin Rate: Standard Heterogeneous Throughput for SPEC CPU

cs.PF cs.ARcs.DC ·2026-08-30 ·Mahesh Madhav, Christoph Müllner, Jiangning Liu, Philipp Tomsich, Jeff Baxter
publishedbenchmark similarity 0.69
Abstract · 6 pages, 1 figure, Presented at the 2026 IEEE International Symposium on Workload Characterization (IISWC 2026), Boulder, Colorado

SPEC CPU has long provided a common foundation for comparing processor, compiler, memory-system, and platform performance. Its multi-copy SPECrate mode measures homogeneous throughput by running many copies of the same benchmark at once. That mode remains valuable, but modern cloud and server systems commonly run heterogeneous collections of jobs whose interactions are shaped by shared caches, memory bandwidth, power management, operating-system scheduling, and noisy neighbors. SPEC CPU 2026 introduces Rolling Round-Robin Rate (RRR), an exhibition run style that uses the existing rate suites to generate deterministic heterogeneous multiprogrammed workloads. This paper describes RRR as a benchmark methodology and proposes a SPEC-like scoring model for future RRR reporting: compute per-benchmark average throughput and coefficient of variation from per-copy ratios, compute a SPEC-style geometric mean for each copy across the suite, and average that population of copy geomeans to form a suite score with its own coefficient of variation. RRR therefore preserves the familiar SPEC throughput tradition while exposing richer information about variability, interference, and heterogeneous-system behavior.

0.28

Tools-CC-Bench: a Benchmark Suite for Collective Communication with Compression in HPC and AI Workloads

cs.DC ·2026-09-08 ·Haozhe Fan, Wei Wang, Xingchen Liu, Man Liu, Xingjian Tian +10 more
publishedbenchmark similarity 0.69
Abstract · 12 pages, 10 figures, accepted by conference IISWC 2026

Distributed HPC and LLM workloads increasingly require efficient communication for scalability, yet growing data movement has become a major performance bottleneck. Communication compression can reduce this overhead and complement execution-level optimizations, but its benefits remain difficult to assess because existing benchmarks lack support for diverse backends, realistic datasets, application-specific accuracy metrics, and overlap-induced resource contention. We present CC-Bench, a lightweight, extensible, and application-oriented benchmark suite for evaluating communication compression under realistic execution conditions. CC-Bench uses declarative application-environment modeling to decouple profiling logic from communication libraries, datasets, and fidelity metrics, enabling portable cross-library evaluation. It further combines function-level interception and hardware counter monitoring to characterize per-phase latency, hardware utilization, numerical fidelity, and computation interference. With representative datasets from HPC and LLM workloads, CC-Bench evaluates three compression-enabled communication libraries on CPU and GPU clusters, revealing accuracy-performance trade-offs and bottlenecks to guide practical deployment and optimization.

0.54

VortexSynth: Profile-guided benchmark reconstruction for GPU architecture research

cs.AR cs.PL ·2026-10-08 ·Huanzhi Pu, Chulhyung Park, Chihyo Ahn, Ramyad Hadidi, Hyesoon Kim
benchmark similarity 0.67
Abstract · 14 pages, 6 figures

GPU architecture research requires representative benchmarks that can execute on new designs. However, many vendor-tailored workloads and library kernels cannot run directly on research architectures because their implementations are closed-source or rely on programming models unsupported by the target architecture. Migrating existing kernels requires manual effort to adapt their implementations and validate their behavior. This creates a need to automate workload migration to target platforms. We present VortexSynth, a framework that uses profile-guided benchmark reconstruction to construct GPU benchmarks from reference workloads. Our workflow selects hot kernels through profiling, synthesizes implementations in the target programming language, verifies output equivalence, and measures profile differences using a translated proxy implementation. The resulting profile signature gap is supplied to an agent for iterative refinement. We evaluate the method on Vortex, an open-ISA GPGPU, using robotics as the application domain. All active kernel pairs pass both output-equivalence checks, and the case studies show that iterative refinement can reduce profile distance.

0.19

HPC-Bench: A Comprehensive Benchmark for High Performance Computing Codes

cs.PF ·2026-11-02 ·Bowen Cui, Junyu Yin, Tejas Ramesh, Leo Lim, Oscar Hernandez +1 more
publishedbenchmarkcode/data similarity 0.66
Abstract · Accepted at NeurIPS 2026 Evaluations & Datasets Track (poster). 31 pages, 13 figures, 19 tables

Large Language Models (LLMs) have shown strong capabilities in code generation and reasoning, motivating extensive benchmarks for evaluating functional correctness. However, performance optimization in High-Performance Computing (HPC) poses fundamentally different challenges that are not captured by existing evaluations: HPC performance depends on low-level code, intricate memory access patterns, and computational kernels embedded within sophisticated execution contexts. We introduce HPC-Bench, a performance-grounded benchmark for evaluating LLMs as optimizers of computational kernels in realistic HPC workloads, including 114 workloads across 16 HPC computational motifs. HPC-Bench provides a fully automated, end-to-end evaluation pipeline that combines program-level correctness verification with two correctness-gated metrics, Fast@k and Speedup@k, to characterize reliability and speedup under a fixed sampling budget. Our evaluation of three representative LLMs across serial CPU, OpenMP, and CUDA settings shows that stricter speedup requirements substantially reduce the probability of finding a correct and fast optimization. HPC-Bench is available at https://github.com/Deep-Learning-Profiling-Tools/HPC-Bench.

0.48

How Weight Encoding Affects Language Model Placement and Performance on the Apple Neural Engine

cs.LG cs.ARcs.PF ·2026-08-22 ·Shahir M A
similarity 0.65
Abstract · 10 pages, 2 figures, 7 tables. v2: corrected byte-traffic accounting, withdrew the 0.77 nominal-byte factor, and attributed normalization el…

Weight compression can alter accelerator placement as well as memory traffic, complicating the interpretation of inference speedups. We investigate this interaction on the Apple Neural Engine through the public Core ML deployment path. Five independently trained language-model checkpoints span two architectures and dense fp16, int8, and ternary weights encoded with two-bit lookup tables. We combine compiler device plans, synchronized memory-controller measurements, and compute-unit exclusion controls for a fixed single-token forward workload. On an M1, the smaller fp16 export executes on the CPU despite permitting ANE execution, whereas its compressed counterparts exhibit ANE activity. The int8 export reduces warm forward latency by a factor of 1.9. The larger fp16 export also uses the ANE, indicating that dense encoding alone does not determine placement. Separate M3 energy measurements support the same direction of change. These results establish encoding-dependent placement in the measured deployment stack and show that compression comparisons require joint measurement of backend selection, latency, and traffic.

0.20

Provisioning Networks for AI Supercomputers: A Trace-Driven Study of Performance Sensitivity at Unprecedented Scale

cs.NI ·2026-10-30 ·Tommaso Bonato, Amirreza Rastegari, Daniele De Sensi, Timo Schneider, Siyuan Shen +3 more
publishedtheory similarity 0.65
Abstract · To appear in the proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2026)

AI workloads on supercomputers make interconnect performance a primary design constraint. Hardware specifications and microbenchmarks alone do not show how real workloads respond to latency, bandwidth, and jitter. We present a trace-driven study of network sensitivity for training and inference workloads at scales up to 4,096 GPUs. Our methodology combines signature-based trace decomposition with linear programming to make broad network-parameter sweeps feasible. It preserves the recorded NCCL schedules and program dependencies, and estimates a lower bound on runtime under the LogGP model. We validate the compositional analysis against monolithic linear programs, LogGOPSim, and hardware measurements at feasible scales. The resulting latency-bandwidth envelopes identify low-sensitivity regions and performance cliffs that vary substantially across workloads. They also show when bandwidth upgrades provide meaningful returns and when additional network performance has little effect. These results translate execution traces into practical guidance for evaluating network configurations while making the model assumptions explicit.

0.39