CS Paper Radar

Repository-Scale Performance Characterization of the IO500 Benchmark

Aasish Kumar Sharma, Anila Ghazanfar, Sepehr Mahmoodianhamedani, Sascha Safenreider, Julian Kunkel

cs.DC ·submitted 2026-10-04 20:31 UTC · v1
publishedbenchmark

Abstract

The IO500 benchmark provides a common basis for evaluating high-performance computing (HPC) storage systems, while its growing submission repository also offers an opportunity to study performance behavior across systems and time. This work presents a repository-scale characterization of 294 post-reset IO500 submissions from 116 sites spanning 2019--2025. We combine descriptive and temporal analysis with phase-level correlation, scale-sensitivity analysis, and benchmark-log examination to investigate how performance characteristics represented by individual benchmark phases relate to composite rankings. The results show strong relationships among phases measuring similar I/O behavior, but weaker and scale-sensitive relationships between bandwidth and metadata performance. Temporal and log-level analyses further reveal information that is not apparent from aggregate scores alone. These findings demonstrate the value of analyzing IO500 results beyond leader-board rankings and identify important considerations concerning system scale, benchmark execution, and repository provenance when using community benchmark data for comparative and longitudinal HPC performance analysis.

Comment
5 pages, 2 figures, 3 tables. Accepted at the Performance Measurement, Modeling, and Benchmarking Workshop (PMBS 2026), held in conjunction with SC26, Chicago, IL, November 2026

Closest papers

Nearest neighbours by meaning across the whole index.

Revisiting Index Performance under High Cache-Miss Penalties

cs.DB ·2026-11-16 ·Mathew M. Kim, Beomseok Nam
similarity 0.61
Abstract

Modern data-intensive applications often query very large datasets, making index performance critical. Recent index structures exploit the fine-grained access capabilities of emerging storage devices such as DCPMM and CXL. However, existing benchmarks typically evaluate indexes in isolation using saturated, closed-loop workloads that emphasize maximum throughput. Such workloads favor highly concurrent designs while overlooking the application work that follows an index lookup and competes for cache, memory bandwidth, and CPU resources. We present AuxBench - a microbenchmark that evaluates indexes as auxiliary access paths within larger application workloads. Using AuxBench, we evaluate four byte-addressable indexes under controlled post-lookup memory activity, concurrency, and request arrival patterns. Our experiments show that post-lookup memory activity can change the relative performance of competing indexes, that concurrency-oriented designs may lose their advantages when substantial application work is introduced, and that time-varying request arrivals can produce severe tail latency near saturation. These findings demonstrate why index-only benchmarks may not adequately predict performance in larger application workloads

0.22

Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM SPE

cs.DC ·2026-10-23 ·Jie Ren, Ivy Peng
similarity 0.60
Abstract

Memory demands from data analytics and LLM workloads grow rapidly, driving up memory costs in cloud and data centers. Multi-tier memory has emerged as a promising solution to balance performance and cost. However, effectively leveraging it requires a deep understanding of workload memory behaviors. This work introduces a high-resolution memory characterization methodology for analyzing workload-specific opportunities for multi-tier memory usage. In response to the increasing deployment of ARM-based processors in data centers, we implement our methodology using the recent Statistical Profiling Extension (SPE) for ARM. By characterizing eight memory-intensive workloads from CloudSuite and LLM applications, we identify region-specific access patterns, evaluate the effectiveness of cloud-oriented cache hierarchies, and propose new metrics based on memory idle periods and latency profiles to determine suitable data placement for tiered memory.

0.19

Design and Empirical Evaluation of a Network-Centric, On-Premises Architecture for Earth Observation Data Access

cs.DC cs.NIcs.PF ·2026-08-20 ·João Pinelo, João Gonçalves, Denis Willett, Amit Ruhela, Derek Steinmoeller +5 more
similarity 0.59
Abstract · 18 pages, 8 figures and 13 tables

Earth observation (EO) programmes generate data at volumes that exceed the transfer and storage capacity of most institutional networks. Public cloud platforms address this for well-resourced organisations, but institutions across the Atlantic basin face constraints in connectivity, sovereignty and funding that make on-premises infrastructure the only viable path. Cloud-native data formats enable efficient partial reads, yet their performance depends on the bandwidth of the underlying network fabric, a dependency rarely measured in isolation. This paper presents a replicable, network-centric architecture for on-premises EO data access, evaluated at its first operational deployment: the AIR Data Centre, founding node of the Atlantic Cloud. The system comprises a MinIO object storage cluster on a 100 GbE fabric, a PostGIS metadata catalogue and an OGC API-EDR access layer. We characterise the fabric under sustained parallel load, evaluate object storage throughput for EO-representative workloads, and compare measured performance against throttled baselines on identical hardware, isolating network bandwidth as the sole variable. Multi-site replication benchmarks with partner institutions characterise the federation primitive the model depends on. Network bandwidth is the dominant constraint on storage throughput for bulk EO data access up to a threshold; beyond it, endpoint memory topology rather than capacity governs how much bandwidth a system can use. For this hardware class that threshold lies above 10 Gbps per server. Below it, network capacity alone sets what the facility can deliver; above it, the return on further network investment depends on endpoint memory provisioning, which can be deferred and bought later.

0.34

PonyTail: Profiling and Improving Service Tail Latency

cs.PF ·2026-10-03 ·Roee Wodislawski, Adam Morrison
similarity 0.58
Abstract

Datacenter services must meet tight tail latency service-level objectives. Research on improving tail latency focuses on reducing queuing delay by approximating optimal request scheduling policies. As request scheduling nears optimality, the primary way to further improve tail latency is to reduce the service time of requests. Unfortunately, datacenter services lack obvious hotspots for general service time optimization. We point out a new optimization opportunity in targeting the tail service time of services with dispersive service times, which are common in datacenters. To show the feasibility and benefit of this approach, we present PonyTail, a performance analysis methodology and toolkit for analyzing service-level outlier behaviors associated with tail service time. PonyTail helps performance engineers identify control-flow and microarchitectural patterns responsible for service time outliers, as well as their root causes. We apply PonyTail to analyze five latency-critical services, including an in-memory database system and an online search engine. Based on PonyTail's analysis, we optimize some of the services, improving their 99th percentile service time and latency by 10.7%--46% and 21%--$5\times$, respectively.

0.22

Shards on a Shoestring: Empirical Characterization of NEAR Protocol Nightshade Sharding on Commodity Hardware

cs.DC ·2026-09-10 ·Sohini Sahukar, Om Amit Gandhi, Ioan Raicu
similarity 0.58
Abstract

NEAR Protocol's Nightshade architecture targets one million transactions per second (TPS) through horizontal sharding of both state and computation. Published benchmarks were produced on expensive Google Cloud Platform infrastructure costing approximately \$700 per hour, leaving a significant reproducibility gap for academic research. We present the first independent empirical characterization of NEAR Nightshade sharding on commodity hardware: a Chameleon Cloud bare-metal node with 48 hyperthreaded Intel Xeon cores, 128\,GB RAM, and HDD storage at 80--100\,MB/s. We systematically sweep shard count from $N{=}1$ to $N{=}24$, measuring aggregate TPS, per-shard TPS, block time, BFT finality, memory, and disk I/O. We identify three distinct bottleneck regimes: L3 cache pressure at low $N$, witness gossip pipeline saturation at mid $N$, and coherence collapse at high $N$. A key unexpected finding is that HDD write latency acts as implicit flow control for the witness gossip pipeline. Removing it via RAM-backed tmpfs causes complete chain stall at $N{=}16$, with a 29$\times$ spike in orphan witness rate at 47\% CPU utilization. Aggregate TPS peaks at $N{=}8$ (+40\% over $N{=}1$) then reverses, with per-shard TPS collapsing 23$\times$ by $N{=}24$. Our dataset provides the first commodity-hardware calibration baseline for the companion SimPy sharding simulator.

0.27

io_uring in Oracle Database: A Hybrid Storage I/O Architecture at Production Scale

cs.DB cs.PF ·2026-09-19 ·Rajarshi Chowdhury, Akshay Shah, Margaret Susairaj, Ayush Agrawal
code/data similarity 0.58
Abstract · 21 pages, 4 figures, 12 tables. Artifacts available at https://doi.org/10.5281/zenodo.20099105

We describe the integration of io_uring into Oracle Database's storage layer and the architectural decisions required to deploy it in a production multi-process RDBMS. Our design uses per-process ring contexts that eliminate inter-process synchronization, a shared buffer registration mechanism now part of the mainline Linux kernel, and a transparent fallback to libaio on error. Evaluation on an internal development build of Oracle Database 26ai shows that io_uring's benefits concentrate on asynchronous batched I/O paths: on a mixed OLTP workload (TPC-C), io_uring delivers identical throughput while reducing server CPU utilization by 1.2 percentage points through more efficient background writes; on analytical queries (TPC-H), CPU per query drops by 8.5% (geometric mean). Isolating the write path alone shows 29% lower CPU per write and 34% higher throughput. Synchronous read paths -- the dominant I/O in OLTP -- show no improvement and even increased CPU usage for very large I/O sizes. These results motivate a hybrid I/O architecture that retains pread/pwrite for synchronous operations, adopts io_uring for asynchronous batched I/O, and falls back transparently to libaio -- a selective strategy that may also be relevant to other database systems facing the same integration question.

0.21

Towards Unified File Management for Data Lakehouse Systems

cs.DB ·2026-11-11 ·Keonwoo Oh, Akshay Mehra, Joshua Kimball, Amol Deshpande
similarity 0.57
Abstract · Submitted to ICDE 2027

Data lakehouse systems have garnered widespread adoption, yet their metadata management systems face a fundamental architectural dilemma. File-based OTFs (Open Table Formats) scale well, but suffer from metadata bloat, high commit latencies, the absence of multi-table transactions, and unsafe garbage collection. On the other hand, catalogs relying entirely on a transactional database incur severe scalability bottlenecks due to analytic file-level metadata. To reconcile this tension, we present Raven, an operational catalog that stores transactional metadata within a database while offloading analytical metadata to underlying OTF structures. Based on this hybrid architecture, Raven introduces an extended Merge-on-Read (MoR) mechanism to reduce commit latency and mitigate metadata bloat, alongside multi-table transactions under snapshot isolation. Furthermore, Raven unifies core data lakehouse semantics including versioning and reference tracking under an extensible ER (Entity-Relationship) metadata model, enabling safe global garbage collection across diverse types of assets. Empirical evaluations demonstrate that Raven substantially reduces commit latency and metadata bloat when integrated with Apache Iceberg and Delta Lake, maintains a catalog storage footprint orders of magnitude smaller than database-only systems like DuckLake, and achieves scalable garbage collection with minimal disruption.

0.14

Rolling Round-Robin Rate: Standard Heterogeneous Throughput for SPEC CPU

cs.PF cs.ARcs.DC ·2026-08-30 ·Mahesh Madhav, Christoph Müllner, Jiangning Liu, Philipp Tomsich, Jeff Baxter
publishedbenchmark similarity 0.57
Abstract · 6 pages, 1 figure, Presented at the 2026 IEEE International Symposium on Workload Characterization (IISWC 2026), Boulder, Colorado

SPEC CPU has long provided a common foundation for comparing processor, compiler, memory-system, and platform performance. Its multi-copy SPECrate mode measures homogeneous throughput by running many copies of the same benchmark at once. That mode remains valuable, but modern cloud and server systems commonly run heterogeneous collections of jobs whose interactions are shaped by shared caches, memory bandwidth, power management, operating-system scheduling, and noisy neighbors. SPEC CPU 2026 introduces Rolling Round-Robin Rate (RRR), an exhibition run style that uses the existing rate suites to generate deterministic heterogeneous multiprogrammed workloads. This paper describes RRR as a benchmark methodology and proposes a SPEC-like scoring model for future RRR reporting: compute per-benchmark average throughput and coefficient of variation from per-copy ratios, compute a SPEC-style geometric mean for each copy across the suite, and average that population of copy geomeans to form a suite score with its own coefficient of variation. RRR therefore preserves the familiar SPEC throughput tradition while exposing richer information about variability, interference, and heterogeneous-system behavior.

0.28