CS Paper Radar

Performance Evaluation of RED-ONION: A High-Speed Disk-to-Disk Transfer System

Keichi Takahashi, Hiroaki Kataoka, Takeo Hosomi, Ayahiro Takaki, Yasunori Kakizawa, Shuichi Ihara, Nobuaki Hashizume, Susumu Date

cs.DC cs.NI ·submitted 2026-08-29 05:02 UTC · v1

Abstract

Modern experimental instruments produce data faster than general-purpose file transfer interfaces can move it, so delivery to the computing infrastructure has become a bottleneck in the research process. At many universities and research institutes, moreover, the instruments that generate research data and the high-performance computing systems that analyze it are separated both geographically and organizationally, because each demands its own expertise and installation environment. Connecting the two seamlessly is a pressing challenge for data-driven science. This article presents RED-ONION, a high-speed disk-to-disk transfer system that connects research facilities, on campus and beyond, to a computing center. The system combines data transfer nodes, a dedicated high-bandwidth network, an all-flash parallel file system, and multi-threaded transfer software that parallelizes network transmission and storage access. The design targets the wire rate both along the entire path, from the read on the sender storage to the write on the receiver storage, and for a single file between one pair of nodes rather than only in aggregate over many files or nodes. We describe the end-to-end optimizations across the transfer software, the operating system, and the storage that this requires. We evaluate a prototype deployed over a 100 Gbps transpacific path between Atlanta and Tokyo with a 150 ms round-trip time, on which a single 1 TB file transfer reached 90 Gbps, delivering a terabyte in approximately 95 s. Moving a dataset of this size therefore becomes a routine step, and the computing center serves an instrument as if the two were co-located.

Comment
11 pages, 8 figures, 3 tables. Submitted to IEEE Access

Closest papers

Nearest neighbours by meaning across the whole index.

Seeking Cost-Optimal Infrastructure Size for Distributed Filesystems: A Ceph Case Study

cs.DC cs.PF ·2026-09-22 ·Niccolo Tosato, Isac Pasianotto, Ruggero Lot, Stefano Cozzini
top venue similarity 0.66
Abstract · Accepted version. Published in SC Workshops '25 (St Louis, MO, USA, 2025). Version of Record: https://doi.org/10.1145/3731599.3767427

Distributed Filesystems (DFS) are a crucial component of modern computing environments, and their performance is critical to the success of all the facilities that rely on them. However, predicting the DFS I/O performance solely based on the storage system hardware is not trivial. In this paper, we address this challenge by presenting an empirical method that tries to quantitatively assess how hardware configuration choices influence the performance of a DFS using Ceph as a case study. We investigate the influence of three hardware parameters-number of CPU cores, amount of RAM, and disk bandwidth. To control these variables, we relied on the Linux hotplug interface and Cgroups, avoiding additional software overhead. Our results reveal that for the analyzed workloads, decreasing hardware resources does not always yield proportional performance losses. This method offers practical insights for designing cost-effective distributed storage systems, remaining general enough to be applied to other filesystems.

0.67

File-Semantic SSD for I/O-centric Architectures with NFS-over-NVMe Protocol

cs.OS ·2026-11-04 ·Kyu-Jin Cho, Injae Kang, Keunsan Park, Jaehoon Shim, Munsung Bae +3 more
similarity 0.62
Abstract

Emerging I/O-centric architectures empower peripheral devices to access storage directly while bypassing the host CPU and kernel. However, SSDs expose only a block interface and rely on the host filesystem for file metadata and access control, making file-level access for applications and peripheral devices inherently difficult in I/O-centric architectures. This tight coupling between the host filesystem and storage device introduces unavoidable software overhead and limits the potential of fully hostless I/O paths. In this paper, we present File-Semantic SSD (FS-SSD), a new SSD architecture that provides a native file abstraction within SSDs, enabling direct file access by peripheral and remote devices without host intervention. FS-SSD adopts the lightweight, stateless NFS interface and introduces NFS-over-NVMe, a protocol extension that embeds NFS operations into the standard NVMe command structure. This allows FS-SSD to support local and remote file I/O through existing PCIe and NVMe-oF transports. Our evaluation shows that FS-SSD provides on-device file abstraction while fully utilizing raw-device bandwidth in I/O-centric architectures.

0.10

Creating SquashFS Images to Support HPC and AI Datasets

cs.IR ·2026-10-14 ·Julie Mullen, Albert Reuther, LaToya Anderson, Michael Jones, William Arcand +16 more
similarity 0.60
Abstract · 8 pages, 5 figures, IEEE High Performance Extreme Computing (HPEC) Conference 2026

HPC users often download datasets on HPC systems for analysis and processing. When these datasets are comprised of hundreds of thousands (or more) of small files, accessing them in parallel can put a tremendous burden on the metadata servers of central parallel file systems. If the dataset is read-only, an effective solution is to compress the dataset into a SquashFS image. There are, however, a number of workflows that can be used to create such a dataset image, and we observed that some were more efficient than others. To provide more effective guidance for our userbase, we benchmarked eleven different workflows for creating SquashFS dataset image files of datasets with thousands to millions of very small files. We took into account whether the data arrived as tar.gz, zip, or files-and-directories downloads, whether the image creation was on the central file system or local compute node storage, and the number of SquashFS file compression threads used. This paper shares the results of these benchmarks and provides recommendations to HPC system users toward the best workflows to use for SquashFS image creation.

0.25

FDP: The Data Placement Promise of Modern NVMe SSDs

cs.OS cs.DB ·2026-10-02 ·Sijie Lan, Hui Qi, Xing He, Mahmut Kandemir, Javier González +1 more
code/data similarity 0.59
Abstract · Includes appendices

NVMe SSDs are now widely deployed as the storage tier in data centers. As SSDs have evolved over the past decade, the commu- nity has continued to debate the interfaces they expose and how operating systems and storage systems should exploit them. The NVMe Flexible Data Placement (FDP) proposal is the latest point in this design space. FDP introduces an interface based on Reclaim Units that enables explicit data placement to reduce device write amplification without the software engineering costs of sequential- write constraints and host garbage collection. FDP-enabled SSDs are emerging in commercial products and early data center de- ployments. Their compatibility with conventional block I/O allows existing applications to run unchanged, allowing a frictionless adop- tion in industry. This paper presents an experimental evaluation of FDP SSDs to characterize their data placement guarantees over the raw device interface. We then revisit two widely deployed and distinct open- source storage systems, MySQL and RocksDB, and examine whether lifetime-based data separation and distinct write patterns built into their architectures can be mapped onto FDP SSDs without inva- sive changes. Our evaluation shows end-to-end WAF reductions at higher device utilization, along with QoS and throughput improve- ments under synthetic and real-world workloads. These results demonstrate that FDP provides a practical and deployable cross- layer mechanism for data placement with open-source ecosystem support on Linux. They also highlight why FDP SSDs are gaining traction in industry.

0.22

Callosal: Rethinking the Kernel Page Cache for Disaggregated Storage and Memory

cs.OS ·2026-10-26 ·James McMahon, Amit Samanta, Seung-seob Lee, Ryan Stutsman, Anurag Khandelwal
similarity 0.58
Abstract

Classic distributed filesystems like NFS centralize filesystem operations on a server and use custom, filesystem-specific approaches for managing cache consistency across client nodes. This limits scalability, and it leads to weak consistency, ruling out applications that concurrently access and mutate shared files. Our key insight is that these legacy designs are now too storage-centric; with today's large memories, OS page caches do most of the "heavy lifting" in accessing data, not the filesystem and block storage layers. In response to this, we present Callosal, a page cache for a cluster of machines that is centered on shared disaggregated memory. Rather than serializing operations through a filesystem server and using ad hoc caching like NFS, Callosal is built around the existing Linux page cache paths and is accelerated by RDMA and programmable switches. Callosal delivers the strong consistency that programs expect with local filesystems (like in ext4 and tmpfs) while avoiding the additional network access delays and bottlenecks that host-based solutions incur. We compare Callosal against NFS, CephFS, and local filesystems; often Callosal can provide caching performance similar to local filesystems while also scaling across nodes. For mixed read/write workloads, on average Callosal has 4.6x higher throughput than NFS when contention is not extreme, and it offers strong consistency guarantees like CephFS, while outperforming it. For a file-backed key-value store with 1% writes and uniform access, Callosal scales 1.7x across 4 nodes, while CephFS collapses across nodes.

0.11

Ditto: Generalized Reconfigurable Linearizable Reads

cs.DC cs.DB ·2026-09-30 ·Aleksey Panas
similarity 0.57
Abstract

Linearizability gives developers the illusion that operations on a distributed datastore execute sequentially on a single machine. Providing it is expensive, so numerous specialized reads algo- rithms have been proposed to make reads significantly faster without sacrificing linearizability, which matters because most workloads are read-dominant by orders of magnitude. Yet, none of these algorithms is the best choice for every deployment. We show why with a framework that classifies these algorithms into a generalized space together with a mathematical analysis of their latencies as a function of this generalization. The analysis exposes two fundamental tradeoffs, one between read and write latency and one between read and write tolerance to network variance, which together leave the best algorithm dependent on network and workload conditions that vary over time. It also exposes a gap: there is a way to minimize the delay a read incurs when it overlaps a concurrent write, at no cost to writes, yet exist- ing algorithms can apply it only for a subset of the tradeoff space, forcing any deployment that needs the full space to settle for a strictly worse solution. We close this gap with a new algorithm, Pairwise Quorums, and build a second, Ditto, that reconfigures across the space at runtime as conditions change. Together they subsume every existing algorithm under our model: against each, the combination is either strictly better, or reconfigures to match it where a genuine tradeoff applies.

0.30

Behind a Simple Read: Understanding Work and Waiting in the Linux I/O Stack

cs.OS ·2026-10-07 ·Yang Shen, Wenzhe Zhang, Yong Dong, Min Xie, Wei Zhang +2 more
similarity 0.57
Abstract · 30 pages, 15 figures, 3 tables

A simple read interface provides uniform semantics but unpredictable performance. Using synchronous large-buffer Linux reads, we decompose the buffered-read path into work volume, processing time, and critical-path exposure. Reducing local cost or accelerating request submission need not shorten a complete read. Linux uses large folios to reduce metadata work and I/O-copy overlap to hide most cold-read copying; an experimental kernel prototype shows that more aggressive overlap and parallelism can shorten the host path when these mechanisms fail. Device waiting dominates SSD cold reads. We abstract first-completion wait F and subsequent completion capacity B_CQ from request-batch timelines. Real-SSD experiments characterize their variation with request and batch size, while MQSim traces explain their internal origins. A wait chain along the folio-bio-request mapping connects device completion to upper-layer waiting. Under strict batching, the median absolute cross-layer prediction error is at most 7.43%. Finally, we apply the completion model and wait chain to request supply, Linux readahead, data layout, and waiting policies. Cases cover gains, no gains, and reversals, comparing predictions with interventions to close the loop from observation and modeling to understanding and control and provide a measurable basis for layered I/O decisions.

0.22

Design and Empirical Evaluation of a Network-Centric, On-Premises Architecture for Earth Observation Data Access

cs.DC cs.NIcs.PF ·2026-08-20 ·João Pinelo, João Gonçalves, Denis Willett, Amit Ruhela, Derek Steinmoeller +5 more
similarity 0.57
Abstract · 18 pages, 8 figures and 13 tables

Earth observation (EO) programmes generate data at volumes that exceed the transfer and storage capacity of most institutional networks. Public cloud platforms address this for well-resourced organisations, but institutions across the Atlantic basin face constraints in connectivity, sovereignty and funding that make on-premises infrastructure the only viable path. Cloud-native data formats enable efficient partial reads, yet their performance depends on the bandwidth of the underlying network fabric, a dependency rarely measured in isolation. This paper presents a replicable, network-centric architecture for on-premises EO data access, evaluated at its first operational deployment: the AIR Data Centre, founding node of the Atlantic Cloud. The system comprises a MinIO object storage cluster on a 100 GbE fabric, a PostGIS metadata catalogue and an OGC API-EDR access layer. We characterise the fabric under sustained parallel load, evaluate object storage throughput for EO-representative workloads, and compare measured performance against throttled baselines on identical hardware, isolating network bandwidth as the sole variable. Multi-site replication benchmarks with partner institutions characterise the federation primitive the model depends on. Network bandwidth is the dominant constraint on storage throughput for bulk EO data access up to a threshold; beyond it, endpoint memory topology rather than capacity governs how much bandwidth a system can use. For this hardware class that threshold lies above 10 Gbps per server. Below it, network capacity alone sets what the facility can deliver; above it, the return on further network investment depends on endpoint memory provisioning, which can be deferred and bought later.

0.34