CS Paper Radar

SKYE: Write-Optimized Key-Value Store with Fine-Grained Control over Persistent Memory Accesses

Soujanya Ponnapalli, Sekwon Lee, Rohan Kadekodi, Vijay Chidambaram

cs.DB ·submitted 2026-09-17 18:28 UTC · v1
benchmark

Abstract

State-of-the-art key-value stores built for persistent memory (PM) provide low latency as they allow application threads to directly access the data on PM and rely on hardware to manage multiple non-volatile DIMMs (NVDIMMs). While this provides low latency, it results in low throughput and scalability. Performance degrades because PM hardware requires fine-grained control over PM accesses; for example, throughput degrades if too many threads write to PM concurrently. We present SKYE, a write-optimized PM key-value store that achieves high throughput and scalability. SKYE builds on the central idea of maintaining fine-grained control over all PM accesses and obtains high PM write-bandwidth utilization. To achieve this, SKYE deviates from current practice and provides indirect access to applications; applications send requests to SKYE, which uses dedicated threads to access PM on their behalf. Instead of relying on hardware-managed PM, SKYE controls how data is placed on individual NVDIMMs. SKYE leverages multiple media to avoid overloading PM and limits remote NUMA accesses for scalable throughput. We show that on a single NVDIMM, SKYE outperforms state-of-the-art PM stores by 2.5-5x on the standard Yahoo Cloud Serving Benchmark (YCSB). With four NVDIMMs across four NUMA nodes, SKYE obtains about 86% of PM write bandwidth, and its write throughput scales by 3.9x.

Closest papers

Nearest neighbours by meaning across the whole index.

SARIA: Persistence-Aware KV Caching with Coordinated DRAM--SSD Reload for LLM Agents

cs.OS ·2026-10-22 ·Hui Li, Qihui Zhou, Xiao Yan, Shumeng Zhang, Zijie Dai +5 more
similarity 0.68
Abstract · 12 pages, 16 figures

Multi-turn LLM agents usually reuse long context prefixes, making key value (KV) caching crucial for reducing prefill latency. SSDs provide a low-cost extension to limited DRAM capacity but introduce two key challenges. Eagerly persisting all KV cache blocks incurs substantial write traffic and rapidly consumes finite SSD endurance. Meanwhile, the small number of SSDs provisioned in typical GPU servers limits aggregate reload bandwidth, making SSD-resident KV cache much slower to retrieve than that in DRAM and inflating prefill latency. We present \textsc{Saria} (Selective Admission and Reload for Inference Acceleration), a KV caching system that jointly manages DRAM and SSD by exploiting the session-level temporal locality of LLM agents to reduce SSD writes and accelerate cache reload. On the write path, \textsc{Saria} uses eviction and reuse history to selectively persist KV cache blocks that exhibit reuse opportunities beyond their DRAM residency, avoiding unnecessary writes for KV cache already hosted in DRAM. On the read path, \textsc{Saria} promotes selected SSD-hit KV cache blocks from GPU memory into DRAM after prefill, allowing subsequent reuse within the same session to reduce SSD read traffic while limiting DRAM contention. A transfer coordinator paces DRAM reads while SSD transfers are active, allowing both paths to make progress and efficiently share GPU ingress bandwidth. We implement \textsc{Saria} on LMCache MP and evaluate it using workloads from TraceLab and WEKA, covering both full-attention and hybrid-attention models. The results show that \textsc{Saria} reduces SSD writes by 57.1--58.9\% compared with popular full duplicate caching and the mean time to first token (TTFT) by 42.3--79.7\% over DRAM-only caching.

0.08

GNoF: A Low-SM-Overhead GPU-Native NVMe-over-Fabrics Completion Path for Data-Intensive GPU Workloads

cs.DC cs.AR ·2026-11-16 ·Lei Tian, Wenhao Gu
similarity 0.66
Abstract · 9 pages, 5 figures, 4 tables

GPUs are the main compute engine for data-intensive workloads whose datasets exceed device memory and reside on remote flash. The conventional host CPU orchestrates every NVMe-over-Fabrics (NVMe-oF) transaction and stages data through host memory, costing high CPU usage, repeated CPU-GPU synchronization, and redundant data movement. GPU-initiated storage removes the CPU from the I/O path, but existing designs leave two problems: synchronous lock-step completion pins streaming multiprocessors (SMs) in spin-polling, wasteful over a 10-100 microsecond RDMA round trip, and current GPU-native remote systems focus on target-side firmware while using per-command doorbells and periodic CQ polling. We present GNoF, a GPU-native NVMe-oF/RDMA data path where GPU threads build command capsules, post send work-queue elements, ring doorbells, and reap completions entirely on the device; the CPU only handles one-time connection setup, the admin queue, and a keep-alive thread. Three mechanisms target long fabric round trips: a doorbell mover that coalesces a burst's doorbells into one MMIO ring; a batched CQ poller with a lock-free fast peek and monotonic completion counters that cannot lose a wakeup; and a set-associative, reference-pinned GPU page cache with split fetch/wait pipelining and orphan-fill takeover. GNoF needs no target or firmware change and runs against an unmodified SPDK target. On a 200 Gb/s InfiniBand fabric, GNoF reaches 1.77 million 4 KB IOPS per client at 0.37% host CPU and about 1% SM occupancy, versus 40% CPU for kernel nvme-rdma and about 100% SM occupancy for lock-step polling. Against three existing baselines (kernel nvme-rdma, BaM over a BlueField-3 DPU, and BaM on a local SSD) under identical workloads, we show 10-22x gains on selective column I/O, CPU-burn immunity, and the regime where bulk staging is faster.

0.18

SURMA: Efficient Fine-Grained Communication Offload for SuperPOD

cs.AR ·2026-11-16 ·Haoran Shang, Weifeng Zhang, Mengzhu Li, Huizheng Wang, Sihan Guan +12 more
similarity 0.66
Abstract

GPU SuperPODs underpin large language model (LLM) serving, but growing bandwidth and round-trip times make inter-GPU communication consume increasing compute resources. SM-driven remote memory access occupies shared memory or registers for in-flight data and competes for SM-to-NoC bandwidth. Dedicated engines can free these resources, but must handle fine-grained transfers, down to a 576\,B KV-cache entry. NVIDIA's CopyEngine uses less than 1\% of NVLink bandwidth at 7\,KB granularity. We present SURMA, an on-die Scale-Up Remote Memory Access engine. Adapting RDMA requires overcoming slow doorbell submission, excessive per-application connection state, head-of-line blocking of completion notifications on shared connections, and costly ordering for multipath transfers. SURMA uses a credit-based shared send queue to reduce submission latency from 1.2\,$μ$s to 400\,ns; fused write-with-atomic semantics to share connections while independently notifying each transaction; and Network Per Path Ordering to ensure data arrives before the completion flag without Sender or Receiver Ordering. We implement SURMA in RTL on a two-FPGA prototype and evaluate it with two 8-GPU NVLink servers. At 512\,B granularity, SURMA saturates attainable bandwidth with a single SM, whereas NVLink needs over 14\% of the SMs. SURMA occupies under 0.5\% of GPU area and accelerates inference on four models by 1.05--1.26$\times$ over NVLink.

0.19

PATTON: Enabling Commodity PIM for Production LLM Serving

cs.AR ·2026-09-10 ·Hangyeol Kim, Sanghyun Lee, Teokkyu Suh, Joo-Young Kim
similarity 0.65
Abstract

Processing-in-Memory (PIM) is promising for accelerating memory-bound decode attention, but attention acceleration alone is insufficient for production LLM serving, where engines dynamically allocate, populate, share, cache, and reclaim logical KV cache blocks. Supporting this lifecycle on commodity PIM requires efficient physical memory allocation, block-to-address mapping, and command generation. For the Value cache, these requirements create a fundamental conflict among GEMV efficiency, single-token write efficiency, and memory capacity: GEMV-optimized layouts scatter newly generated Value vectors across rows, making writes costly, while finer-grained memory sharing improves capacity utilization but fragments GEMV reductions. We present PATTON, a PIM runtime that integrates production LLM serving engines with commodity PIM. PATTON introduces hierarchical granule allocation: block-sized Key and Value granules map one-to-one to logical token blocks, fixing their physical placements and commands, while coarser granules group blocks for efficient GEMV execution and memory utilization. A Commit Zone stages partial Value blocks for efficient single-token writes before committing them to GEMV-optimized locations. PATTON tracks these placements to generate KV cache writes and QK-transpose/SV commands. Across attention execution and runtime-induced prefill recomputation, PATTON achieves an average 1.95x speedup and 4.83x higher energy efficiency over evaluated baselines, requires no PIM processing-unit modifications, and maintains a KV cache hit rate comparable to the native GPU KV cache in vLLM.

0.25

CVM-Store: A Secure Storage System for Confidential Virtual Machines

cs.CR ·2026-10-25 ·Thore Sommer, Masanori Misono, Dimitrios Stavrakakis, Pramod Bhatotia
top venue similarity 0.65
Abstract · accepted at NDSS'27

Stateful cloud workloads require high-performance persistent storage when running on untrusted cloud infrastructure. Confidential Virtual Machines (CVMs) promise to protect these workloads through hardware-enforced memory encryption and isolation without requiring application changes. However, they protect only data in use; persistent storage lies outside the CVM trust boundary, exposing all stateful data to the cloud provider. Extending the trust boundary to persistent storage presents inherent challenges. High-bandwidth workloads require near-raw NVMe throughput, yet the default kernel CVM I/O stacks are inadequate, and userspace I/O libraries are not designed with CVM use cases in mind. Access patterns and storage semantics vary between workloads, necessitating access-pattern-aware security abstractions rather than uniform block-level encryption. In order to mitigate against rollback, freshness is needed, but CVMs lack hardware primitives for this purpose, requiring alternative solutions. We present CVM-Store, a storage system that co-designs a high-throughput userspace I/O engine with CIF (confidentiality, integrity, freshness) data protection. The key insight is that neither authenticated encryption nor I/O is a bottleneck by itself, but their naive combination is. CVM-Store exploits this by interleaving them, eliminating redundant copies and OS/hypervisor mediation. CIF guarantees are provided as composable primitives that adapt protection to workload access patterns, from which we derive three concrete authenticated abstractions: a KVS, a log, and a filesystem. An asynchronous trusted counter service provides freshness guarantees with epoch-based batching, which enables rollback protection without dedicated hardware support. CVM-Store delivers full CIF guarantees within $\approx$1% (write) and 1-19% (read) of raw NVMe bandwidth for large objects.

0.40

Decoupling Disaggregated Memory Optimizations from Indexing: A Compiler-Runtime Approach

cs.DB ·2026-09-02 ·Xinpeng Zhao, Zeling Long, Chaichon Wongkham, Srijan Srivastava, Jiayi Liu +3 more
similarity 0.65
Abstract

Disaggregated memory (DM) decouples compute and memory into independently scalable pools, connected over a slower interconnect rather than a local bus. This decoupling is exactly what makes DM attractive--but it also means that every index must now reason explicitly about remote-memory access and its associated optimizations. State-of-the-art index designs respond to this by embedding remote-memory logic and optimizations directly into their core data structures and concurrency control mechanisms. Consequently, an optimization tuned for one index cannot be lifted and reused in another, and even the same index cannot be ported to a different DM architecture without a fresh round of redesign. This escalating, per-index, per-platform engineering burden is unsustainable as hardware and index requirements evolve. In this paper, we present Nox, a compiler-runtime framework that breaks this coupling by taking an unmodified, concurrent index as input and automatically generating its disaggregated-memory counterpart, without touching the original index logic. A compiler layer rewrites the index's LLVM IR to expose allocation, address, and pointer-dependency information that a centralized runtime uses to drive caching and address translation. Empirically, Nox-generated B+-trees, hash tables, and skip lists scale robustly on real RDMA and CXL hardware across every workload tested. They can also outperform some specialized, hand-crafted indexes and match others, especially on workloads that are closer to real-world ones. These results show that today's fastest disaggregated-memory optimizations need not stay locked inside monolithic, hand-crafted code--a compiler-runtime stack can generalize them while preserving the scalability of proven index implementations, without sacrificing it for portability.

0.46

TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

cs.DC cs.AIcs.LG ·2026-09-28 ·Jay H. Park, Hyungjun Kim, Dong Kim
similarity 0.65
Abstract

Reusable prefix key-value (KV) caches can outgrow GPU memory in large language model (LLM) serving. A memory-semantic flash hierarchy offers SSD-backed capacity with a limited fast tier, but a logical KV hit is not necessarily ready for GPU retrieval. Demand staging exposes SSD latency, whereas immediate staging can reserve fast-tier capacity long before retrieval begins. We present TempoKV, a timing-aware resource-commitment layer that separates early knowledge of reuse from the acquisition of staging resources. It records reusable-KV hits as metadata-only claims and requests commitment when the runtime-estimated time until retrieval falls to the storage-estimated time needed to make KV resident and protected against eviction. These estimates adapt to runtime progress and staging state, while commitment remains subject to available protected capacity. We implement TempoKV in vLLM and LMCache on an SSD-backed CXL memory device without changing request scheduling. Across two models and three prefix cache ratios, TempoKV reduces protected fast-tier byte-time per request by 63-91% versus immediate staging while retaining much of the serving benefit of advance staging. In a fast-tier capacity sweep, output throughput and p95 time to first token (TTFT) remain nearly unchanged as capacity decreases from 100 to 25 GiB. Compared with unmodified LMCache's Device-DAX L1 configuration, TempoKV reduces p95 TTFT by up to 48.0% and increases output throughput by up to 27.8%.

0.12

CMM-MP: A Novel Memory Pooling Architecture using Purpose-built Memory Device with CXL and Ethernet

cs.AR ·2026-10-15 ·Michael Wasef, Gongjin Sun, Edmund Au, Ranjitha Gopalakrishna, Elham Movahedian +13 more
similarity 0.65
Abstract · 12 pages. Submitted to IEEE Transactions on Computers. CMM-MP is also evaluated in "LLM KV Cache Storage using CXL Memory," IEEE Computer Ar…

CXL Memory Module - Memory Pooling (CMM-MP) introduces a novel memory pooling architecture to meet the unique requirements of emerging memory-intensive workloads. This industry-first, purpose-built memory-pooling device combines the benefits of Compute Express Link (CXL) with the scalability and cost advantages of Ethernet. Multiple CMM-MP devices together provide a flexible, composable, and disaggregated memory expansion solution. The CMM-MP enables 100s of TB capacity for GPU/CPU applications without modifications. The memory pool is exposed as a large-capacity logical aggregate memory, with all pool discovery, composition, and backend network datapath tasks fully offloaded to the device. CMM-MP supports both memory pooling and sharing usage modes. It provides an application latency of 2.7 us and up to 23 GB/s of memory throughput. Our evaluations demonstrate that CMM-MP significantly outperforms RDMA-based solutions in latency and throughput. We also evaluated real-world use cases of Valkey+YCSB and DiskANN using CMM-MP.

0.39