CS Paper Radar

MTRDMA: Lightweight User-Space Control-Plane Isolation for Multi-Tenant RDMA

Hongyu Wang, Peihao Huang, Bangbo Liang, Lei Wang, Guo Chen

cs.NI ·submitted 2026-11-09 07:21 UTC · v1

Abstract

In managed multi-tenant clouds, RDMA's lack of control-plane (CP) resource isolation lets a noisy tenant exhaust shared NIC microarchitectural caches, degrading victim throughput by up to 96.4\%. Data-path schedulers leave CP resources unprotected, and hardware approaches require FPGAs or SmartNICs rather than the commodity RNICs operators already deploy. We present MTRDMA, a lightweight user-space CP isolation mechanism for commodity RNICs, applicable to managed runtimes such as containers and sandboxed processes. MTRDMA intercepts library-level CP operations to enforce hierarchical quotas while preserving the zero-copy data path, using atomic shared-memory quota state with per-process cached admission (${\sim}$0.13\,$μ$s), a hybrid garbage collector that reclaims orphaned quota within seconds, and rdma-cgroup caps with ledger--kernel reconciliation to bound and detect residual tampering. On ConnectX-6 RNICs, MTRDMA cuts victim degradation under Queue Pair (QP) cache exhaustion from 94.5\% to 0.08\%, extends protection to Memory Region (MR) cache exhaustion, and scales to 256 concurrent tenants with negligible data-plane overhead.

Comment
18pages, 17 figures

Closest papers

Nearest neighbours by meaning across the whole index.

CapBuddy: Efficient Physical Memory Management for Capability-based Resource-controlling Microkernel

cs.OS ·2026-11-09 ·Zhuang Lu, Guangtao Zhu, Shijun Zhao, Qianying Zhang, Lixin Chen +3 more
top venue similarity 0.66
Abstract · Accepted by ACM Transactions on Architecture and Code Optimization (TACO)

Capability-based resource-controlling microkernels securely multiplex hardware by exposing low-level resource abstractions guarded by capability systems. Such a design helps provide an application-specific resource management policy to achieve high performance without sacrificing security. However, the misuse of capability-protected secure abstractions can introduce performance overhead in user-level resource management, affecting the runtime performance of dynamic applications in emerging scenarios. In this paper, we focus on the secure abstraction of physical memory in microkernels with capability-based resource management delegation to investigate an appropriate design for dynamic memory management. We propose CapBuddy, an approach to building an efficient buddy system on capability-protected secure abstractions of memory, to bypass the performance overhead of the capability system by offloading functionalities that can run offline from the critical path of serving a memory request. We implemented a prototype system on the seL4 library OS, and the evaluation results show that our approach improves the performance of the seL4 user-level buddy system by 20 to 54.5 times and achieves the same order of magnitude as Linux. Further burst experiments evaluate CapBuddy's scalability and show that it maintains low median allocation latency as the number of concurrent clients increases.

0.36

KVGuard: A KV Cache Safety Checker for LLM Serving Systems

cs.CR ·2026-11-04 ·Jonguk Cheong, Myungbin Hwang, Eulgyu Kim, Byoungyoung Lee
similarity 0.65
Abstract · 15 pages, 9 figures, 5 tables

Modern LLM serving systems manage the KV cache through PagedAttention, a custom GPU allocator that partitions a single CUDA-allocated region into fixed-size physical blocks and maps each request to those blocks through a CPU-built block table. This block-based design exposes a previously unstudied class of memory safety bugs in the KV cache, including double-free, use-after-free, out-of-bounds access, uninitialized read, and permission violation, which silently corrupt outputs or break request isolation without ever crashing the system. Existing memory sanitizers cannot detect them, because they reason about CUDA-level addresses rather than KV cache blocks, requests, and block tables, and their redzone- and quarantine-based designs do not fit the scarce GPU memory budget of LLM serving. This paper presents KVGuard, the first systematic memory-safety abstraction for KV cache management in LLM serving systems. KVGuard maintains a compact shadow block for each physical KV block, recording per-request permission and per-entry initialization, and uses this metadata to perform five safety checks across the PagedAttention workflow, spanning CPU allocation, block table construction and translation, and GPU-side block table lookup and KV entry access. Each check fires at the boundary that introduced the violation, enabling early detection during development rather than reporting downstream output corruption many requests later. A prototype of KVGuard on vLLM and SGLang covers all 29 historical real-world KV cache safety bugs collected from the two systems' patch histories. KVGuard further found 8 previously unknown safety violations while reporting no false positives. It adds 3.0% and 7.4% geomean TPOT overhead on Qwen 2.5 14B on vLLM and SGLang and at most 0.11% of the KV cache in memory overhead, two orders of magnitude below NVIDIA Compute Sanitizer on the same workloads.

0.24

SURMA: Efficient Fine-Grained Communication Offload for SuperPOD

cs.AR ·2026-11-16 ·Haoran Shang, Weifeng Zhang, Mengzhu Li, Huizheng Wang, Sihan Guan +12 more
similarity 0.65
Abstract

GPU SuperPODs underpin large language model (LLM) serving, but growing bandwidth and round-trip times make inter-GPU communication consume increasing compute resources. SM-driven remote memory access occupies shared memory or registers for in-flight data and competes for SM-to-NoC bandwidth. Dedicated engines can free these resources, but must handle fine-grained transfers, down to a 576\,B KV-cache entry. NVIDIA's CopyEngine uses less than 1\% of NVLink bandwidth at 7\,KB granularity. We present SURMA, an on-die Scale-Up Remote Memory Access engine. Adapting RDMA requires overcoming slow doorbell submission, excessive per-application connection state, head-of-line blocking of completion notifications on shared connections, and costly ordering for multipath transfers. SURMA uses a credit-based shared send queue to reduce submission latency from 1.2\,$μ$s to 400\,ns; fused write-with-atomic semantics to share connections while independently notifying each transaction; and Network Per Path Ordering to ensure data arrives before the completion flag without Sender or Receiver Ordering. We implement SURMA in RTL on a two-FPGA prototype and evaluate it with two 8-GPU NVLink servers. At 512\,B granularity, SURMA saturates attainable bandwidth with a single SM, whereas NVLink needs over 14\% of the SMs. SURMA occupies under 0.5\% of GPU area and accelerates inference on four models by 1.05--1.26$\times$ over NVLink.

0.19

GNoF: A Low-SM-Overhead GPU-Native NVMe-over-Fabrics Completion Path for Data-Intensive GPU Workloads

cs.DC cs.AR ·2026-11-16 ·Lei Tian, Wenhao Gu
similarity 0.65
Abstract · 9 pages, 5 figures, 4 tables

GPUs are the main compute engine for data-intensive workloads whose datasets exceed device memory and reside on remote flash. The conventional host CPU orchestrates every NVMe-over-Fabrics (NVMe-oF) transaction and stages data through host memory, costing high CPU usage, repeated CPU-GPU synchronization, and redundant data movement. GPU-initiated storage removes the CPU from the I/O path, but existing designs leave two problems: synchronous lock-step completion pins streaming multiprocessors (SMs) in spin-polling, wasteful over a 10-100 microsecond RDMA round trip, and current GPU-native remote systems focus on target-side firmware while using per-command doorbells and periodic CQ polling. We present GNoF, a GPU-native NVMe-oF/RDMA data path where GPU threads build command capsules, post send work-queue elements, ring doorbells, and reap completions entirely on the device; the CPU only handles one-time connection setup, the admin queue, and a keep-alive thread. Three mechanisms target long fabric round trips: a doorbell mover that coalesces a burst's doorbells into one MMIO ring; a batched CQ poller with a lock-free fast peek and monotonic completion counters that cannot lose a wakeup; and a set-associative, reference-pinned GPU page cache with split fetch/wait pipelining and orphan-fill takeover. GNoF needs no target or firmware change and runs against an unmodified SPDK target. On a 200 Gb/s InfiniBand fabric, GNoF reaches 1.77 million 4 KB IOPS per client at 0.37% host CPU and about 1% SM occupancy, versus 40% CPU for kernel nvme-rdma and about 100% SM occupancy for lock-step polling. Against three existing baselines (kernel nvme-rdma, BaM over a BlueField-3 DPU, and BaM on a local SSD) under identical workloads, we show 10-22x gains on selective column I/O, CPU-burn immunity, and the regime where bulk staging is faster.

0.18

SBA: A Practical Memory Performance Denial-of-Service Attack

cs.AR ·2026-11-12 ·Amin Mamandipoor, Connor Rudy Sullivan, Cole Ridge Strickler, Mohammad Alian, Heechul Yun
top venue similarity 0.64
Abstract · Accepted for publication at the 3rd Microarchitecture Security Conference (uASC 2027)

Modern cloud platforms use memory bandwidth regulation to isolate co-located workloads, implicitly assuming that high aggregate bandwidth consumption implies high interference. We show that this assumption is fundamentally flawed. We present the Single-Bank Attack (SBA), a practical memory denial-of-service attack that induces extreme cross-core slowdowns---over three orders of magnitude on memory-intensive victims (e.g., LLM decode)---while consuming less than 1 GB/s of memory bandwidth. SBA exploits the gap between peak and worst-case DRAM throughput by concentrating memory write operations on a single bank, forcing persistent row conflicts and frequent write draining, thereby collapsing the memory service rate. Across Intel and AMD systems, including DDR5- and HBM2E-based platforms, SBA consistently causes orders-of-magnitude slowdowns under realistic multi-tenant settings, revealing a fundamental limitation of bandwidth-centric memory isolation. We further show that this localized contention propagates system-wide: single-bank congestion saturates memory controller queues and induces back-pressure into shared cache and coherence structures, ultimately throttling memory request injection across cores. Consequently, severe interference can arise even at low aggregate bandwidth, evading existing bandwidth-based QoS mechanisms such as Intel RDT MBA. To address this limitation, we propose IQ-MemGuard, a lightweight, software-based memory bandwidth regulator that detects sustained contention using microarchitectural signals and selectively throttles pathological request injection. Our implementation on an Intel platform effectively mitigates SBA, whereas Intel's hardware QoS mechanism fails completely.

0.46

The Storage View Of KV Cache: Characterizing Cache Behavior Across LLM Workloads

cs.OS ·2026-11-06 ·Talia Q. Chen, Daniel Waddington, Narangerelt Batsoyol, Vijay Chidambaram
similarity 0.64
Abstract

Modern LLM serving increasingly extends KV caches beyond GPU memory into host DRAM, SSDs, and remote storage, yet these stores remain largely governed by workload-agnostic policies such as LRU and fixed TTLs. We ask what execution context is needed by storage to make retention, reclamation, and placement decisions across heterogeneous workloads. We characterize KV-cache reuse, lifetime, and footprint at block granularity across 11 production and public traces spanning chat, API serving, RAG, coding agents, tool-using agents, and multi-lineage workflows. We examine reuse sources, recurrence timing and frequency, useful lifetimes, and storage consumption. Across workloads, KV state takes two recurring forms. Single-session state is reused within one execution: its footprint grows as sessions deepen and accumulates across concurrent sessions; it can remain useful through long idle periods but loses value at session completion. Multi-session state recurs across sessions of the same workload, outlives individual sessions, and spans hot prefixes and a large infrequent tail. While prior systems often exploit application-, agent-, or workflow-specific knowledge, our results suggest that three generic signals (a workload identifier, a session identifier, and session completion) expose much of the structure needed for KV-storage management. Trace-driven comparisons show that workload-specific recurrence estimates retain up to 22% more within-session hits and reduce dead state by up to 40%; a session completion signal eliminates the fundamental idle-time reclamation tradeoff for single-session state; and session-aware reuse classification identifies shared state 4-700x more selectively than request-level access counts. These results motivate a small cross-layer interface for workload-aware KV storage without application-specific semantics.

0.14

Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

cs.DC cs.LG ·2026-09-16 ·Dev Bali, Soujanya Ponnapalli, Yichuan Wang, Natacha Crooks, Scott Shenker +1 more
similarity 0.64
Abstract

LLM serving is typically offered as a shared, multi-tenant service, where high-demand workloads from one client can cause latency SLO violations for others. Existing solutions for performance isolation equalize client throughput in the long run, for example through queueing and batching fairness. However, these approaches do not provide latency isolation guarantees; as a result, well-behaved clients can still experience significant degradation to their token-level latencies. In this paper, we present FairInference, which provides the novel δ-token fairness guarantee: for a well-behaved client, if a token is generated in d time units in isolation, it will be generated within d + δ time units in multi-tenant execution, providing strong latency isolation guarantees for LLM serving. To achieve this, FairInference addresses a key challenge of LLM serving: bounding delays from sharing GPU resources without support for fine-grained scheduling or resource allocation. In FairInference, the scheduler enforces per-token deadlines, while bounding the delays from GPU compute sharing and accounting for the additional delays introduced by the shared KV caching in GPU memory. We show that FairInference effectively bounds token-level latency spikes for well-behaved clients and improves overall throughput compared to state-of-the-art LLM serving systems.

0.46

CXLResume: Retained-State Checkpoint and Restore for In-Memory Services

cs.OS ·2026-10-20 ·Haonan Li, Zhaoxiang Huang, Zhihao Zhang, Yiming Zhang
similarity 0.64
Abstract · 10 pages

Checkpointing and restoring an in-memory service typically require writing its memory pages to checkpoint images and refilling them at the destination. When application state already resides in shared memory that remains accessible after failover, copying those pages into and out of checkpoint images is unnecessary. We present CXLResume, a prototype that retains the service's main heap objects in shared memory while checkpointing the remaining execution state and mapping metadata. Checkpointing preserves the shared heap at the same freeze point as the serialized process state. During recovery, CXLResume remaps the same shared memory at its original virtual addresses before restoring execution, preserving direct pointers within and across shared and private memory. We evaluate CXLResume with Redis and Memcached using QEMU CXL functional emulation. At 6 GiB, CXLResume reduces combined checkpoint and restore time from 15.015 s to 0.242 s for Redis and from 19.770 s to 0.276 s for Memcached in the evaluated deployments. These results show that retaining shared application state removes large application heaps from the checkpoint data path and substantially reduces checkpoint and restore latency.

0.20