CS Paper Radar

HBFSim: Fast and Faithful Simulation of High-Bandwidth Flash Under Real GPU Execution

Yanpeng Hu, Yiwei Yang, Yuanwu Zhu, Yusheng Zheng, Andi Quinn, Wei Zhang

cs.AR cs.ET ·submitted 2026-09-09 06:51 UTC · v3
code/data

Abstract

High-Bandwidth Flash (HBF) places high-capacity NAND beside HBM to relieve the memory-capacity bottleneck of LLM inference, yet its system-level behavior cannot be evaluated before hardware becomes available. Cycle-level GPU simulators are too slow for production-scale models. Trace replay has a further shortcoming: it cannot capture the allocation, migration, and execution changes induced by different HBM-HBF configurations. Our key insight is that HBF need not be evaluated by simulating the GPU: only the program-visible effects of HBF need to be modeled. And only a real LLM workload running on real hardware can answer the arguments about HBF. Hence the modeled service has to be injected into that running program, and the injection must not destroy the GPU concurrency that would hide the original I/O latency. We present HBFSim, an open-source HBF simulator that executes LLM workloads on a real GPU while modeling HBF timing, thermal, and other behaviors online. HBFSim rewrites the PTX of the workload's kernels and routes accesses inside a registered address range into the HBF simulator. It supports asynchronous TMA transfers and capacities beyond physical GPU memory. HBFSim leaves the model's run unaffected across ordinary-memory, TMA, and capacity-mode tests. The delay it injects matches the delay requested to within 0.152%. We also design a coupled thermal module that puts HBF, HBM, and the GPU in one advanced package, which is important for answering how severe the hot throttling problem becomes after HBF runs for a long time. Experiments with Qwen3-30B show how package heating, HBM-HBF allocation, and shared MoE demand jointly constrain the design space of future HBF accelerators. The source code of HBFSim is available at https://github.com/SlugLab/hbfsim/.

Closest papers

Nearest neighbours by meaning across the whole index.

HBF-Sim: An Extensible HBF Simulator for Large-scale GPU Memory Systems

cs.AR ·2026-09-24 ·Yaqi Li, Jing Wang, Junfeng Wang, Long Yang, Han Yan +2 more
similarity 0.85
Abstract · 12 pages, 9 figures

High-bandwidth flash (HBF) is introduced to address the memory wall, which can co-package a dense NAND stack with the GPU, targeting the performance gap between near-accelerator bandwidth and flash density. HBF, however, is neither a large HBM nor a fast NVMe SSD. Its usable bandwidth depends on how GPU cache-line requests map onto NAND pages, how concurrency spreads across channel-affine die sets, and how media management interacts with the GPU memory pipeline. To our knowledge, existing GPU, SSD, or HBF simulators cannot faithfully model this behavior. We present HBF-Sim, an extensible, reusable, and faithful HBF simulator integrated with simulated GPUs. It closes the loop between GPU issue limits, device queuing, and NAND behavior in one end-to-end request path. HBF-Sim separates a GPU-HBF interaction controller from page-based parallel stack storage, and it models the full GPU-HBF request path. It provides an MSHR-based address mapping table that merges cache-line requests into page-based operations, as well as a page-based multi-stack flash manager for highly parallel reads and writes. Validation tests and device-level microbenchmarks expose performance bottlenecks caused by limited channel distribution and resource conflicts, offering concrete guidance for next-generation HBF architectures.

0.16

HBFSim: A Simulator for Exploring HBF in Memory Hierarchies for LLMs

cs.AR cs.ET ·2026-10-16 ·Jinheng Li, Guosheng Wang, Hongchao Du, Shangyu Wu, Qiao Li +1 more
publishedcode/datahighly cited similarity 0.79
Abstract

High Bandwidth Flash (HBF) has recently emerged as a promising technology that provides terabyte-scale accelerator-local capacity at bandwidths approaching HBM, with a potential cost-per-bit advantage over HBM. However, its NAND foundation brings long latency, out-of-place updates, and finite write endurance directly into the accelerator memory hierarchy, making its effects on performance and lifetime difficult to predict. To study this emerging design space, we present HBFSim, an LLM-aware event-driven simulator for heterogeneous HBM+HBF memory systems that preserves application-level memory semantics while modeling flash-internal behavior. HBFSim models HBM, HBF with host-managed flash translation, and conventional capacity tiers in one simulation driven by a calibrated serving frontend. Using HBFSim, we study decoding of Llama-3.1-70B and Qwen3-235B-A22B under three workloads that differ in their input and output lengths. HBF serves data that does not fit in HBM 11.0--19.0$\times$ faster than host DRAM; HBM is better used to hold the running sequences' KV cache than as a cache for HBF; and at a published peak load of Llama-3.1-70B, HBF lasts 2.0 to 6.9 years, in proportion to the stacks that share the KV writes. The artifact is available at https://github.com/FlashLabAE/HBFSim and will track public HBF device data as it becomes available.

0.52

Potential Applications of HBF in LLM Serving Systems

cs.AR ·2026-08-13 ·Yihan Yin, Yinlun Zhao, Zhixin Yun, Guanying Wu, Feng Zhu +6 more
similarity 0.78
Abstract

LLM serving is increasingly constrained by memory capacity as model weights, KV caches, and the number of served model variants continue to grow. This report examines High-Bandwidth Flash (HBF) as a capacity-oriented extension to HBM-based serving systems. We first discuss how HBF can be integrated into the GPU memory hierarchy without undermining the bandwidth expected by the compute die. We then model the system-level value of added capacity as expanded residency for read-mostly model-state objects. Under this view, HBF can improve MoE serving by enabling more expert replicas and can improve multi-model serving by reducing model loading and supporting hot-model replication. Our simulation results show that these benefits depend on preserving the HBM-resident execution path while using HBF to expand the resident set of model weights.

0.42

HBF-EventLab: A Discrete-Event Simulator for Studying Host-Device Coordination in High Bandwidth Flash

cs.AR cs.ET ·2026-11-15 ·Ziqing Mei, Yiwei Huang, Zhiyi Zeng, Yuhao Xiao, Pu Pang +3 more
code/data similarity 0.77
Abstract · 15 pages, 12 figures

High Bandwidth Flash (HBF) combines NAND flash density with high-bandwidth accelerator connectivity, offering additional memory capacity for large language model inference. However, limited hardware availability makes it difficult to evaluate how host behavior and device resource constraints jointly affect HBF request processing. We present HBF-EventLab, an open-source, transaction-level discrete-event simulator based on the Open Compute Project HBF specification. HBF-EventLab connects host request dispatch, command admission, fragmented write assembly, read dispatch, NAND execution, and completion feedback within a shared event loop. It explicitly models distinct resource lifetimes and separates configurable scheduling policies from ordering and resource constraints, enabling controlled exploration of host-device coordination. Experiments show how resource limits, address interleaving, fragment arrival patterns, and scheduling policies affect write completion and request latency. The results highlight the need to consider resource limits and scheduling policies jointly, accounting for their interactions with write assembly and read eligibility. We also integrate HBF-EventLab with real GPU execution of Qwen3-0.6B, combining profiled GPU activity durations with simulated weight-access delays to estimate time to first token and time per output token, with logits and output tokens remaining bitwise identical to those from native GPU execution. HBF-EventLab provides a configurable foundation for studying HBF mechanisms, exploring scheduling policies, and assessing how HBF access delays affect application-level timing. The source code is publicly available at https://github.com/sjtu-epcc/HBF-EventLab.

0.28

When Is High-Bandwidth Flash Cost-Effective for LLM Inference?

cs.AR ·2026-11-08 ·Zhiyu An, Yiping Li, Wan Du
similarity 0.76
Abstract

Can high-bandwidth flash (HBF) make LLM inference cheaper than adding GPUs? By storing weights closer to the GPU than SSD, HBF may reduce the number of GPUs needed for model capacity. However, this integration is only justified if it meets service targets and its cost is recovered. We simulate this integration using public component data across nine dense and mixture-of-experts (MoE) models, five workloads, two GPU platforms, and 1--100 GPUs. At fixed demand, HBF reduces the minimum GPU count to meet a service target in 98 of 468 comparisons; majority of them are paused and short sessions, 63 remain cost-effective at half a GPU price per HBF stack. For single short-chat users, HBF enables large MoEs, e.g. DeepSeek V3 and Kimi K2, to meet a relaxed service target on one RTX~4090 GPU, where conventional serving fails. At fixed GPU count, HBF improves throughput in 48\% of matched comparisons below 16 GPUs and 13\% at larger counts. Even at 100 RTX~4090-scale GPUs, cascaded HBF raises offline throughput by 16.4--26.5$\times$ for three large MoEs. We found savings and throughput gains arise in different scenarios.

0.22

FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration

cs.AR cs.AIcs.DC ·2026-08-25 ·Geraldo F. Oliveira, Arash Tavakkol, Xiangyu Zhu, Ahmet Caner Yüzügüler, Vamanan Arulchelvan +8 more
similarity 0.74
Abstract

LLM inference is increasingly constrained by accelerator memory capacity rather than compute throughput. This constraint is especially acute in single-accelerator and small-node inference systems, where limited on-package memory capacity restricts the size of deployable models. HBF is an emerging 3D-stacked NAND flash technology that provides multi-terabyte near-accelerator capacity, making it a promising capacity tier for storing LLM weights. However, existing HBF-based proposals face three adoption challenges: they (1) rely on coarse-grained static prefetching for LLM weights aiming to hide the microsecond-level read latency of the NAND flash device while maximizing HBF's read throughput, (2) expose NAND flash management tasks (e.g., refresh operations) to the accelerator-visible critical inference path, and (3) miss optimization opportunities to specialize and optimize the flash-management mechanisms to the workload behavior. Our goal is to design an efficient HBF substrate that integrates HBF as a memory-capacity tier alongside HBM while addressing these three challenges. To this end, we propose FLINT, a workload-driven HBF substrate for capacity-scalable LLM inference. FLINT introduces three mechanisms: (1) a hardware burst-buffer controller that dynamically coalesces and pipelines HBF reads aiming to utilize existing NAND flash buffers while sustaining high HBF bandwidth, (2) a phantom-plane refresh mechanism, which removes refresh from the critical inference path by moving refresh-related NAND flash operations outside the read foreground back via low-cost resource duplication, and (3) a read-only FTL, which replaces SSD-class support for arbitrary writes with a compact table that translates logical weight bursts to physical HBF locations.

0.25

SlipStream: Enabling HBF for GPU Systems by Mitigating the Latency and SRAM Overheads

cs.AR cs.DCcs.LG ·2026-11-10 ·Anish Saxena, Hritvik Taneja, Christos Kozyrakis, Aamer Jaleel
top venue similarity 0.73
Abstract · To appear in 2027 IEEE International Symposium on High-Performance Computer Architecture (HPCA)

Large language model (LLM) inference demands ever more memory capacity for larger models and batch sizes. High-Bandwidth Flash (HBF) replaces the DRAM dies of HBM with NAND flash and provides up to 16x the capacity at comparable read bandwidth. However, HBF can incur a read latency of nearly 20$μ$s, 20x that of HBM, so on-demand HBF accesses slow inference by up to 12.8x. Conventional cache-line prefetchers require 20x more in-flight tracking state to cover this latency. Tensor-streaming designs, such as H3, follow virtual-address order as opposed to kernel-demand order, and therefore need hundreds of megabytes of buffering. We propose SlipStream, a software-hardware co-designed prefetcher that turns prefetching into a demand-paced producer-consumer pipeline. In software, SlipStream profiles the repeatable weight-page demand order of data-independent LLM kernels and inserts a staging warp that replays this order with non-blocking STAGE instructions. In hardware, the staging warp acts as the producer and the work warps as consumers. Our key insight is to retire each STAGE tracking entry only when demand first accesses the page, as opposed to when the page returns from HBF. Demand thus paces staging and bounds how far staging leads consumption. SlipStream tracks 4 KB pages instead of cache lines and allocates buffer slots only when pages return, which decouples the buffer size from outstanding traffic. Across seven dense models spanning 16-800 GB, SlipStream with a 40 MB staging buffer achieves 92.4% of the performance of an ideal HBF with HBM-like latency. H3 achieves only 43.1%, even with a 320 MB buffer. The buffer sizing of SlipStream is independent of HBF latency and workload. SlipStream with 16 MB buffer, 20x smaller than that of H3, performs within 4% of SlipStream-40MB. SlipStream also supports Mixture-of-Experts models and reaches 91.2% of ideal performance (40 MB buffer).

0.55

Beyond Capacity: Scalable MoE LLM Inference via High-Bandwidth Flash with Direct GPU and HBM Paths

cs.AR ·2026-08-14 ·Seeyeon Kim, Juhyeong Jin, Joo-Young Kim
highly cited similarity 0.72
Abstract

Modern mixture-of-experts (MoE) language models increasingly strain the capacity and cost efficiency of high-bandwidth memory (HBM), as rapidly growing expert weights must be provisioned close to GPUs. High-bandwidth flash (HBF) offers substantially greater capacity, but conventional designs typically deliver HBF-resident expert weights to the GPU through HBM, leaving an additional direct GPU-HBF connection underutilized. We explore an HBF organization that simultaneously exploits two independent expert-delivery routes: a direct path that transfers expert weights from HBF to the GPU and a relay path that transfers them from HBF through the HBM base die to the GPU. Whole experts are assigned to one of the two routes, and transfers over both routes proceed concurrently, increasing aggregate expert-delivery bandwidth without replicating expert weights or introducing a shared relay bottleneck. Early expert determination identifies upcoming experts ahead of their conventional execution point, allowing HBF read latency to overlap with preceding computation, while separate management of immutable expert weights and mutable KV-cache data reduces interference between the two traffic classes. We evaluate the architecture using an event-driven continuous-batching LLM serving simulator with empirically measured GPU compute latencies. Across representative MoE workloads, concurrently utilizing the direct GPU-HBF and HBF-HBM-GPU routes consistently improves expert-delivery efficiency over designs restricted to either route alone. For a representative workload, the proposed architecture can achieve 1.94$\times$ higher throughput and 1.90$\times$ end-to-end speedup over a design that delivers all HBF-resident expert weights to the GPU through the HBM base die.

0.43