CS Paper Radar

Deterministic Regime Switching and Feasibility Inversion in Dynamic Tensor Rematerialization

Mahesh Reddy Pagadala

cs.LG cs.DC ·submitted 2026-09-25 13:26 UTC · v1
positioncode/data

Abstract

We report fine-grained, deterministic instability in Dynamic Tensor Rematerialization (DTR), an online eviction policy for memory-constrained DNN training, measured on the reference DTR simulator (simrd) using public execution traces. On an LSTM trace, memory budgets differing by 0.10% of unconstrained peak memory select fast and slow execution regimes whose overheads differ by as much as 7.3x; the slow regime is driven by broadly repeated re-eviction of the same storages (evictions per storage rise from 1.33 to 8.27 while the set of distinct evicted storages is essentially unchanged: 5,233 vs 5,236, with the two sets overlapping at Jaccard 0.999). On a ResNet-32 trace, a fine budget sweep reveals a deterministic feasibility inversion: the run is feasible at ratio 0.101, infeasible (OOM) across 0.102-0.106, and feasible again from 0.107. We trace the immediate cause of the OOM to a fully pinned recursive rematerialization frontier that exceeds the budget after every evictable tensor has been evicted. Ablations using the DTR authors' own variants implicate the joint size-staleness scoring term in the observed LSTM instability. We argue these are at least two distinct budget-sensitive pathologies rather than one mechanism, and we separate what is demonstrated from what remains hypothesised. All results concern the reference simulator; reproduction in a production runtime is future work. Code, instrumentation, and raw results accompany this preprint.

Comment
6 pages, 5 tables, 2 figures. Code and data: https://github.com/lonewolf15116/dtr-regime-switching

Closest papers

Nearest neighbours by meaning across the whole index.

When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization

cs.LG cs.DC ·2026-10-09 ·Mahesh Reddy Pagadala
code/datahot direction similarity 0.82
Abstract · 8 pages, 2 figures. Simulator study. Code and execution records: https://github.com/lonewolf15116/dtr-stability

Dynamic Tensor Rematerialization (DTR) trains neural networks under a memory budget by evicting tensors and recomputing them on demand. A natural expectation is that a budget large enough to complete training stays sufficient when it is increased. Using DTR's reference simulator, simrd, to replay the public DTR training traces, we show that this expectation fails. We call the exceptions feasibility holes: memory budgets at which an eviction policy runs out of memory although both a smaller and a larger budget complete, a form of non-monotone feasibility. Under a written sweep protocol with dated amendments, a 0.01-step budget grid shows no such hole for DTR on any of six confirmatory traces, although unresolved timeouts prevent it from establishing monotonicity. Finer sampling finds holes on InceptionV4 (three regions) and U-Net (two regions), in addition to one on the ResNet-32 development trace. Every hole is reproduced by three repeats and by unmodified simrd. For one InceptionV4 pair (0.2343 versus 0.235 of peak memory) we trace the failure to individual allocations. The larger budget avoids one eviction at an operator where the smaller budget was 173,479 bytes short. The extra data it carries forward forces a later eviction of a tensor needed three operators afterwards, and rebuilding that tensor triggers a recomputation cascade that cannot fit. Keeping that one tensor resident under the same budget prevents the failure. The two simple score changes we tested do not remove holes across traces; both add failing budgets on some traces. All results are from simulation, with no allocator model; behaviour on GPU hardware is untested.

0.37

TapeSAC: Automated, Pareto-Optimal Fine-Grained Memory Management for Large Model Training

cs.DC ·2026-11-03 ·Kaihui Ren, Yifan Zhao, Zejian Wang, Yingtong Xiong, Xiaoling Zhou +9 more
similarity 0.62
Abstract

Selective activation checkpointing (SAC) decides which activations to save for the backward pass and which to recompute: each choice is a point in the plane of peak memory and recomputation time, and the goal is that plane's Pareto frontier. Current practice falls short of this goal. Hand-tuned policies fix a single point that does not adapt when the model, batch or kernels change; PyTorch's SAC reruns each block's forward pass in program order, recomputing values the backward pass never reads; and the memory budget of torch.compile treats recomputation as free or forbidden and selects saved tensors by bytes alone, an assumption that fails for the custom kernels that produce many of a large model's activations. We present TapeSAC, which automates SAC in eager PyTorch without editing the model's forward code. TapeSAC records each block's forward pass at the dispatcher and, during backward, replays only the producers of the values autograd consumes, so saving a tensor eliminates every producer needed only for it. A planner selects the saved tensors from measured recomputation costs under a single limit on step-peak memory. On Qwen3.8-27B and Qwen3.6-35B-A3B, replay alone reduces step time by 3.0% and 1.7% versus full checkpointing at no greater memory cost; memory-aware planning raises these gains to 6.3% and 13.9% under a 90%-of-device-memory limit. The evaluated hand-tuned policies fall short of this memory-recomputation frontier and, on Qwen3.6-35B-A3B, run out of memory at larger batches. Over 500 steps on real data, TapeSAC reduces training time by 8-11%; on Qwen3.5-397B, it is 2.7% faster than a model-specific hand-tuned policy while using 14% less memory.

0.12

Elasticy: A Runtime-Reconfigurable Architecture for Dynamic Streams

cs.AR ·2026-11-13 ·Konstantin J Hossfeld, Nathan Sobotka, Gina Sohn, Tun-Yu Chang, Kunle Olukotun
similarity 0.60
Abstract · 14 pages, 11 figures, 7 tables

Reconfigurable Dataflow Accelerators (RDAs) have demonstrated high performance and energy efficiency for dense, regular tensor applications. However, conventional RDAs with static configurations show dramatically reduced performance and efficiency on irregular tensor workloads with data-dependent, non-uniform, dynamic tensor shapes. We propose Elasticy, a dynamically reconfigurable dataflow accelerator that employs streaming and tile abstractions to minimize the overhead of dynamic support. By encoding dynamic shapes into streams using rank-variable expressions, Elasticy exploits data-dependent tensor shape dynamism to minimize off-chip data traffic and wasted compute cycles. Elasticy employs stream-embedded reconfiguration information to trigger fine-grained runtime scheduling decisions. Elasticy also introduces a novel distributed linked streaming memory (DLSM) that more effectively uses on-chip memory to minimize off-chip data traffic and support dynamically sized tensors. Both reconfiguration and shape dynamism incur minimal overhead in Elasticy: reconfiguration occurs infrequently at the granularity of a stream of tiles, while rank-variable expressions enable lightweight handling of dynamic dimensions. On a Llama 3-8B decode layer, Elasticy achieves 92.6% of its roofline performance, compared to 69.7% on an H100. On evaluated GEMMs, the DLSM reduces total off-chip traffic by up to 61.4% relative to the best evaluated cache configuration, with no additional burden of programmer-managed memory. Elasticy's distributed runtime scheduler overlaps dependent kernels, speeding up kernel pairs by up to 1.42x. Programs can also run in both space and time: each phase of a layer's graph is pipelined spatially, while phases fold onto the same fabric over time. Compared to a traditional spatial accelerator, Elasticy achieves a 6.34x-31.67x improved area delay product on full layers.

0.26

Zero-I/O Fault Recovery for Sharded Deep Learning via Dynamic Framework Dependency Rebinding

cs.DC ·2026-09-16 ·Genlang Chen, Junyi Zhu
similarity 0.58
Abstract · 8 pages, 4 figures, 10 tables

Distributed model training at scale is frequently interrupted by transient network failures, conventionally forcing cluster managers to abort all processes and roll back to the latest checkpoint. While periodic checkpointing provides durability, frequent snapshotting introduces severe storage backpressure: our measurements on a 1.216B-parameter decoder reveal that per-update asynchronous checkpointing incurs up to a +656.7% latency overhead, consumes 32.5 GiB of host memory, and generates 3.39 TB/hour of storage traffic. To eliminate this overhead, we present AccelPact, a parallel runtime enabling zero-I/O in-memory fault recovery for sharded distributed training. When communication fails at a committed optimizer step, device memory remains quiescent and uncorrupted, yet continuation fails because frameworks like PyTorch FSDP cache internal communication handles across module wrappers and parameter hierarchies. AccelPact resolves this dependency invalidation via a non-invasive reference-rebinding mechanism coordinated by an out-of-band Gloo consensus protocol. On 16 NVIDIA RTX 5880 GPUs training full-parameter Mistral-7B, AccelPact eliminates checkpoint replay, yielding a 1.197x whole-run goodput improvement over cold restart and 1.194x over NVRx checkpoint restoration at checkpoint age 5, rising to 1.698x at age 18. Across ten successive fault injections, all 16 ranks maintain bit-identical parameter state with zero numerical drift. Across 4-to-16 GPU cluster topologies, reference rebinding executes in constant time (0.493-0.518 ms). Operating directly on native communicator instances, AccelPact requires zero application-code modifications and avoids compiler graph breaks under torch.compile, providing an efficient foundation for resilient deep learning.

0.27

Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading

cs.OS cs.LG ·2026-10-29 ·Zhiyuan Fang, Dingyang Li, Jianfeng Zhong, Bin Wang, Weixi Zhu +4 more
similarity 0.58
Abstract

Modern LLMs strain device memory through large model weights and attention states, particularly in resource-constrained scenarios. Existing systems separately provision memory for model weights and attention states, imposing fixed fixed boundaries that cannot adapt to the changing memory demands of workloads. Under heavy workloads, insufficient state capacity limits request concurrency and reduces throughput; under light workloads, idle memory in one region cannot be reallocated to another, wasting device memory. Meanwhile, modern LLM architectures introduce heterogeneous tensors with diverse layouts and lifetimes, further complicating memory management. These problems motivate unified memory management for heterogeneous tensors. However, a shared pool alone does not make memory directly reusable across tensors with different allocation granularities and execution layouts. This paper presents Rimuru, a modern LLM serving system that unifies memory management through elastic expert offloading. Rimuru establishes a shared page pool to enables elastic memory sharing across heterogeneous tensors. To accommodate different allocation granularities, Rimuru introduces memory views that allow each tensor type to interpret the shared memory space using its own page size and page table. Furthermore, Rimuru combines huge-page alignment with a multi-level page allocator that splits and merges pages, enabling cross-type capacity reuse. A page-table-aware fused MoE kernel directly accesses non-contiguous expert weights, avoiding copy overhead. Building on these mechanisms, Rimuru employs a workload-aware adjuster that dynamically reallocates pages across tensor types according to runtime demands. Implemented on top of vLLM, Rimuru remains compatible with modern LLMs. Evaluations show that Rimuru achieves up to 19.03x baseline throughput and reduces TTFT by 98.1% on average.

0.18

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

cs.LG ·2026-10-15 ·Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu
similarity 0.58
Abstract

Reducing memory use in large language model (LLM) training is an ongoing challenge. Activation checkpointing saves memory by recomputation, while lossy compression changes the stored activations. The reemergence of sparse $\relu$ and $\relu^2$ activations creates a third opportunity: losslessly compressing the activation cache itself. We present BitSparse, a GPU-resident lossless compression format for sparse activations. Compression and reconstruction run in custom Triton kernels fused into an autograd function, shrinking the activation cache while recovering it exactly for the backward pass. Bfloat16 (BF16) and FP8 activations are supported. Optional sign-bit packing further increases compression up to close to the theoretical limit. We analyze its compression ratio and compare it to the theoretical optimum. BitSparse lowers per-token VRAM by 31\% and 30\% Modded-NanoGPT and Nemotron-H respectively, with overheads of no more than 1\% at practical sequence lengths. Additionally, a new preallocated buffer system avoids host synchronization during training when variable size tensors are allocated.

0.14

PlaneZip: Lossless Bit-Plane Compression of Training Activations Offloaded over PCIe

cs.DC ·2026-10-18 ·Zhufan Gu, Qizhe Tang, Yuxin Wang, Xun Zhou, Hui Feng
similarity 0.58
Abstract

Training under a device-memory budget relies on tensor rematerialization: an evicted activation is either swapped to host memory or dropped and recomputed. Because GPU compute has outgrown PCIe bandwidth, cost-model planners such as MegTaiChi, which charge a swap its full transfer time, almost never swap, and swaps that fill idle link time carry only a small fraction of the saved activations. Saved activations, however, are homogeneously typed arrays with bit-level redundancy beyond zeros, which lightweight bit-plane compressors can remove at line rate. We present PlaneZip, which compresses swapped tensors on the copy path and plans around the compression. Its codec picks, for every 128-byte block, the smallest of zero-value, delta bit-plane, bit-plane, and a new zero-masked bit-plane encoding. A per-tensor predictor reuses the ratio of the previous iteration, and the planner budgets link time in compressed transfer times. On activations captured from five models trained on real data, the codec compresses BF16 activations by 1.71x and FP32 activations by 1.39x (geometric means), against 1.18x and 1.20x for zero-value compression. In trace-driven simulation of a PCIe 3.0 consumer GPU and a PCIe 4.0 datacenter GPU at half the activation memory, PlaneZip cuts the rematerialization overhead by 47% and 31% relative to a bandwidth-aware planner without compression, and by 73% and 54% relative to MegTaiChi's cost model, with an estimated engine area of 0.036 square millimeters.

0.15

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks

cs.OS ·2026-08-12 ·Yuan Si, Yufeng Lin, Daming Li, Jialu Zhang
similarity 0.58
Abstract

Open-weight models can occupy a middle capacity regime: active weights fit in DRAM as cached file pages, but a second framework-owned copy does not fit or must be refilled as layers run, so low-batch decode rereads the weights every token. On integrated and coherent-memory systems those file pages are already GPU-readable, yet ordinary loading paths copy them into framework allocations before use. We call this copy the ingestion tax. We present file-backed weight adoption: a framework-independent producer maps each tensor with MAP_SHARED, wraps the pages as a no-copy GPU buffer, and exports a DLPack capsule that PyTorch or MLX imports as ordinary storage. Zero-copy import alone is insufficient: the implementation must also keep activations accelerator-resident and establish ordering on the GPU; an adopter that omits both runs a dense decode stage 2.3x slower than stock in the live system. With both in place, adoption removes the tax: the public route reaches 516 GB/s versus 53-82 for the default constructors, matches the identical kernel over resident storage ([-0.66%, +0.48%], paired), and is within 1.3% of a resident control on a matched Qwen2.5-72B (7.14 vs. 7.23 tok/s). At the same throughput, the weights remain clean, shared, evictable file pages: N processes decode from one mapped copy where resident loading creates N copies (at capacity, 5.5 vs. 0.08 tok/s), and a 65 GB checkpoint cuts time to first token by 6.4x versus stock loading. In Kimi K3, a 2.8T-parameter MoE, the dense int8 spine stage falls from 2.62 to 0.35 s per token (7.5x; 3.8x from storage alone). The same mechanism improves llama.cpp by 1.21x at half the footprint on an AMD APU, falls inside the 5% selection band of overlapped streaming on a capacity-exceeding GH200 workload, and is 39x slower across PCIe. The deployment rule follows memory topology: adopt file pages only where the GPU can already read them.

0.36