CS Paper Radar

Decoupling Logical Masks from GPU Execution for Dynamic Block-Sparse Attention

Shanghao Liu, Xiaoyun Yu, Wanting Li, Wenqi Jiang

cs.AR ·submitted 2026-09-22 08:33 UTC · v1

Abstract

Attention computation makes inference expensive in video diffusion transformers (vDiTs), which generate videos through iterative denoising. Block-sparse attention (BSA) reduces this cost by computing only blocks selected by a logical mask, which specifies attention interactions to compute. However, coupling logical block geometry to execution choices limits adaptation to varying masks and graphics processing units (GPUs), while runtime kernel specialization can incur preparation overhead that outweighs execution time savings. We present Tessera, a specialized runtime for dynamic BSA that decouples logical masks from GPU execution while preserving specified attention interactions. Its physical mapping layer retains, combines, or subdivides logical attention blocks into physical tiles suited to different attention mask shapes and GPU architectures. Its task organization layer groups and schedules tiles within GPU tasks to reuse data, expose parallelism, and overlap data movement with computation. Finally, profile-guided regime selection enables low- overhead execution plan selection through a lookup table constructed from offline profiling. We implement Tessera with specialized CUDA kernels supporting four NVIDIA GPU generations. Evaluated on 2,315 real attention masks and industrial video diffusion models, Tessera achieves up to 6.79x BSA request speedup over baseline systems in the evaluated video diffusion models.

Comment
15 pages, 10 figures, 9 tables. Preprint

Closest papers

Nearest neighbours by meaning across the whole index.

DexAttention: Distributed Sparse Attention through Rapid Online Planning and Persistent GPU Execution

cs.DC ·2026-11-17 ·Sirui Chen, Xuehai Qian
code/data similarity 0.77
Abstract · 14 pages, 7 figures, 7 tables. Code: https://github.com/CsRic/DexAttention

Sparse attention reduces computation, but distributed execution may still transfer unnecessary data and incur substantial per-operation overhead. Mask-aware assignment and scheduling can reduce these costs, but replanning for dynamic masks must be cheaper than the execution time it saves. We present DexAttention, a distributed sparse attention system that co-designs fast online planning with persistent execution. Its assigner uses GPU-accelerated, topology-aware clustering to balance computational load and reduce communication. A lightweight greedy scheduler orders coarse communication and computation operations, while a persistent kernel on each GPU enforces fine-grained dependencies and dispatches ready tasks. This division keeps detailed synchronization out of planning while enabling overlap without repeated attention launches. The framework supports static and dynamic per-head masks across multiple attention compute kernels. On a four-node, 16-GPU cluster, DexAttention achieves a $1.39\times$ geometric-mean end-to-end training speedup over the fastest applicable baseline across 11 configurations. For dynamic video attention, it achieves $2.11$--$2.37\times$ speedups over Ring Attention including planning.

0.32

KPage: Cross-Block Page Compaction for Hardware-Efficient Sparse Attention

cs.DC ·2026-10-20 ·Junwei Feng, Qiaorui Chen, Chenyu Liang, Zhen Guo, Jingyi Yang +5 more
code/data similarity 0.73
Abstract

Sparse attention reduces the computational cost of long-sequence video Transformers, but three-dimensional token layouts often leave selected blocks partially occupied, wasting physical tile capacity. We present KPage, a block-sparse attention scheme that packs valid 16-row K/V pages across logical-block boundaries into large tiles directly during loading, without materializing compacted tensors in high-bandwidth memory. KPage also routes Q blocks between 64x256 and 128x128 tiles to reduce query-side padding. Exact masking and K/V-owned page packing preserve the original sparse graph in forward and backward execution. Across five representative workloads, KPage achieves 1.35x-2.06x bare forward-kernel speedup over NVIDIA Block-Sparse Attention (BSA), with a geometric mean of 1.77x. It is near parity for fully occupied blocks (1.03x) but achieves 2.29x geometric-mean speedup on 3D layouts with partial blocks. KPage reduces physical query-key pair capacity from 334.1G to 177.5G while preserving 160.6G useful interactions. Across eight layouts from open-source SSTA, STA, and VSA systems, KPage achieves 1.13x-4.25x forward and 1.02x-2.71x forward-plus-backward speedup over the strongest exact sparse baselines. With independent Q/K partitions, KPage also achieves 1.49x speedup over the strongest exact sparse baseline and 1.57x over BSA on the SVG2 Wan 14B/720p configuration. After compliance review, we will open-source KPage and submit a pull request to NVIDIA cuDNN Frontend.

0.32

MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers

cs.CV cs.AI ·2026-10-05 ·Jiarui Chen, Zeqiang Lai, Jiangshan Wang, Ziheng Ouyang, Ye Huang +3 more
highly cited similarity 0.72
Abstract · 11 pages, 8 figures

Sparse attention is a primary approach to reducing the latency of diffusion transformers in long-sequence generation tasks, such as video and high-resolution 3D asset generation. However, existing methods can degrade generation quality and fidelity at high sparsity levels. Through controlled oracle comparisons, we trace this degradation to three sources: constraints imposed by token grouping, inaccurate interaction selection, and the attention contributions lost when tokens are discarded. Guided by this analysis, we propose Meta-Cached Sparse Attention (MC-Sparse), a training-free framework that selects individual key-value (KV) tokens while organizing similar queries into tile-aligned groups for efficient GPU execution. MC-Sparse caches metadata comprising query groups, KV indices selected using exact attention probabilities, and residuals between dense and sparse attention outputs, and reuses them across subsequent denoising steps. Across video and 3D generation models, MC-Sparse achieves higher fidelity to dense-attention outputs and larger denoising speedups than existing sparse-attention baselines, without visible quality degradation. Relative to dense attention, it delivers a $1.80\times$ denoising speedup on Minimax-H3-Base and a $2.32\times$ speedup on 3D asset generation, both with negligible quality loss.

0.45

Autotuning GPU Thread Block Sizes for Large-Scale Stencil Applications Using the OPS DSL

cs.DC ·2026-10-31 ·Kevin Antonio Nava Garcia, Istvan Zoltan Reguly
published similarity 0.72
Abstract · Accepted on HPEC Conference 2026 : https://ieee-hpec.org/proceedings

Large-scale structured-mesh stencil applications can run for hours or days on modern HPC GPU platforms, making launch-configuration choices important. OPS enables high-level stencil codes to target multiple GPU architectures, but its GPU backends rely on a default or user-selected global block/workgroup shape, even though different kernels, problem geometries, and architectures may favor different configurations. This paper presents a transparent per-kernel block/workgroup-shape autotuning extension for OPS. The extension explores valid block/workgroup shapes during execution, records the fastest configuration for each kernel, and enables precomputed replay of the selected configurations in subsequent runs. We evaluate the approach on structured-mesh proxy and academic research applications, covering 459 unique kernels across 3 NVIDIA and 2 AMD GPUs. The evaluation reports speedup distributions, representative wall-time summaries, timing variability, and input problem shape effects. Replaying the best per-kernel launch shapes improves 923 of 995 measured application/input/GPU cases over the default OPS shape. Depending on the GPU, the median speedup ranges from 1.02x to 1.18x, with individual configurations of LAPL2D, TTI3D, and OpenSBLI-based TGV solvers exceeding 1.5x. The results also show that performance sensitivity is not limited to launch shape. At fixed cell count, changing the input problem shape changes precomputed-replay runtime by 0.8-34.6% in representative cases, while kernel-level timing variability depends on the interaction between runtime contribution, input problem shape, and GPU/backend behavior. Overall, automatic per-kernel block/workgroup-shape selection improves OPS performance across GPUs, provided shape sensitivity and timing stability are interpreted together.

0.30

Mitigating SM-Level Load Imbalance in 3D Gaussian Splatting on GPUs

cs.DC cs.CVcs.GRcs.LG ·2026-11-09 ·Junha Ryu, Hoi-Jun Yoo
similarity 0.71
Abstract

3D Gaussian Splatting (3DGS) enables photorealistic, real-time rendering on commodity GPUs and is increasingly adopted in generative world models and autonomous driving simulators. However, existing optimizations rely on algorithmic changes that often require retraining, or on specialized accelerators that lack flexibility across evolving 3DGS variants. In this paper, we identify a fundamental inefficiency in GPU-based 3DGS rendering: severe SM-level load imbalance caused by uneven workloads across screen tiles. This imbalance creates a long execution tail, leaving most SMs idle while a few process computationally intensive tiles, particularly in large-scale urban scenes. To address this, we present a software-defined GPU scheduling approach that mitigates the execution tail. Our approach uses a persistent kernel to schedule heavy tiles first and selectively chunks outlier tiles across multiple CTAs. A lightweight per-frame policy determines when and how much to chunk using tile statistics already available in the rendering pipeline. Our approach requires neither retraining nor specialized hardware and can be applied orthogonally to existing optimizations across diverse 3DGS variants. Averaged per scene, it accelerates alpha blending by up to 2.93x on a large-scale urban scene, and by up to 5.6x on its most imbalanced frames, while retaining the original alpha blending formulation.

0.14

LiquidCP: Decoupling Attention Tasks from Communication and Kernel Boundaries in Context Parallelism

cs.DC ·2026-11-03 ·Hongyu Chen, Yikang Ruan, Yuhai Long, Ziyu Huang, Yiyu Luo +12 more
similarity 0.71
Abstract · 13 pages, 16 figures

Context parallelism (CP) partitions long sequences across GPUs for LLM training and inference. For variable-length training and full prefill, attention-balanced placement can leave token loads uneven, while ring execution repeats overhead at each computation and communication step. In stage-serialized decode CP (DCP), long-running tasks delay dependent work and leave substantial GPU hardware idle. We propose LiquidCP, which unifies training and inference through a common model of Transfer, Compute, and Reduction tasks with tile-level dependencies, decoupling attention tasks from communication and kernel boundaries. For training and full prefill, MegaRing incorporates BuddyRing placement to balance attention and token loads, and coalesces ready KV regions during forward attention to amortize overhead. For decoding and mixed batches, MegaDCP selects history-KV splits by jointly modeling the completion of attention and its dependent reductions. A shared persistent GPU-resident engine executes both designs. Compared with the fastest baseline per configuration on eight H100 GPUs, MegaRing achieves geometric-mean core-attention speedups of $1.23\times$ forward and $1.18\times$ backward, and accelerates a Transformer layer by $1.08$--$1.16\times$. MegaDCP achieves geometric-mean core-attention speedups of $1.20\times$ for decode-only and $1.35\times$ for mixed batches. In vLLM serving with KV prefix cache, it achieves an average time-to-first-token speedup of $1.22\times$.

0.20

From Splats to Silicon: Rethinking Computational Efficiency of 3DGS

cs.AR cs.AIcs.GR ·2026-09-05 ·Minnan Pei, Qiwei Dong, Yihan Zhou, Gang Li, Yuchen Zhu +5 more
similarity 0.71
Abstract

3D Gaussian splatting (3DGS) represents scenes with explicit primitives and supports real-time novel-view synthesis, yet its system efficiency varies substantially across scenes, viewpoints, rendering paths, and platform constraints. Existing studies pursue efficiency through representation and algorithm design, GPU runtime optimization, and architectural support, but their reported gains correspond to different points along the rendering and update paths. Connecting these indicators to end-to-end system benefit requires tracing how each optimization changes Gaussian selection, screen-space work, data movement, and stage or frame time. We therefore use a workload-centric framework to connect representation and algorithm research, GPU runtimes, and hardware architectures and to identify recurring workload patterns. We complement literature analysis with reproduced measurements and controlled GPU profiling of selected implementations, relating workload counts to stage time and memory traffic. Together, these comparisons show that system gains depend on workload reductions reaching downstream execution, granularity matching each stage, and the cost of data transfers, synchronization, and cached results, gradients, and optimizer data. Building on these findings, we discuss more consistent evaluation under rendering-quality constraints and identify key directions for future system design.

0.14

QUILT: Rethinking Sparse-Attention Prefill through Shared Query Execution

cs.LG ·2026-10-08 ·Zhenduo Zhao, Qihui Zhou, Mingcong Song, Zhiyi Chen, Chuangguan Ye +5 more
similarity 0.70
Abstract

Sparse attention reduces the cost of long-context attention, but existing kernels typically process queries independently, repeatedly loading and dequantizing KV entries shared across queries. We observe substantial overlap in the KV entries selected by neighboring queries, creating opportunities for cross-query reuse. We present QUILT, a workload-aware sparse-attention execution mechanism that jointly processes neighboring queries and reuses shared KV entries to reduce redundant memory traffic and computation. QUILT introduces Shift-and-Compare Set Decomposition (SCSD), which transforms irregular set operations into regular data-parallel primitives suitable for modern accelerators, and pipelines SCSD with attention computation to hide its overhead. Cascaded sharing captures reuse hierarchically at multiple granularities. A tile-aware execution strategy balances sharing granularity with hardware tile utilization and selectively removes low-importance query-specific tails to eliminate underutilized tiles. We evaluate QUILT on LongBench using GLM-5.3 and DeepSeek-3.2 under both tensor and sequence parallelism. Compared with the state-of-the-art sparse-attention kernel, QUILT reduces average kernel latency by up to 55.1% and processed KV data by up to 55.9%, while reducing time-to-first-token (TTFT) latency by up to 36.8% with negligible accuracy degradation.

0.17