CS Paper Radar

StarMPI-SVD: Distributed-Memory Parallel Star-M Tensor SVD

Naman Pesricha, Grey Ballard, Srinivas Eswar, Md Taufique Hussain, Vishwas Rao, Phani Motamarri

cs.DC ·submitted 2026-11-19 17:18 UTC · v1

Abstract

The star-M tensor algebra generalizes the optimal compression guarantees of the matrix singular value decomposition (SVD) to higher-order tensors. The t-SVDM-I decomposition is computed using a set of invertible transformations and slice-wise truncated SVDs with a prescribed rank, and its adaptive, multirank variant t-SVDM-II can determine slice-wise ranks to satisfy a prescribed error tolerance. Existing implementations of these algorithms are limited to shared-memory environments, restricting their applicability to modern large-scale datasets. In this work, we present the first distributed-memory implementations of t-SVDM-I and t-SVDM-II, which are generalized to N-way tensors. To address computational and communication bottlenecks, we design asynchronous, non-blocking communication strategies for efficient data redistribution to obtain highly efficient key computational kernels. Strong-scaling experiments demonstrate the efficiency and scalability of our approach (at least 70% parallel efficiency on up to 128 nodes, or 6144 cores), and we apply the star-M compression techniques to a set of application datasets of size up to 768 GB.

Closest papers

Nearest neighbours by meaning across the whole index.

Low-Latency Coded Tensor--Matrix Multiplication for Distributed Signal Processing Systems

eess.SP cs.IT ·2026-09-24 ·Ahmad Tanha, Ali Khalesi
similarity 0.69
Abstract · To appear in Proceedings, IEEE International Workshop on Signal Processing Systems (SiPS), November 2nd-4th, 2026, Bordeaux, France

Large-scale signal processing systems, including aeronautical and aerospace platforms, increasingly rely on tensor operations, whose distributed execution is constrained by communication, memory, and latency bottlenecks. In this paper, we propose a tensor-aware coded computation framework for distributed mode-1 tensor--matrix multiplication that operates directly on tensor subtensors, avoiding explicit unfolding and preserving multi-dimensional structure. For a fixed partitioning configuration, the proposed tensor-PolyDot scheme achieves the same recovery threshold, communication cost, worker-side computation, and memory requirements as conventional matrix-based PolyDot schemes. However, by leveraging multivariate encoding, it replaces high-degree univariate interpolation at the fusion node with structured low-dimensional decoding. This leads to a substantial reduction in decoding complexity and latency, without affecting any other system metric. Additionally, the tensor-based approach improves memory locality and enables parallel decoding across tensor modes, making it well-suited to modern hardware architectures. Numerical results show decoding speedups proportional to the tensor partitioning factor, reaching up to an order-of-magnitude improvement in practical settings. These advantages make the proposed framework particularly relevant to latency-critical aeronautical and aerospace distributed signal processing systems.

0.44

Parallel Sparse Outputs for Tensor Compilers

cs.DC ·2026-10-13 ·Adrian Gushin, Pau Sum, Willow Ahrens
similarity 0.68
Abstract

Sparse tensor compilers are commonly used high-performance tools that optimize tensor algebra programs over notoriously complex sparse data structures. Despite the ever-increasing size and scale of sparse datasets, there exists astonishingly little research on developing a parallel sparse tensor compiler. Existing approaches fail to parallelize or make assumptions about access patterns or output formats that limit generality. This paper presents Birdseed, the first tensor compiler to support all loop orderings over any standard sparse format in parallel. More specifically, Birdseed supports parallel sparse scatters and directly writes into list, bytemap, hash, and dense levels of fibertree sparse formats. Our approach exhibits asymptotic memory and time improvements over all previous work. We additionally demonstrate empirical speedups on challenging kernels to parallelize, such as histogram construction and SpMSpV.

0.23

BasisKV: Value Latents for Communication and Key Routing

cs.AI ·2026-11-02 ·Lixun Zhang, Zishan Shao, Kangning Cui, Yu Liang, Fan Yang +1 more
code/data similarity 0.63
Abstract

Long-context large language model (LLM) inference is increasingly constrained by the cost of maintaining an ever-growing KV-cache. Low-rank K/V projections provide a promising way to compress the KV-cache with minimal quality loss, but prior work has largely optimized KV-cache compression in isolation, leaving distributed inference, especially with tensor parallelism where attention computation is partitioned across multiple GPUs, largely underexplored. We make two observations: (i) tensor-parallel communication is an important cost when deploying low-rank KV-cache compression in distributed inference; and (ii) Value compression can be associated with the output projection, allowing the same low-rank Value representation to reduce both Value-cache width and tensor-parallel communication. Based on these observations, we propose BasisKV, a low-rank KV-cache compression method for communication-intensive, long-context LLM inference. BasisKV jointly optimizes the low-rank factors across per-GPU attention partitions against the complete attention output, rather than fitting each partition in isolation. The compressed Value cache is further reused to help identify which Key pages should be accessed: it predicts part of the Key information needed for routing, while a Key-specific residual stores what cannot be recovered from Value. The original Keys are then retrieved only for the selected pages to compute exact QK scores. Across multiple LLM families and system settings, BasisKV achieves a better quality--efficiency tradeoff than existing low-rank baselines, reaching up to 1.42x end-to-end speedup under 8-GPU tensor-parallel inference. Our code is available at https://github.com/alexz949/BasisKV

0.29

AscSpMM: Efficient SpMM on Ascend NPUs via Pipelined Vector-Cube Collaboration

cs.AR ·2026-11-09 ·Fei Li, Haoxin Li, Kehong Liu, Yanzi Zhou, Weiguo Wu +3 more
code/data similarity 0.62
Abstract · 13 pages, 11 figures

Neural processing units (NPUs) provide high compute density and energy efficiency through specialized matrix engines and software-managed on-chip memory. However, their libraries primarily target dense computation. Sparse matrix-dense matrix multiplication (SpMM), a key kernel in graph analytics, sparse neural networks, and scientific computing, requires index-driven operand selection, indirect addressing, scattered memory accesses, and load balancing. These requirements conflict with the fixed shapes and contiguous layouts preferred by NPU matrix engines. GPU Tensor Core SpMM methods cannot be directly transferred to NPUs with decoupled vector and matrix engines. We present AscSpMM, which separates irregular operand preparation from regular matrix computation through a parameterized Vector-CSR representation. The vector engine decodes indices and gathers and reorganizes dense operands, while the Cube engine performs matrix multiply-accumulate operations. A double-slot ring buffer enables dependency-aware asynchronous pipelining, while operand reuse and runtime-adaptive tiling reduce cross-engine stalls. On an Ascend 910B3, AscSpMM achieves geometric-mean speedups of 2.36x over GEMM, 19.92x over an existing SpMM implementation, and 5.82x over cuSPARSE on an A100. It reduces matrix-input storage and peak HBM footprint by 48.24% and 22.98% on average, while remaining competitive with specialized GPU Tensor Core SpMM kernels. The code is available at https://github.com/LfLab-AI/Ascend-NPU-SpMM.

0.35

Lightweight Representation Learning for Low Tucker Rank Tensors

cs.LG cs.AI ·2026-10-31 ·Hiroki Hasegawa, Yukihiko Okada
benchmark similarity 0.62
Abstract · 26 pages

Low Tucker rank tensors provide compact representations of multidimensional dependencies, but standard learning pipelines can incur substantial computational and memory overhead from redundant intermediate quantities. We reduce this overhead without modifying the underlying representation objective or parameterization. Our approach combines three complementary components: (i) sequential extraction of only task-required spectral components, avoiding unnecessary full decompositions; (ii) factor-wise multilinear contractions without materializing large Kronecker operators; and (iii) compact latent-state updates without storing full ambient tensor iterates. These components preserve the algebraic operations and learning trajectories of the original algorithms while reducing their computational and memory footprints. Across five low Tucker rank tensor estimators, our implementations match the original trajectories to numerical precision (relative discrepancies of $10^{-16}$ to $10^{-13}$), while reducing per-iteration runtime by up to $3.12\times$ and resident memory by up to $616\times$ ($99.8\%$) on the main order-3 benchmark at $n=100$, with memory reductions exceeding $900\times$ across the full evaluation suite. These results show that substantial computational redundancy can be eliminated from Tucker representation learning without changing the underlying model or learned representation.

0.32

Dual-Core Tensor Wheel Decomposition with Structure-Cached Acceleration for Tensor Recovery

cs.LG ·2026-11-10 ·Wenchao Xie, Qingsong Wang
similarity 0.62
Abstract

Tensor wheel decomposition captures complex correlations in high-dimensional data through interactions between outer-ring factors and a high-order central core. However, updating these components requires the explicit construction and manipulation of large-scale matrices, which limits computational efficiency and incurs particularly high computational and memory costs for the central-core update. To address these limitations, we propose dual-core tensor wheel decomposition (DCTW), which factorizes the central core into two lower-order cores linked by a shared latent bond. Each core is coupled to a different group of outer-ring factors. We further propose a structure-cached acceleration strategy that avoids explicit large-scale matrix multi- plications and caches intermediate tensors generated during tensor network contractions for reuse across block updates. This reuse reduces redundant computation and avoids the explicit construction of large matrices. Complexity analysis and experiments show that the strategy based on structural caching accelerates optimization compared with explicit and implicit up- date schemes while limiting the increase in peak memory as the ranks grow. Experiments on color video completion and robust denoising show that DCTW matches or outperforms representative tensor network methods in reconstruction quality, computational efficiency, and scalability.

0.22

NUMA-Torch: Compiler-Driven NUMA-Aware LLM Prefill on Multi-Socket CPUs

cs.DC cs.PFcs.PL ·2026-11-04 ·Thisara Rathnayaka Weerakoon, Ishan Dewpura, Niman Ransindu, Shavin Anjitha, Tharindu R. Patabandi +1 more
published similarity 0.62
Abstract · 11 pages, 8 figures. Accepted at the 2026 IEEE 33rd International Conference on High Performance Computing, Data, and Analytics (HiPC), Beng…

For sufficiently large transformer-based Large Language Models (LLMs), whose parameters cannot be retained in on-chip caches, prefill throughput on multi-socket CPUs is often limited by sustainable DRAM bandwidth rather than peak floating-point throughput. In the standard PyTorch compilation (i.e., TorchInductor) path, model weights are allocated without considering the Non-Uniform Memory Access (NUMA) topology: specifically, the affinity between memory banks and compute cores is disregarded. This causes a substantial fraction of weight accesses to traverse the inter-socket interconnect, severely degrading the effective memory bandwidth available to each socket. We address this bottleneck through a compiler-level solution that makes TorchInductor NUMA-aware. Treating each NUMA socket as an independent tensor-parallel partition, we introduce NUMA-aware Tensor Parallelism (NUMA-TP). This approach transparently shards weights across sockets and schedules compute kernels such that thread pools and memory allocations remain strictly local to their respective sockets using CPU affinity. We evaluate the approach on GPT-2 (117M, 345M, 762M) and BERT (Small, Large), which exercise standard multi-head attention and MLP, as well as Gemma 3 270M, which combines Multi-Query Attention with a GeGLU-gated MLP. By strictly enforcing data locality, our approach successfully mitigates cross-socket traffic, achieving prefill speedups of up to 2.63x on a dual-socket Haswell platform, up to 1.87x on a dual-socket Cascade Lake platform, and up to 2.27x on a four-socket Haswell-EX platform.

0.37

SparseConflicts: Handling Conflicting Data Layouts in Sparse Tensor Contractions

cs.PL ·2026-09-06 ·Adhitha Dias, Kirshanthan Sundararajah, Artem Pelenitsyn, Milind Kulkarni
similarity 0.61
Abstract · 25 pages, 18 figures, 2 tables

Optimizing sparse tensor computations is challenging due to the use of compressed storage formats, which leads to non-affine loop nests and a vast, complex schedule space. The performance of a given schedule is sensitive to the sparsity pattern of the input tensors, making it difficult to find a single optimal solution. When input tensors in the same tensor contraction have conflicting data layouts in relation to the iteration order, it requires costly-both in time and memory-layout transformation, such as transposition. A promising but under-explored alternative is to generate a schedule that avoids explicit transposition, but this has not been systematically supported in existing compilers. This paper presents a new code generation strategy that generalizes the intermediate representation of the TACO sparse tensor compiler to generate a single loop nest, circumventing explicit transposition of tensors when the tensors have conflicting data layout iteration orders. We extend TACOś iteration graph to express a search-based strategy for locating elements in tensors with conflicting layouts, and we introduce new intermediate representation nodes to lower these schedules to efficient code. This enables the systematic generation of loops that do not require explicit transposition, thus avoiding the overhead of materializing temporary tensors. We evaluate our approach on a set of sparse tensor contractions using both real-world and synthetic datasets. Our results demonstrate that for computations with misaligned data layouts, our fused approach achieves up to 2x speedup for some sparsity patterns over the traditional approach of explicitly creating a transposed temporary. We also provide guidelines for when this new scheduling strategy is likely to be beneficial.

0.35