CS Paper Radar

Bit-Sparsity Aware Acceleration with Compact CSD Code on Generic Matrix Multiplication

Zixuan Zhu, Xiaolong Zhou, Chundong Wang, Li Tian, Zunkai Huang, Yongxin Zhu

cs.AR ·submitted 2026-11-11 07:25 UTC · v1
top venuehighly cited

Abstract

The ever-increasing demand for matrix multiplication in artificial intelligence (AI) and generic computing emphasizes the necessity of efficient computing power accommodating both floating-point (FP) and quantized integer (QINT). While state-of-the-art bit-sparsity-aware acceleration techniques have demonstrated impressive performance and efficiency in neural networks through software-driven methods such as pruning and quantization, these approaches are not always feasible in typical generic computing scenarios. In this paper, we propose Bit-Cigma, a hardware-centric architecture that leverages bit-sparsity to accelerate generic matrix multiplication. Bit-Cigma features (1) CCSD encoding, an optimized on-chip sparsification technique based on canonical signed digit (CSD) representation; (2) segmented dot product, a multi-stage exponent matching technique for long FP vectors; and (3) the versatility to efficiently process both FP and QINT data types. CCSD encoding halves the cost of CSD encoding while achieving optimal bit-sparsity, and segmented dot product improves both accuracy and throughput. Bit-Cigma cores are implemented using 65 nm technology at 1 GHz, demonstrating substantial gains in performance and efficiency for both FP and QINT configurations. Compared to state-of-the-art Bitlet, Bit-Cigma achieves 3.2\texttimes{} performance, 6.1\texttimes{} area efficiency, and 15.3\texttimes{} energy efficiency when processing FP32 data while ensuring zero computing error.

Comment
Accepted for publication in IEEE Transactions on Computers (TC)
Journal ref
IEEE Transactions on Computers, Vol. 74, No. 2, pp. 414--426, Feb. 2025
DOI
10.1109/TC.2024.3483632

Closest papers

Nearest neighbours by meaning across the whole index.

Easy Multi-Precision Acceleration via Vectorization

cs.PF cs.DC ·2026-11-03 ·Chansup Byun, Piotr Luszczek, Jeremy Kepner, LaToya Anderson, William Arcand +16 more
similarity 0.73
Abstract

High level programming languages and accelerators are powerful enablers for a wide range of applications. Accelerators offer significant performance via diverse numeric precision and increased memory bandwidth and capacity. These enable new algorithms to be explored. Construction of these algorithms can be enabled by the ready accessibility of these performance features in high-level languages. Achieving scalable vertical (within a compute node), precision (over various numeric precisions), and temporal (over different generations of hardware) performance while retaining productivity requires effective abstractions. Vectorized coding that expresses operations as high-level matrix mathematics has historically been an important method for achieving portable performance. This paper examines the performance availability across a wide range of precisions and hardware using simple vectorized coding techniques applied to several example programs that highlight bandwidth, real and complex matrix multiplication, and 2D convolution. The systems evaluated span nearly a decade of computing hardware, enabling direct comparison of performance gains over this period. This work demonstrates that near hardware peak performance is readily accessible from high-level programming environments. In particular for memory bandwidth, real and complex matrix multiplication in double/single/half precision, and 2D convolutions using the Fast Fourier Transform (FFT). Furthermore, algorithms that trade decreased compute intensity for increased memory intensity have significant potential.

0.24

FIGFLEX: A Mixed-Precision FP-INT GEMM Accelerator Co-Designed with Low-Bit LLM Quantization

cs.AR cs.LG ·2026-10-21 ·Faraz Tahmasebi, Haocheng Xu, Andrew Trauger, Yomal Perera, Zhenyu Tang +2 more
similarity 0.72
Abstract · 15 pages, 12 Figures

Weight-only quantization keeps LLM activations in floating point and stores weights as low-bit integers, turning the dominant workload into floating-point-integer (FP-INT) general matrix multiplication. Recent quantizers reach a 2-3-bit average with mixed precision, but hardware has not kept pace: GPUs dequantize weights to FP16 before the tensor-core multiply-accumulate, and bit-parallel FP-INT accelerators run at a single weight precision, so fewer bits save memory but not computation. In this paper, we present FIGFLEX, an algorithm-hardware co-designed mixed-precision accelerator whose bit-parallel datapath scales throughput as precision falls. FIGFLEX contributes (1) a shift-add, radix-4 reformulation of FP-INT multiplication that fuses the multiply-accumulate into a unified, precision-flexible partial-product reduction and roughly halves the partial products, together with a novel INT3 optimization that reclaims the idle datapath radix-4 leaves at that precision; and (2) a mixed-precision quantizer that assigns each channel group its bit-width according to the error GPTQ cannot compensate, with a per-input-channel column scaling whose power-of-two normalization keeps the scaled activations out of the FP16 subnormal range, so that the quantized model maps directly onto FIGFLEX's datapath with the same accuracy as on a GPU. On LLaMA-2-7B, FIGFLEX attains 5.85 and 8.77 WikiText-2 perplexity at 3- and 2-bit average precision, lower than GPTQ, AWQ, SliM-LLM, AMQ, and QuIP# (without fine-tuning), while delivering 1.56x and 1.42x higher whole-chip throughput per area on the linear GEMMs of a 3-bit mixed-precision model than AxCore and FIGLUT, respectively.

0.13

Precision-Aware Variable Bit Processing Elements for Hardware-Efficient Systolic Array Designs

cs.AR cs.ETcs.LGeess.IV ·2026-08-23 ·Dantu Nandini Devi, Madhav Rao
published similarity 0.71
Abstract

Systolic arrays (SAs) have emerged as prominent hardware accelerators for matrix operations in deep learning, while floating point number formats enable precision control across computational domains. This research investigates approximate computing techniques for floating point (FP) multipliers in Weight Stationary Systolic Arrays, focusing on IEEE 754 (FP32), TensorFloat-32 (TF32), and Brain Floating point (BF16) formats. By integrating partial product matrix (PPM) column truncation with positive and negative compressors in the FP multiplier architecture, we optimize the trade-off between computational efficiency and accuracy. NSGA-II optimization algorithm was employed to explore the vast design space for evolving FP multiplier designs, towards achieving substantial hardware improvements while maintaining acceptable output quality. Substantial hardware benefits were observed in the FP multiplier designs across various applications, while preserving output quality. The FP approximated Processing Elements designed in the SA was found to offer comparable CNN accuracy for models trained on MNIST, F-MNIST, and CIFAR-10 dataset. The FP approximated SA designs that fall in the top 10 CNN performance offered substantial hardware gains in the range of 66% to 92% footprint savings, 60% to 93% of power benefits with 21% to 54% improvement in the delay when compared with the corresponding exact implementations mentioned in the literature for running the model trained on CIFAR-10 dataset. The TF32 and BF16 approximated SA designs also achieved substantial gains while maintaining comparable CNN accuracy. Our findings confirm that targeted approximation in FP multiplier design significantly improves the efficiency of hardware accelerators for error-tolerant applications, establishing an effective approach to hardware resource optimization in contemporary computing architectures.

0.27

Decode Late, Multiply Once: Rethinking the Dot Product for Quantized Weights

cs.LG cs.AR ·2026-10-22 ·Jonathan Tapson
similarity 0.69
Abstract

Large neural networks are increasingly deployed with 1-4 bit weights and 16- or 32-bit activations, yet their dot products still use one multiplier per element. We show that a length-n dot product with k-bit weight codes can be computed with n+2^k additions and at most 2^k decodes and 2^k multiplications. Each weight code is used directly as an address offset that selects one of 2^k accumulators; each activation is added into the selected accumulator; and the accumulated sums are decoded and multiplied only once, at the end, in a fused decode-and-multiply step. The decoded weight vector is never formed. The refactoring is exact for integer arithmetic, works with integer or floating-point operands and with arbitrary, including nonlinear, decoders, and extends to group-wise scales and to matrix and tensor products. The multiplier count falls by n/2^k, while total arithmetic energy improves by a smaller, bounded factor that we quantify with a cost model. We describe a scatter-accumulate processing element (SA-PE) that replaces the MAC unit in a weight-stationary systolic array, with a load/stream/reduce/drain schedule in which a small, time-shareable multiplier runs only 2^k times per tile. Because decoded weights are not stored, local weight memory shrinks by up to w_d/k for w_d-bit decoded weights. An RTL implementation synthesized for FPGA and compared with a matched MAC array shows that an unshared SA-PE is larger than a MAC PE, and that the saving appears when the reduction multiplier is shared across PEs. Software models confirm exact integer equivalence and floating-point error no worse than the conventional pipeline. We do not claim the underlying identity is new, and we discuss related work on weight sharing and lookup-based kernels.

0.23

Efficient Reconfigurable Architectures for Deep Learning

cs.AR ·2026-10-31 ·Endri Taka
hot direction similarity 0.67
Abstract · Ph.D. dissertation, Department of Electrical and Computer Engineering, The University of Texas at Austin, 2026

The growing computational demands of deep learning (DL) have driven AI-optimized reconfigurable devices, such as the AMD Versal AI Engines (AIEs), the Intel Stratix 10 NX Tensor Blocks (TBs), and the AMD Ryzen AI NPUs. Their heterogeneous resources and novel dataflow models make efficient exploitation nontrivial, and FPGAs lack native support for structured sparsity. This dissertation optimizes dense general matrix multiplication (GEMM), the dominant DL kernel, on these devices, proposes FPGA architectural enhancements for structured sparsity, and extends the techniques to AMD Ryzen AI NPUs. First, MaxEVA optimizes GEMM on Versal AIEs, achieving up to 2.19$\times$ higher throughput and 20.4% higher energy efficiency than the state of the art. It is then extended to arbitrary matrix sizes, and a systematic framework is proposed for the Stratix 10 NX TBs. Both frameworks reach state-of-the-art int8 throughput of up to 77 and 68 TOPS, and energy efficiency of up to 0.94 and 1.35 TOPS/W, on Versal and Stratix, respectively. Next, Systolic Sparse Tensor slices (SSTs) are proposed as in-fabric hard blocks supporting dense, 2:4, 1:4, and a new 1:3 sparsity level. SST-based GEMM accelerators achieve up to 5$\times$ higher frequency and 10.9$\times$ lower area than conventional FPGA logic, and on sparse ViT and CNN models SSTs deliver up to 3.52$\times$ speedup over dense in-fabric acceleration with up to 13.3% area increase. Finally, a common systematic methodology is proposed to optimize GEMM across the XDNA and XDNA2 AMD NPUs, reaching state-of-the-art performance of up to 6.76 and 38.05 TOPS (int8) and 3.14 and 14.71 TOPS (bf16), respectively.

0.24

Algebraic Integer Quantization for Machine Learning Applications

cs.LG ·2026-09-25 ·Anthony Kostal-Vazquez, Vassil Dimitrov, Branden Butler
similarity 0.67
Abstract

In this article, we propose algebraic integer number formats (AINF) as a framework for highly configurable and accurate compressed computation. A key property of AINF is its straightforward reduction to purely integer arithmetic, enabling error-free multiply-accumulate operations. We develop a matrix multiplication engine that exploits this property, obtaining performance within a predictable factor of the most efficient numerical counterparts on modern hardware accelerators. Another key advantage of AIQ is the freedom to select algebraic integers tailored to the numerical characteristics of a given use case, offering optimization opportunities that are fundamentally unavailable in floating-point or fixed-point formats. Motivated by these properties, we introduce algebraic integer quantization (AIQ), a lossy compression format for large language model (LLM) inference, and provide results demonstrating that AIQ achieves competitive accuracy compared to the industry-standard MX format across a diverse set of models and benchmarks at equivalent bit-per-parameter budgets. The experiments presented in this article employ algebraic integers of degree two (quadratic integers); generalizations to higher-order and, in particular, multidimensional algebraic integers promise even greater computational advantages and remain a promising direction for future work.

0.28

TOMOS: Eliminating Data Duplication in Mixed-Precision Matrix Computations

cs.MS ·2026-10-13 ·Vasileios Georgiou, Petros Drineas
similarity 0.66
Abstract

Mixed-precision algorithms exploit reduced-precision hardware, however computing with the same matrix at distinct precisions typically requires duplicate storage, repeated conversions, or operation-emulation. We introduce TOMOS, a composite-precision matrix format that provides implicit 32-bit and 16-bit access from a single representation. Each FP32 entry is split into a 16-bit primary word for low-precision computation and a complementary word that enables exact FP32 reconstruction. The encoding scheme improves numerical quality of the low-precision view through subnormal representations, and allows exact recovery of high-precision inputs. TOMOS uses FP32-sized storage, with data movement scaling with compute precision. Optimized GEMM and GEMV kernels achieve throughput comparable to vendor libraries on NVIDIA A100 GPUs, while a mixed-precision least-squares solver achieves up to 1.6x speedup over FP32 without redundant matrix allocations.

0.16

Taming Bitwise Behavior in GPU Kernels with Tensor Core: Black-Box Reconstruction, Compiler Enforcement, and Static Verification

cs.DC cs.PFcs.PL ·2026-09-10 ·Ziteng Yang, Nicholas J. Riasanovsky, Warren Deng, Vivek Sarkar
hot direction similarity 0.66
Abstract

Determinism and numerical reproducibility are increasingly required of GPU kernels in machine learning systems, yet deterministic implementations of the same kernel can still differ bit for bit. Floating-point reduction order is the primary cause, alongside partial-sum precision, fused multiply-add operations, and rounding placement. These choices may be hand-coded, selected by a block-level language such as Triton, or hidden inside a closed-source library such as cuBLAS or rocBLAS. A tile shape chosen for performance therefore also determines the arithmetic, potentially breaking batch invariance. Preserving a fixed order can cost up to 20 percent, while an autotuner cannot identify which configurations are bitwise equivalent. We characterize the factors determining the bitwise behavior of reductions and general matrix multiplication (GEMM). First, we introduce a descriptor of GEMM reduction order, including the partitioning of K in split-K GEMM. Using it, we perform the first black-box reconstruction of a closed-source library's arithmetic for bit-level correctness. Our family of Triton GEMMs matches NVIDIA cuBLAS in all tested cases on Blackwell and Hopper. For realistic LLM shapes with fused epilogues, it matches or exceeds torch.compile performance. Second, we enforce balanced-tree reduction during Triton lowering and introduce a data-layout optimization that brings 19 of 27 kernels on GB300 and H100 within 10 percent of free-order performance. Third, we develop sound static checkers for bitwise equivalence between compiled GPU kernels, including the first checker spanning NVIDIA PTX and AMD GCN. Integrated into Triton's autotuner, the checker restricts search to a single bit-equivalence class.

0.53