CS Paper Radar

Massively Parallel Algorithms for Huffman Coding

Masoud Seddighin, Saeed Seddighin

cs.DS ·submitted 2026-10-06 22:00 UTC · v2

Abstract

Huffman coding is one of the oldest and most fundamental problems in computer science. Given a string of length $\TextLength$ over a general alphabet, the goal is to assign a binary codeword to each character so that no codeword is a prefix of another and the total encoded length of the string is minimized. Huffman coding is widely used in practical compression systems, including file compression. As modern datasets continue to grow, it is natural to study whether a Huffman code can be constructed efficiently in the massively parallel computation (\MPC) model. The celebrated Huffman coding algorithm admits two straightforward \MPC implementations: for any constant $ε\in(0,1)$, one uses $O(\TextLength^ε)$ memory per machine but requires $Θ(\log \TextLength)$ rounds, while the other runs in $O(1)$ rounds but requires $Θ(\sqrt{\TextLength})$ memory per machine. We give the first nontrivial \MPC algorithm for Huffman coding that bypasses both limitations. For every constant $ε>0$, our algorithm uses $O_ε(\log\log \TextLength)$ rounds and $\softO(\TextLength^ε)$ memory per machine, while its total memory and total computation are $\softO(\TextLength)$. {\color{black}Although our framework is tailored to the \MPC model, we show that it has applications in the \PRAM setting as well. Specifically, we show that the same algorithm can be implemented in the \PRAM model with total work close to linear. In particular, whenever the text length is at most $2^{O\left(\left(\frac{\log \AlphabetSize}{\log\log \AlphabetSize}\right)^2\right)}$ (slightly superpolynomial), where $\AlphabetSize$ is the alphabet size, our parallel running time is asymptotically better than that of all previously known \PRAM algorithms and is essentially optimal.}

Closest papers

Nearest neighbours by meaning across the whole index.

Binary Huffman Coding with Unequal Letter Costs Is NP-Complete

cs.DS cs.CCcs.CGcs.IT ·2026-11-03 ·L. Sunil Chandran, Rishikesh Gajjala
similarity 0.66
Abstract · 51 pages

Binary prefix coding assigns prefix-free binary words to symbols to minimize their frequency-weighted cost. Huffman solved the equal-letter-cost case in polynomial time in 1952 [Proc. IRE'52]. For unequal letter costs, Karp gave an integer-programming formulation in 1961 [IRE Trans. Inf. Theory'61]. Exact polynomial-time algorithms for fixed integer costs were given by Golin and Rote [IEEE Trans. Inf. Theory'98] and Bradford, Golin, Larmore, and Rytter [J. Algorithms'02]. For input-dependent costs, Golin, Mathieu, and Young gave a polynomial-time approximation scheme [SICOMP'12]. The exact complexity when letter costs are part of the input had remained unresolved. We resolve this $65$-year-old problem by proving that deciding whether there exists a binary prefix-free code of frequency-weighted cost at most a given threshold is NP-complete, even when the letter costs are consecutive positive integers $(M,M+1)$ supplied as part of the input. All frequencies, letter costs, and thresholds are encoded in binary. Consequently, finding an optimal code is NP-hard. Our proof uses two polynomial-time reductions, starting from Planar Independent Set. The intermediate problem prescribes the number of leaves at each depth of a full ordered binary tree and minimizes a signed linear score on their sorted right-edge counts. The first reduction represents an independent-set verification program by a tree growing level by level. A fixed-profile identity converts narrow coordinate restrictions into linear penalties that enforce the computation. The second reduction adds large dyadic terms to the symbol frequencies to force prescribed depths. These terms contribute a constant to the weighted number of ones, leaving smaller signed perturbations to reproduce the tree score.

0.23

LazyWeights: Decode-on-Demand Edge LLMs

cs.LG cs.ARcs.CLcs.IT ·2026-11-12 ·Arnab Sanyal, Gourav Datta, Sandeep P. Chinchali
similarity 0.60
Abstract · 6 pages, 1 reference page, 5 figures, 6 tables

Memory determines which large language models (LLMs) an edge device can run and how fast it runs them: capacity limits a 4 GB Jetson Nano, and bandwidth limits an 8 GB Jetson Orin Nano Super. Deployed weight formats (GPTQ, AWQ, and the GGUF K- and I-quants) spend the same number of bits on every integer code of a matrix, although the codes are far from uniform. We present LazyWeights, which keeps these formats, tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7-31.4% smaller than the best GGUF format. A bit-exact 0.21 mm^2 ASIC decoder unit (1 GHz, 7 nm) cuts Orin latency and energy per token by up to 36% and 38% (54% and 44% with prompt-lookup speculation). On the Nano, LazyWeights needs 58-64% less time and 53-57% less energy per token than the GGUF runtime on the CPU, and runs a streamed Mistral-7B 6.6x faster at 3.89 bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput, and the unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).

0.17

Efficient Lempel-Ziv Prediction with the Run-Length BWT

cs.DS cs.LG ·2026-11-01 ·Ohad Avnery
similarity 0.60
Abstract · 21 pages, 1 table

Lempel-Ziv Plurality (LZP) is an algorithm of Kosoy that predicts the next symbol of a string by using its Lempel-Ziv factorization, and makes at most $O(z\log n)$ mistakes on every string of length $n$ with $z$ LZ77 phrases. We give efficient online implementations of LZP and of a new variant of it. Our first implementation maintains a run-length encoded Burrows-Wheeler transform (RLBWT) of the reversed history. It implements LZP, uses $O(z\log z\log^2 n)$ bits, and has worst-case update and prediction times $O(\log z+\log\log n)$ and $O(σ(\log z+\log\log n))$, respectively, where $σ$ is the size of the alphabet. This is an exponential improvement over Kosoy's implementation, whose update time is $O(z\log^7 n)$. We then introduce the Lempel-Ziv Majority (LZM) predictor, which has the same mistake bound and update time, and predicts in $O(1)$ time independently of $σ$. These two predictors accept new symbols at any time, without knowing the alphabet in advance. For an alphabet known in advance, a different data structure gives $O(σ(1+\log z/\log\log n))$ time per symbol, which is constant when $σ=O(1)$ and $z=\mathrm{polylog}(n)$.

0.25

FMT: Lossless Image Compression in a Fixed Number of Parallel Steps

cs.CV ·2026-11-17 ·Ammar Ali
position similarity 0.58
Abstract

Learned lossless codecs make smaller files than common classical codecs, yet the images we archive and train on still use classical codecs. A main reason is time: a learned codec takes seconds to decode a photo that a classical codec opens in milliseconds, and their faster variants give up most of their gain in size. We argue that much of this time is spent waiting, not computing: the network waits for the values it conditions on, and the arithmetic decoder reads them one at a time. We introduce FMT (Fixed-step Multiscale Tree), a lossless image format whose decoding order and entropy coder are designed together so that neither wait grows with the image. FMT stores an image as a coarse-to-fine tree whose steps hold independent values: a GPU predicts each step at once and decodes it in parallel rANS lanes that carry their states from step to step, so an image of any size decodes in a fixed number of sequential steps. On this tree we build a family of bit-exact codecs, from a rule with no network to large networks, and no codec we ran on the same GPU beats any of them on both size and speed. Without a network, FMT beats NVIDIA's HTJ2K on size and speed; with a large one, it decodes Kodak photos 9 to 61X faster than learned codecs we ran on the same A100, at the same size or smaller.

0.35

Binary Cyclic Gradient Coding: Exact Recovery and Robustness to Compression

cs.IT ·2026-10-14 ·Hassan Ou-azzou, Imad Kissami, El Houcine Bergou
theory similarity 0.58
Abstract

We introduce binary cyclic gradient coding (BCGC), an exact gradient coding scheme for $n$ workers and up to $s$ stragglers. Its encoding matrix is binary, circulant when $(s+1)\mid n$ and nearly circulant otherwise, and the master decodes by a single summation. Like the fractional repetition coding scheme (FRC) and its generalized version (GFRC), BCGC splits the workers into $s+1$ classes that each cover the whole dataset, but it assigns the data through cyclic supports. Its encoding matrix has rank $n-s$, and $n-s-1$ workers keep the balanced load $s+1$, which is the largest possible number. We prove that encoding matrix of BCGC is not permutation-equivalent to those of FRC and GFRC schemes. Moreover, We also prove that a binary coded scheme with compression is equivalent to an uncoded compressed scheme on a coarser partition of the data. This explains why binary schemes remain stable under compression, whereas the large decoding coefficients of MDS codes amplify the compression error and lead to divergence in our experiments. For every contractive compressor, the standard convergence guarantee of compressed gradient descent holds on each coarse partition. FRC, GFRC and BCGC therefore share the same convergence guarantee and differ only in their partitions, with one partition for FRC, two for GFRC and $s+1$ distinct partitions for BCGC. Choosing among these partitions can recover the exact gradient in cases where FRC and GFRC cannot. Experiments on four datasets confirm the theory. BCGC matches GFRC, keeps its accuracy under top $κ$ sparsification while the MDS codes diverge, and on heterogeneous data its partition diversity reduces the compression error and improves the accuracy.

0.18

The Invocation Bottleneck in LLM-Based Lossless Compression: What Can Be Removed and What Must Be Traded

cs.IT cs.DB ·2026-11-04 ·Zhe Yan, Yihao Liu, Antonio Boffa, Huanchen Zhang
positioncode/data similarity 0.57
Abstract · 14 pages, 7 figures, 4 tables. Code: https://github.com/Angie-Yan/LLM-based-Data-Compression

Large language models (LLMs) compress text far better than classical compressors, but LLM-based compressors are slow, particularly during decompression. We argue that their decoding time has two fundamentally different sources. Execution redundancy, such as per-token host transfers, CPU-side coding, and token-by-token encoding, is work that the coded output does not require and can be removed without changing a single output bit. Sequential dependency, one model call per decoded token, is inherent to lossless decoding and cannot be removed by better execution, only traded for more calls or for more bits. We remove execution redundancy with a GPU-resident pipeline whose block-parallel encoding is made lossless in half precision by batch-invariant kernels, speeding up arithmetic coding by up to 5.8x in encoding and 3.0x in decoding without changing its compression rate. We then study the two ways of trading sequential dependency, speculative dual-model coding (SDMC) and multi-token single-model coding (MSMC), characterize the compression cost of predicting several tokens from a shared prefix, and identify the workloads in which fewer model calls are worth that cost.

0.58

FastPair: GPU-Optimized String Decoding

cs.DB cs.ARcs.DC ·2026-09-14 ·Joseph Isaacs, Francesco Gargiulo, Peter Boncz, Robert Kruszewski, Nicholas Gates +3 more
similarity 0.57
Abstract

Modern data systems compress data at rest and decompress it only when needed to preserve interconnect bandwidth. This design is often inefficient on GPU-based compute platforms because many conventional compression techniques exhibit serial data dependencies that limit GPU parallelism, leaving resources idle. Recent NVIDIA GPUs address this decoding deficiency through the Decompression Engine (DE), an on-die, fixed-function decompression accelerator for general-purpose compression formats such as Deflate, LZ4, and Snappy. Recent work has proposed string codecs that replace frequent substrings with fixed-width codes from a small, trained dictionary, making each code's lookup independent. While these lookups can run in parallel, the resulting scattered reads and short output writes still do not align well with GPU hardware, which handles contiguous memory accesses more efficiently. We present FastPair, a GPU decoder that optimizes the existing dictionary decoding process by reorganizing lookups and assembling decoded substrings for contiguous output writes. On a B300, FastPair decodes ten real-world columns 2.4 to 4.2x faster than the DE, reaching up to 1.6 TB/s.

0.20

Copying along Arithmetic Progressions Cannot Simulate Interleaving with Polylogarithmic Loss

cs.DS cs.FL ·2026-11-16 ·Matthias G. Mayer
code/data similarity 0.56
Abstract · 45 pages. Lean 4 formalisation: https://github.com/matthiasGmayer-research/copying-vs-interleaving-lean

Two classical ways to compress a word are straight-line programs (SLPs), which build it by concatenating smaller pieces, and the Lempel-Ziv 77 (LZ77) factorisation, which copies earlier parts of it; their sizes are measures of repetitiveness. LZ77 approximates the smallest SLP efficiently: it has no more factors than a smallest SLP has size, and yields an SLP at most a logarithmic factor larger in near-linear time. Motivated by online prediction, Kosoy introduced zipline programs, which are analogous to SLPs but interleave their pieces round-robin instead of concatenating them. Her generalisation of LZ77, the arithmetic repetition system, describes a word by copies from one arithmetic progression of positions to a later one. Do arithmetic repetition systems approximate the smallest zipline program as well? Kosoy proved that the least size $\mathrm{ARC}(x)$ of an arithmetic repetition system for a word $x$ of length $n$ is at most a factor $O(n^{1/3})$ larger than the least size $\mathrm{ZLC}(x)$ of a zipline program for it, and asked whether this factor is tight. Her polynomial-time next-letter predictor makes $O(\mathrm{ARC}(x)\log n)$ mistakes, so a polylogarithmic factor would bound its mistakes by $\mathrm{ZLC}(x)\,\mathrm{polylog}(n)$, which no known polynomial-time predictor achieves. We show that the optimal exponent is $1/4$, not $1/3$: $\mathrm{ARC}(x) \le 2n^{1/4}\,\mathrm{ZLC}(x)$ for every word $x$, and at every large enough length $n$ some binary word has $\mathrm{ARC}(x) \ge n^{1/4-o(1)}\,\mathrm{ZLC}(x)$.

0.53