CS Paper Radar

Binary Cyclic Gradient Coding: Exact Recovery and Robustness to Compression

Hassan Ou-azzou, Imad Kissami, El Houcine Bergou

cs.IT ·submitted 2026-10-14 12:24 UTC · v2
theory

Abstract

We introduce binary cyclic gradient coding (BCGC), an exact gradient coding scheme for $n$ workers and up to $s$ stragglers. Its encoding matrix is binary, circulant when $(s+1)\mid n$ and nearly circulant otherwise, and the master decodes by a single summation. Like the fractional repetition coding scheme (FRC) and its generalized version (GFRC), BCGC splits the workers into $s+1$ classes that each cover the whole dataset, but it assigns the data through cyclic supports. Its encoding matrix has rank $n-s$, and $n-s-1$ workers keep the balanced load $s+1$, which is the largest possible number. We prove that encoding matrix of BCGC is not permutation-equivalent to those of FRC and GFRC schemes. Moreover, We also prove that a binary coded scheme with compression is equivalent to an uncoded compressed scheme on a coarser partition of the data. This explains why binary schemes remain stable under compression, whereas the large decoding coefficients of MDS codes amplify the compression error and lead to divergence in our experiments. For every contractive compressor, the standard convergence guarantee of compressed gradient descent holds on each coarse partition. FRC, GFRC and BCGC therefore share the same convergence guarantee and differ only in their partitions, with one partition for FRC, two for GFRC and $s+1$ distinct partitions for BCGC. Choosing among these partitions can recover the exact gradient in cases where FRC and GFRC cannot. Experiments on four datasets confirm the theory. BCGC matches GFRC, keeps its accuracy under top $κ$ sparsification while the MDS codes diverge, and on heterogeneous data its partition diversity reduces the compression error and improves the accuracy.

Closest papers

Nearest neighbours by meaning across the whole index.

Deterministic Distributed Linearly Separable Computation via Nested Cyclic Codes

cs.IT ·2026-10-13 ·Anu Mathew, M. Nikhil Krishnan
similarity 0.70
Abstract · 13 pages, 1 figure

We study the $(K,N,S,M,K_c)$ distributed linearly separable computation (DLSC) setting, where a master employs $N$ workers with a per-worker computation load of $M$ to recover $K_c$ deterministic linear combinations of $K$ messages in the presence of up to $S$ stragglers. We propose a DLSC framework based on nested cyclic codes that achieves $M=\frac{K}{N}(S+K_c)$. For sufficiently large $K$, we establish the matching converse $M\geq\frac{K}{N}(S+K_c)$, proving the asymptotic computation load optimality of the proposed framework. By instantiating cyclic locally recoverable codes (LRCs) as base codes in the framework, the master can recover the desired $K_c$ linear combinations by contacting fewer than $N-S$ workers under many straggler scenarios. This essentially softens the straggler threshold without increasing the computation load or communication cost. Finally, we extend the construction to reduce the per-worker communication load by a factor of $m$, while achieving an asymptotically optimal computation load of $M=\frac{K}{N}(S+mK_c)$.

0.20

Communication-Efficient Distributed Training via Ring-Based Coded Approximate All-Reduce

cs.IT ·2026-09-20 ·Sifat Munim, Aditya Ramamoorthy
similarity 0.61
Abstract · 21 pages, 1 figure

Ring All-Reduce is a widely used protocol for aggregating gradients across workers in large-scale distributed training. For a system with $N$ workers, its normalized per-worker communication is $2(N-1)/N$. In this work we present a communication-efficient Ring All-Reduce (CERAR) protocol for ``approximate'' gradient aggregation. CERAR partitions each local gradient into $c$ components and relies on linear encoding and decoding operations. It performs $L=c+N-2$ communication rounds over an $N$-worker ring, yielding normalized communication rate $1+(N-2)/c$ and normalized storage rate $1+N/c$. We present an explicit Vandermonde matrix based construction whose approximation error can be made arbitrarily small, with communication and storage rates approaching one as $c$ increases. However, this comes at the cost of ill-conditioned encoding matrices. We therefore give an alternate construction whose error is $O(ε)$, while the relevant condition numbers are $O(ε^{-r_\star})$, where $r_\star=\lceil c/N\rceil-1$. This motivates a condition-number-constrained optimization formulation for obtaining numerically stable practical designs. Our experiments show that, compared with the production-standard NCCL on NVIDIA A100 PCIe GPU clusters with low-bandwidth PCIe interconnect, CERAR reduces raw All-Reduce time by up to 47\%. For training a 1.4-billion-parameter Pythia model, it reduces training time by about 17\% for the same number of iterations while achieving essentially identical validation loss. On GPU clusters with high-bandwidth NVLink interconnects, CERAR performs worse in the All-Reduce phase and slightly worse in overall training time. However, the gap narrows for larger model sizes, suggesting that further gains may be possible with more optimized CERAR implementations.

0.29

Computed Permutation Trellis Coding: Near-Optimal Quantization from Two Multiplications and a 64-Byte Table

cs.IT cs.AIcs.ARcs.LG ·2026-11-17 ·Muhammed Fatih Balın
similarity 0.60
Abstract · 38 pages, 6 figures, 27 tables

We propose Computed Permutation Trellis Coding (CPTC), a low-bit quantization format for tensors written once and read many times. Formats that round each value on its own, such as INT4 and FP4, waste a large part of their bits against the information-theoretic ideal. Trellis codes recover this loss by encoding values jointly, but prior decoders either need a large lookup table for their best quality or compute values with a hash that restricts what they can decode. CPTC's decoder multiplies its state by two coefficients and adds the top bits of the products to index a 64-byte table. Because that sum is not uniformly distributed, the rarely used ends of a small table can hold values far out in the tails. One decoder serves every bit width from 1 to 6, reproduces the standard integer and floating-point formats exactly, and comes within 0.07 bits of the ideal at 4 bits. It matches a random-permutation trellis code whose table is thousands of times larger, and exceeds every QTIP codebook. In LLM weight quantization without fine-tuning, CPTC has the lowest KL divergence to the unquantized model at every tested bit width on Llama 3.1 8B and 70B Instruct against QTIP's codebooks (by up to 27%). The decoder has been built into the AWS Trainium 4 AI accelerator, where it runs at the throughput of the native FP4 and FP8 formats for a negligible fraction of the systolic array's area. Fused encoding kernels quantize an 8B model's linear layers in under two minutes on one GPU and, in cycle-level simulation, on one Trainium 4 chip.

0.40

The ordered sum-rank metric and linearized multiplicity codes: NRT meets sum-rank

cs.IT ·2026-11-15 ·Umberto Martínez-Peñas
similarity 0.60
Abstract

In this work, we introduce linearized multiplicity codes, which are a common generalization of linearized Reed--Solomon codes and univariate multiplicity codes. We also introduce the ordered sum-rank metric, which is a common generalization of the sum-rank metric and the Niederreiter-Rosenbloom-Tsfasman (NRT) metric (or ordered Hamming metric). We provide a Singleton bound for the ordered sum-rank metric and show that linearized multiplicity codes attain it. That is, they are maximum ordered sum-rank distance (MOSRD) codes. We introduce the support lattice for the ordered sum-rank metric and study duality, support distributions and weight distributions of MOSRD codes. We also show that MOSRD codes are ordered MDS, dually quasi MSRD and, under some divisibility conditions, MSRD (and thus MDS). For linear MOSRD codes, we give lower bounds on the field extension degree by the number of matrix columns, and upper bounds on the block length by the base field size. We conclude by showing that linearized multiplicity codes attain such bounds exactly in almost all cases, and the same growth order in all cases.

0.05

Secure Multi-Access Coded Caching: A Lifting Approach

cs.IT ·2026-10-04 ·Han Fang, Qinyi Lu, Nan Liu, Wei Kang
theory similarity 0.60
Abstract · 48 pages, 2 figures. This paper was presented in part at the 2024 IEEE Information Theory Workshop(ITW)

We construct secure cyclic multi-access coded caching schemes by re-encoding the caches of a secure single-access scheme and reusing its multicast message unchanged. For a library of $N$ files, each of $K$ users reads $L<K$ consecutive shared caches and must recover its requested file while learning no information about the other files. Starting from a parent scheme with a cyclic representation of its cached symbols, the transformation lets each access window recover the corresponding parent-cache symbols and, conditioned on them, learn nothing about the remaining symbols. A deterministic Pascal-window encoder handles symbols whose consecutive appearances span at least $L$ users. A randomized complement-dual encoder handles shorter spans. Over a suitable finite field, these encoders attain the exact minimum per-block storage for every input length in the block-separable, parent-preserving class. Applying the transformation to two secure single-access schemes gives an achievable memory-rate family for $N,K\ge2$ and $1\le L<K$, including the exact rate-one cache size $(N-1)(K-1)/L$. Using converse bounds for arbitrary secure multi-access schemes, we prove approximation guarantees for the achievable family and give a proof of the exact $L=K-1$ tradeoff.

0.34

Concatenated Codes with One Inner Code: Low-Rate Gilbert-Varshamov Asymptotics from Algebraic-Geometric Codes

cs.IT ·2026-11-12 ·Yota Maeda, Ruho Kondo
theory similarity 0.59
Abstract · 25 pages

Constructing codes that attain the Gilbert-Varshamov (GV) bound while admitting efficient encoding and decoding is a fundamental problem in coding theory. We study concatenated codes that reuse a single binary inner code across all outer coordinates. To exploit existing encoding and decoding algorithms, we restrict the outer codes to random subcodes of scaled copies of prescribed algebraic codes and investigate their minimum distance. Our first contribution is a general upper bound on the minimum-distance failure probability that separates two terms controlled by the inner code's bias moments and the dual weight distribution of the prescribed code, together with sufficient conditions for exponential decay. Our second contribution is a concrete construction satisfying these conditions using algebraic-geometric codes from the Garcia-Stichtenoth tower. For every sufficiently small $\varepsilon>0$, we construct a family of random binary codes $C_m$ whose rates approach $\varepsilon^2$ and satisfy \[ \Pr\left[δ(C_m)\geq\frac12-\sqrt{\frac{\ln 2}{2}}\left(\varepsilon+\frac{\varepsilon^3}{2}\right)\right]\geq 1-2^{-c\varepsilon^4N_m}, \] for all sufficiently large $m$, where $N_m$ is the block length and $c>0$ is independent of $\varepsilon$ and $m$. Thus, the distance guarantee matches the leading coefficient of the GV bound at low rates. For each fixed $\varepsilon$, the construction uses a single inner encoding map throughout the family and admits polynomial-time encoding and unique decoding within a guaranteed radius. This answers the question of Doron, Mosheiff, and Wootters for subcodes of scaled algebraic geometry codes at low rates.

0.19

Single-Fragment Forensic Coding via Multidimensional Cyclically Permutable Codes

cs.IT ·2026-09-21 ·Yubo Sun, Yijun Zhang, Gennian Ge
similarity 0.59
Abstract · 23 pages

The proliferation of 3D printing raises new security and forensic challenges, including the risk of unauthorized fabrication of untraceable firearms and other regulated items. To enable traceability, we consider \emph{single-fragment forensic coding}, in which a unique identifier is embedded into a printed object and must be recoverable from any fragment containing an axis-parallel box of prescribed minimum volume and coordinatewise thickness, even in the presence of substitution errors. To address this problem, we introduce and study multidimensional cyclically permutable codes (CPCs), for which every cyclic translate uniquely determines both the original codeword and the applied translation. By applying periodic lifting to a CPC base array, every complete period contained in a fragment corresponds to an unknown cyclic translate of a codeword, thereby reducing the forensic alignment problem to multidimensional cyclic synchronization. We establish theoretical bounds and develop explicit constructions in both the noiseless and substitution-error settings. For fixed dimension $d$ and alphabet size $q$, we obtain the optimal redundancy $d\log_q k+o(1)$ for noiseless $d$-dimensional CPCs of side length $k$. For fixed $t$, the optimal redundancy of $t$-substitution-correcting $d$-dimensional CPCs lies between $(t+1)d\log_q k+O(1)$ and $(2t+1)d\log_q k+O(1)$. For binary alphabets and $d\geq2$, an explicit robust row-anchor construction achieves $(t+1)d\log_2 k+O(\log\log k)$ redundancy, matching the optimal leading term. In the thick-fragment regime $h=cM^{1/d}$, where $M$ and $h$ lower-bound the volume and the side lengths of a box contained in the fragment, periodic lifting yields single-fragment forensic codes of rate $c^d-o(1)$. In particular, the rate approaches one when $c=1-o(1)$.

0.44

CPM-LDPC Codes Attaining the Minimum-Distance Bound

cs.IT ·2026-09-15 ·Kenta Kasai
theorycode/data similarity 0.59
Abstract · 13 pages, 1 figure, 2 tables. Verification code and data: https://github.com/kasaikenta/cpm-ldpc-distance

We study binary quasi-cyclic LDPC codes whose parity-check matrices are full arrays of single circulant permutation matrices (CPMs), referred to here as CPM-LDPC codes. Their minimum distance is at most $(J+1)!$, where $J$ is the column weight. For every fixed pair of column and row weights $2\le J<L$, we show that this bound is attained for all sufficiently large integer lift sizes. First, we give one integer exponent matrix independent of the lift size $P$. Second, we show that independent uniform exponent choices attain the bound with probability $1-O_{J,L}(P^{-1})$. Both proofs use cycle conditions required by low-weight codewords and a lower bound on the number of terms in vectors satisfying polynomial check equations. Neither construction requires $P$ to be prime. We also give small-lift arrays attaining the bound 24 for $J=3$, $L=4,\ldots,8$, and arrays with distance at least 28 for $J=4$, $L=5,\ldots,8$, together with computational distance verification.

0.35