CS Paper Radar

Pairwise distortion distribution for compression and quantization

Carl Nuzman, Jeongmin Chae, Kursat Rasim Mestav, Urbashi Mitra

cs.LG ·submitted 2026-11-09 06:50 UTC · v1
publishedtheory

Abstract

Choosing the latent dimension is a critical challenge in data compression, where conventional hyperparameter optimization is costly and ad hoc. Intrinsic dimension provides a good alternative, which connects information-theoretic notions such as Renyi information dimension (RID) to rate-distortion dimension and random codebook compression. In this work, we introduce the pairwise distortion distribution (PDD) and its asymptotic limit, pairwise dimension (PD), which generalize the correlation integral to general distortion measures. We show that PD bounds RID, and PDD provides a lower bound for random codebook quantization performance, which can guide a practical use for compression, quantization, and dimensionality reduction applications.

Journal ref
ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Closest papers

Nearest neighbours by meaning across the whole index.

Pairwise Distortion Distribution with Graph-Based Importance Sampling for Quantization Error and Intrinsic Dimension Estimation

cs.IT stat.ML ·2026-11-03 ·Xi Chen, Carl Nuzman, Shirin Jalali
theory similarity 0.84
Abstract · 22 pages, 5 figures

Quantifying the dimensionality of data is a fundamental need in machine learning and signal processing applications such as representation learning, data compression, and inverse problems, where the extrinsic dimension in which the data is presented can be very different from the intrinsic dimension that determines the operational performance. Various concepts, such as Rényi information dimension, rate-distortion dimension, and quantization dimension are useful here. Despite their theoretical significance, estimating these measures are challenging given their asymptotic definitions. Recent research has shown that analyzing the pairwise distortion distribution (PDD) of a data source can provide a tractable lower bound on the distortion of random codebook quantization, from which a tractable notion of intrinsic dimension can be derived. In this paper we extend this line of research in three main ways. First, we develop a related upper bound on distortion of random codebook quantization. Second, we improve the computability of both the lower and upper bounds, using a graph-based adaptive importance sampling approach. Third, we leverage the bounds to define new notions of intrinsic dimension which form upper and lower bounds on the Rényi information dimension, rate-distortion dimension, and quantization dimension, and which can be empirically estimated from unknown data sources. We illustrate the concepts by analyzing several artificial and empirical distributions.

0.26

A Unified Rate-Distortion Perspective on Vector, Product, and Scalar Quantization

cs.LG cs.CV ·2026-09-02 ·Xianghong Fang, Wenlong Mou, Yuan Yuan, Dehan Kong, Tim G. J. Rudner
similarity 0.66
Abstract · 26 pages, 2 figure, 8 tables

Discrete visual tokenization, predominantly driven by vector, scalar, and product quantization, lacks a unified conceptual framework for understanding quantization tradeoffs. In this paper, we propose a unified rate--distortion perspective on modern discrete visual tokenization. By viewing quantization as lossy compression, we characterize the nominal fixed-length coding rate through token count and codebook size, and quantization error as the distortion. Within this framework, we resolve three central questions. First, we theoretically and empirically show that minimizing distortion, rather than maximizing codebook utilization, is the primary intrinsic objective for reconstruction fidelity, with a direct connection to the STE-induced gradient discrepancy. Second, we establish two critical fairness conditions for intrinsic quantization comparison: controlling latent feature statistics and enforcing identical coding rates. Third, under these conditions, we recover the VQ--PQ--SQ distortion hierarchy in modern visual tokenization and show empirically that modern VQ methods achieve the lowest distortion. This work provides a foundational rate--distortion reframing of modern discrete visual tokenization, resolves ambiguities in quantizer evaluation, and provides a controlled framework for isolating intrinsic quantization effectiveness under fixed-rate constraints.

0.26

Third-Order Asymptotics for Fixed-Length Lossy Compression of Discrete Memoryless Sources

cs.IT ·2026-11-10 ·Vincent Y. F. Tan
hot direction similarity 0.61
Abstract · 16 pages, 2 figures

This paper determines the third-order asymptotics of the logarithm of the minimum codebook size for fixed-length lossy compression of discrete memoryless sources. For an excess-distortion probability constraint $\varepsilon\in(0,1)$ and every interior distortion level, we establish a third-order correction of $\frac12\log n+O(1)$, where $n$ is the blocklength. This sharpens the general bounds of Kostina and Verdú [IEEE Trans. Inf. Theory, 2012]. The main technical innovation lies in the converse for positive dispersion. Under a reverse change of measure, we retain the exponential weights associated with tilted information and distortion slack and control them jointly using the multivariate weighted-strip estimate of Scarlett, Martinez, and Guillén i Fàbregas [Proc. IEEE ISIT, 2015]. A pairing argument verifies the required covariance nondegeneracy uniformly over all reconstruction words whose distortion balls meet a typical set, yielding the matching converse.

0.31

Lossy Compression via Sparse Regression Codes: Generalized Construction and Finite-length Bounds

cs.IT ·2026-08-14 ·Galen Reeves, Ramji Venkataramanan
similarity 0.61
Abstract

We study sparse regression codes (SPARCs) for lossy compression under simple greedy encoding rules, including both correlation-based and distance-based methods. We generalize the SPARC construction, and consider the class of \emph{additive orthogonal} regression codes, of which standard SPARCs are a special case. For this class of codes, we derive nonasymptotic bounds on the squared-error distortion by tracking the evolution of the encoding residual across stages. Our results highlight the role of power allocation in controlling the distortion, allowing us to optimize the allocation based on the parameters of the code. The optimized allocation improves the finite-length compression performance of SPARCs, and our bounds provide distortion guarantees for lower complexity variants of SPARCs, like signed SPARCs and $K$-sparse SPARCs.

0.04

Fast Quantized Johnson$-$Lindenstrauss Transforms

cs.LG cs.DS ·2026-10-12 ·Angelos Korakitis, Vasileios Nakos
theory similarity 0.60
Abstract

Quantization has emerged as one of the primary techniques for reducing the memory footprint of large language models during inference. A prominent example is Quantized Johnson$-$Lindenstrauss (QJL), which compresses KV caches using 1-bit asymmetric quantization while approximately preserving the inner products required by the attention mechanism. Recent theoretical and practical work aims at replacing the dense Gaussian projections used by the aforementioned transforms with more structured embeddings $-$ such as compositions of randomized Hadamard transforms $-$ while retaining the same embedding dimension. In this paper, we introduce two constructions based on a composition of randomized Hadamard transforms, albeit with a Gaussian rather than Rademacher diagonal in the second stage. The first adds Gaussian dithering to smooth the quantizer, while the second inserts a random permutation to regularize the covariance structure of the resulting Gaussian measurements. We prove embedding-dimension and running-time guarantees analogous to those of Fast JL in the non-quantized setting. The first construction achieves $m=O(\varepsilon^{-2}\log^2(d/δ))$, while the second achieves $m=O(\varepsilon^{-2}\log(d/δ))$ under a restriction on $m$. Both constructions can be applied in $O(d\log d)$ time.

0.15

Differentiable Bit-Widths: Co-optimizing Pruning and Quantization via SVD for Ultra-Efficient LLM Compression

cs.CL cs.AI ·2026-10-05 ·Hankyul Kang, Jongbin Ryu
top venuecode/datahot direction similarity 0.60
Abstract · Accepted to Advances in Neural Information Processing Systems (NeurIPS 2026)

SVD-based pruning and quantization have recently emerged as a promising strategy for the ultra-efficient compression of large language models. In these methods, compression is performed in two stages: components are first truncated, and the remaining ones are subsequently quantized. Although this decoupled pipeline benefits from both pruning and quantization, it requires separate optimization for each stage and fails to fully exploit their balance, which can lead to suboptimal performance under aggressive compression. To address this limitation, we propose a new LLM compression method that co-optimizes pruning and quantization in a unified framework. Our key idea is a differentiable method for learning component-wise bit-widths, allowing less important components to be assigned 0-bit precision and pruned away. Notably, our method performs favorably against two-stage baselines, even when subjected to extreme quantization settings ($1.61$ bits) designed for ultra-efficiency. Code: https://github.com/MMAI-Laboratory/DBW.

0.52

Practical Semantic Communication for General-Purpose Networks: Compression of High-Dimensional Embeddings Under Quantization

cs.IT cs.LG ·2026-11-16 ·Leonardo Roque, Louis-Adrien Dufrène, Quentin Lampin, Grégoire Lefebvre, Mohamad Assaad
hot direction similarity 0.59
Abstract

This paper introduces a practical framework for semantic communication over wireless channels, focusing on the compression of high-dimensional embeddings under quantization and channel errors. Building upon a relaxed Gaussian Information Bottleneck (GIB) model, the approach provides analytical solutions for task-relevant information preservation, even with non-Gaussian targets. The framework relaxes traditional Gaussian assumptions, enabling effective compression for discrete and complex data such as text embeddings. It derives closed-form gradients for optimizing the compression matrix and establishes bounds linking continuous representations to discrete, noisy system performance. Comprehensive experiments on text classification tasks demonstrate that the proposed method outperforms task-agnostic quantization-based compression methods, maintaining high semantic fidelity at low bit budgets per embedding. The approach offers a scalable, interpretable, and resource-efficient solution for deploying semantic communication in general-purpose networks.

0.43

Maximal correlation under cardinality constraints

cs.IT ·2026-08-17 ·Dror Drach, Tomer Berg, Or Ordentlich, Ofer Shayevitz
theory similarity 0.59
Abstract

In this paper, we define and analyze the quantized maximal correlation, an extension of the notion of maximal correlation restricted to functions taking values in sets of bounded cardinality. We derive an upper bound on the quantized maximal correlation by showing that the correlation between any quantized functions of $X$ and $Y$ is related to the MMSE distortion in quantization of a particular linear combination of random variables. Following this, we leverage rate-distortion techniques and anti-concentration inequalities to further bound this MMSE, which results in explicit bounds on the quantized maximal correlation. Unlike the quantized maximal correlation itself, which does not generally tensorize, our bounds on the mean squared error do tensorize, resulting in a dimension-free upper bound on the quantized maximal correlation for product distributions. Our results also lead to improved bounds on the isoperimetric constants of reversible Markov chains and product chains, strengthening classical results such as those by Alon and Milman.

0.31