CS Paper Radar

Deterministic Causal Quantization with Bounded Error: Adaptive Reconstruction and Entropy bounds

Rama Cont, Massoud Heidari

cs.IT math.PR ·submitted 2026-10-20 11:51 UTC · v2
theory

Abstract

We study compression of real-valued stationary sources under a deterministic error bound: every sample must be reconstructed within a prescribed tolerance $\varepsilon$ on every input sequence. Reconstructions are causal, depending only on present and past inputs. Rates are measured by the entropy of the quantizer output, which block entropy coding can approach, and compared with the minimum rate of unrestricted block quantization under the same error constraint. A uniform scalar quantizer with cells of width $2\varepsilon$ is within one bit per sample of this minimum whenever finite-entropy scalar quantization is possible, without density, independence, or mixing assumptions. Under entropy and Sobolev regularity conditions, the gap is at most $2\mathcal{J}\varepsilon^2/(π^2\ln 2)$, where $\mathcal{J}$ is the asymptotic Fisher information per sample. Thus the cost of causal reconstruction vanishes at a quadratic rate as the tolerance decreases. A uniform grid, however, cannot adapt to the source. We construct an adaptive quantizer whose encoder and decoder maintain a common finite set of reconstruction values, updated from the transmitted symbols alone; suitable initialization reproduces any uniform grid. When its initial values are at least $\varepsilon$ apart, its rate is within two bits per sample of the minimum for every such source. For sources uniform within separated intervals of width $2\varepsilon$, storing the interval centers attains the minimum and its Shannon lower bound. Such configurations can outperform every uniform grid of the same cell width: over sources with well-separated intervals, the best uniform grid's worst-case excess rate is $1/(2\ln 2)\approx 0.72$ bit per sample.

Comment
41 pages

Closest papers

Nearest neighbours by meaning across the whole index.

Entropy-Constrained Adaptive Stochastic Quantization

cs.LG cs.AIcs.DScs.IT ·2026-08-10 ·Ran Ben Basat, Yaniv Ben-Itzhak, Michael Mitzenmacher, Shay Vargaftik
similarity 0.63
Abstract

Adaptive stochastic quantization (ASQ) is a recently introduced quantization approach that optimizes the Mean Squared Error (MSE) for a given input while preserving unbiasedness. It is designed to alleviate the communication and memory bottlenecks of modern data and machine learning workloads, including model, gradient, and KV-cache compression and nearest-neighbor search. Further, practical systems can then compress quantized data with a lossless entropy encoder. However, existing unbiased methods, including ASQ, choose their quantization values without considering this later encoding stage, leaving accuracy on the table. We formulate the Entropy Constrained Adaptive Stochastic Quantization (ECASQ) problem, which jointly selects adaptive quantization values to minimize MSE under an entropy budget and an unbiasedness constraint. We give an optimal dynamic program with $O(sd^2)$ time and $O(d^2)$ space for a length-d vector and at most s quantization values, and a GPU-friendly approximate dynamic program with $O(sd^2)$ time and $O(d)$ space. The approximation guarantees that the solution has an MSE no larger than the optimal solution that uses one fewer bit of entropy per entry. We also provide an iterative refinement procedure for the approximation solution that, in our experiments, yields near-optimal results while retaining a substantial speed advantage over our solver for the optimal solution.

0.34

Optimal Min-Entropy Estimation with High Probability

cs.IT math.STquant-ph ·2026-11-17 ·Qisheng Wang
similarity 0.60
Abstract · 9 pages

Min-entropy plays a fundamental role in assessing the unpredictability of randomness sources. In particular, NIST SP 800-90B specifies min-entropy estimation as a key procedure for entropy sources used in cryptographic random bit generators, illustrating the practical importance of reliable min-entropy estimation. In this paper, we settle the sample complexity of min-entropy estimation with high probability in both the classical and quantum cases. Specifically, for min-entropy estimation with additive error $\varepsilon$ and probability at least $1-δ$, the sample complexity is $Θ(d\log(d/δ)/\varepsilon^2)$ for classical distributions on $d$ symbols and $Θ((d^2+d\log(1/δ))/\varepsilon^2)$ for quantum states on a $d$-dimensional Hilbert space. Our results admit short and simple proofs.

0.37

Optimal compression with quantum retrieval

cs.DS quant-phcs.IRcs.IT ·2026-10-05 ·Shyam Dhamapurkar, Mohit Garg, Manaswi Paraashar, Jaikumar Radhakrishnan
similarity 0.59
Abstract · 13 pages, 3 figures

We consider the following data compression problem. Given a string $x \in \{0,1\}^m$ of Hamming weight at most $n$, compress it into a shorter string $y \in \{0,1\}^s$ so that any bit $x_i$ of $x$ can be retrieved without any error using at most $t$ quantum queries to the standard oracle encoding of $y$. If queries are allowed to be adaptive we show how optimal compression up to a logarithmic factor can be achieved. If the queries are required to be made non-adaptively, we show schemes whose space is optimal in its dependence on $m$ except for a logarithmic factor, and is at most quadratically worse when compared to the optimum in its dependence on $n$.

0.12

Minimax Quantile Bounds via Information Measures

cs.IT math.ST ·2026-08-21 ·Amedeo Roberto Esposito
similarity 0.59
Abstract

We develop a unified information-theoretic framework for lower bounding minimax quantiles. The starting point is a loss-adapted Neyman--Pearson metaconverse that bounds the minimax success probability at every loss threshold and confidence level. The bound separates the small-ball behaviour of the prior under the loss from the statistical distinguishability of the observation model, and is optimised over an auxiliary output distribution. Different relaxations of this Neyman--Pearson bound yield converses based on \(f\)-informativity, Sibson mutual information \(I_α\), Maximal Leakage, and Amemiya norms. Classical Fano and Le Cam lower bounds are recovered as special cases. The framework also clarifies why different information measures are suited to different recovery criteria. Maximal Leakage is exact for a class of symmetric exact-recovery problems. We use this identity to derive finite-sample bounds on the full minimax exact-recovery risk in the balanced Gaussian weighted stochastic block model, as well as two-sided finite-sample minimax-quantile bounds for low-rank matrix estimation under isotropic bounded-energy noise. For approximate Hamming recovery, we exhibit a heterogeneous binary model in which an optimised finite Sibson order yields a strong converse while the Maximal Leakage specialisation is trivial. Finally, for one-coordinate Poisson localisation, a Bennett-type Young function used through its Amemiya norm recovers the exact success-probability scale, whereas classical Fano and fixed-power relaxations are strictly weaker. These results show that sharp converses for minimax quantiles require adapting the information measure to the recovery resolution, whether exact or approximate, and to the tail behaviour of the likelihood ratio.

0.45

From Pixels to Privacy-Sensitive Features: An Information-Theoretic Framework for Reconstruction Bounds

cs.LG stat.ML ·2026-10-09 ·Henry Bourne, Robert F. Allison
theory similarity 0.58
Abstract · 40 pages, 9 figures

What can an adversary recover from a model trained with DP-SGD? Its $(\varepsilon,δ)$ guarantee does not answer this question directly. We derive a distortion bound that can be computed before training and applies even to an adversary who observes the parameter trajectory during training, knows every other training record and the population distribution from which the training records are drawn. The analyst may choose the representation in which error is measured, allowing the bound to target the aspects of reconstruction that are privacy-sensitive in a given application. Our information-theoretic framework separates two contributions: the information about a record revealed by training, measured through conditional mutual information, and the uncertainty already present in the data distribution, measured through source entropy. Rate-distortion theory combines them into a lower bound on the expected error of every reconstruction attack. For DP-SGD, the leakage bound depends only on the sampling rates and noise multipliers, while the source entropy is lower-bounded from a sample with finite-sample confidence. An orthogonal block construction keeps the resulting bound informative for high-dimensional images. We instantiate the framework in pixel space and in LPIPS feature space, obtaining, to our knowledge, the first lower bound on reconstruction distortion measured directly by LPIPS. On a range of image datasets the bounds establish non-trivial unavoidable expected reconstruction error within heuristically recommended privacy-budget ranges and beyond. Experiments compare the bounds with strong reconstruction attacks and identify the principal sources of the remaining slack.

0.22

A Quantization Problem Posed by Adaptive Streaming

cs.IT ·2026-09-03 ·Yuriy A. Reznik
similarity 0.58
Abstract

In adaptive bitrate (ABR) streaming, the delivery technology behind most Internet video, each title is encoded at several bitrates, forming an \emph{encoding ladder} of \emph{renditions}. Each client plays the highest rendition that its network bandwidth can sustain. We show that choosing the ladder is a problem of \emph{scalar quantization} of the bandwidth distribution. However, this quantization problem is of an unusual kind: the client's logic pins each quantization cell's reproduction value to the cell's \emph{left edge}, and the distortion measure is a one-sided quality loss rather than a squared error. The constraint reshapes the classical theory. The \emph{quality gap} of a ladder is the distance between its average delivered quality and the \emph{quality limit} that infinitely many renditions would attain. For optimal $n$-rendition ladders the gap $\mathcal{G}_n^*$ decays as $Θ(1/n)$, not the textbook $Θ(1/n^2)$. Moreover, with large $n$, $n\,\mathcal{G}_n^*\to C_w=\frac{1}{2}(\int\!\sqrt{p\,Q'}\,\,d B)^2$, with the limit attained when the ladder's rates follow the density $\sqrt{p\,Q'}$, which replaces Panter--Dite's $p^{1/3}$; here $p$ is the bandwidth density and $Q$ the quality--rate curve of the content. The law yields closed-form design rules: quantile rung placement, the rendition count $n(\varepsilon)\approx C_w/\varepsilon$ needed to reach tolerance $\varepsilon$, and the economic ladder size $n^*=\sqrt{λC_w/κ}$ when a rendition costs $κ$ to operate and the quality gap is priced at $λ$. The same analysis extends to the design of ladders employing different video resolutions, codecs, perceptual quality metrics, and the two-dimensional adaptation logic of modern web players.

0.21

Learning Finite-Memory Binary Sources

cs.IT math.PR ·2026-10-29 ·Maciej Skorski
similarity 0.58
Abstract

Estimating biased finite-memory sources from a single trajectory is a fundamental problem with practical impact, for example in estimating entropy for cryptography. We study order-$d$ binary sources whose conditional probabilities have bias at most $ε$, and establish *sharp* sample complexity bounds: relative accuracy $γ$ with confidence $1-δ$ needs $γ^{-2}\,(\tfrac12-ε)^{-d}\,(\log d+\log\tfrac1δ)$ samples up to constants depending on $ε$, with matching upper and lower bounds and no dependence on mixing time or spectral gap. The proof rests on a sharp bound on expected hitting times of the sliding-window (de Bruijn) chain, which stay of order $1/π(w)$ uniformly in $d$ exactly when $ε<\tfrac3{10}$, and on majorization of the window masses by those of independent biased coins. Applied to published entropy-source validations, the bounds show that their trace lengths may be underpowered: $10^6$ bits cannot resolve all transitions of an $8$-bit model to better than about $4.5\%$ relative error.

0.17

When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

cs.AI cs.LGstat.ML ·2026-10-08 ·Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu, Baihua He, Xinyu Zhang +4 more
similarity 0.58
Abstract

Weight-only post-training quantization (PTQ) relies heavily on reconstruction loss minimization to preserve model quality at low precision. We show that the weights favored by minimizing this loss need not yield better model performance on new tasks. In fact, we find that lower reconstruction loss can even degrade model performance on the same calibration data. Our analysis further shows that weights with lower reconstruction loss on calibration data can have higher loss than other weights when the distribution of input activations changes. Motivated by these observations and our analysis, we propose Distributionally Robust Quantization (DRQ), a post-hoc refinement process that minimizes worst-case reconstruction loss over a constrained set of input activation distributions. DRQ refines the integer codes representing quantized weights within the existing quantization grid, keeping quantization parameters and inference operators unchanged. Extensive experiments show that DRQ improves models quantized by six representative PTQ methods, including AWQ, GPTQ, and ParoQuant, and delivers gains across both dense and mixture-of-experts large language models. These results establish DRQ as a general post-hoc refinement framework for weight-only PTQ, achieving better downstream performance without adding inference overhead.

0.17