CS Paper Radar

Relative Entropy under Ordered Chi-Squared Constraints: Exact Bounds and Extremizers

Igal Sason

cs.IT ·submitted 2026-11-18 15:30 UTC · v1
theory

Abstract

We derive a sharp upper bound on the relative entropy $D(P\|Q)$ in terms of the two ordered chi-squared divergences $χ^2(P\|Q)$ and $χ^2(Q\|P)$. For any $A,B>0$, the constraints $χ^2(P\|Q)\leq A$ and $χ^2(Q\|P)\leq B$ yield upper bounds on $D(P\|Q)$ and $D(Q\|P)$. Two pairs of Bernoulli distributions each attain both bounds simultaneously. As a consequence, for sequences of probability mass functions $\{P_n\}$ and $\{Q_n\}$ on a common finite or countably infinite alphabet, both $D(P_n\|Q_n)$ and $D(Q_n\|P_n)$ tend to zero whenever one of the two ordered chi-squared divergences tends to zero and the other remains uniformly bounded. Without this boundedness assumption, $D(P_n\|Q_n)$ can fail to converge to zero even when $χ^2(Q_n\|P_n)\to 0$. Finally, in contrast to the attainable upper bounds, on any finite or countably infinite alphabet containing at least three elements, the infimum of $D(P\|Q)$ under the exact constraints $χ^2(P\|Q)=A$ and $χ^2(Q\|P)=B$ is zero and is not attained.

Closest papers

Nearest neighbours by meaning across the whole index.

Sharp Rogers--Shephard Inequalities for Rényi Entropy

cs.IT math.PRmath.MG ·2026-10-11 ·Bang-Xian Han, Guoqing Zhan, Chiyu Zhou, Zhuo-Nan Zhu
theory similarity 0.62
Abstract · 29 pages; comments are welcome!

We establish sharp weighted Rogers--Shephard inequalities for Rényi differential entropy. For independent copies $X,X'$ of a log-concave random vector in $\R^n$, we prove $h_p(aX-bX')\le h_p(X)+n\log(a+b)$ for all $a,b>0$ and $1\le p\le\infty$, where orders $1$ and $\infty$ correspond to Shannon entropy and min-entropy, respectively. The Shannon case settles the entropic Rogers--Shephard conjecture of Madiman and Kontoyiannis and its higher-dimensional formulation. For $1\le p<\infty$, equality holds precisely for invertible affine images of random vectors with independent exponential coordinates. At $p=\infty$, the larger equality class is characterized by possibly extended-valued convex potentials that are positively homogeneous of degree one. For every $0<p<1$, uniform laws on sufficiently high-dimensional simplices violate the unweighted bound. Thus $p=1$ is the critical order for this bound to hold in every dimension. The proof combines weighted convex intersections with an exponential lift to a log-affine probability density. At the Shannon endpoint, this lift yields an exact deficit decomposition with a conditional mutual information term for an additive channel with asymmetric Laplace noise.

0.43

On Measures of Dissimilarity on the Majorization Simplex

cs.IT ·2026-10-12 ·Tarun Hariharan, Prasad Krishnan, Abhishek Deshpande
theory similarity 0.60
Abstract

The majorization simplex $\mathcal{P}_n$ is the set of probability vectors of length $n$ whose entries are arranged in non-increasing order. Under the majorization order, $\mathcal{P}_n$ is a lattice, and it is the natural space for comparing distributions when the labels of the outcomes are not relevant. In this work, we study measures of dissimilarity (and, similarity) on $\mathcal{P}_n$. Our first set of results concerns the Rényi-entropy based distance $d_α(p,q)=H_α(p)+H_α(q)-2H_α(p\vee q)$ proposed by Yadav and Shkel (2026), which generalizes a metric of Cicalese, Gargano and Vaccaro (2013). We show that $d_α$ is a metric on $\mathcal{P}_n$ for $α\in[1,\infty)$, while for $α\in[0,1)\cup\{\infty\}$, $d_α$ is not a metric in general. Our main result in this part identifies the necessary and sufficient conditions for equality in the triangle inequality of $d_α$ when $α>1$, for which we identify the equality conditions for the supermodularity of the Rényi entropy $H_α$. Our second set of results concerns the total variation distance and the Hellinger affinity (also called the Bhattacharyya coefficient) on $\mathcal{P}_n$. We obtain a number of bounds on these measures, along with some of their equality conditions. The resulting Hellinger affinity based upper bound on the total variation distance can be smaller than the bounds obtained from Pinsker's and the Bretagnolle--Huber inequalities.

0.24

Hermite-Fisher bounds and stability for min-entropy power inequalities

cs.IT math.FAmath.MGmath.PR ·2026-09-18 ·Silouanos Brazitikos, Martin Rapaport, Tomasz Tkocz
similarity 0.60
Abstract

We derive explicit lower bounds for relative Fisher information by combining a variational principle with suitably orthogonalized Hermite-polynomial test functions. The resulting cumulant bounds are asymptotically sharp and yield lower bounds for Gaussian entropy deficits. We also establish quantitative versions of sharp min-entropy power inequalities in all dimensions. En route, we develop a stability result for Brzezinski's sharp bound for block sections of products of Euclidean balls, which may be of independent interest.

0.39

Contraction of Rényi Divergences for Discrete Channels

cs.IT math.PR ·2026-09-28 ·Adrien Vandenbroucque, Amedeo Roberto Esposito, Michael Gastpar
highly cited similarity 0.58
Abstract

We investigate Strong Data-Processing Inequality (SDPI) constants for Rényi Divergences on finite spaces. We study their dependence on the Rényi order $α$, proving that they are non-decreasing and that their scaling by $(α-1)$ is convex for $α\geq1$. We also identify several support restrictions on the probability measures involved in determining these constants. In particular, in the distribution-independent setting, measures supported on a common set of at most two points suffice to evaluate the Rényi-SDPI constant. For $α\in[0,1]$, we further prove equality with the $χ^2$-SDPI constant, while at order infinity we obtain a closed-form expression. In order to link contraction over product spaces to contraction along individual coordinates, we provide tensorisation bounds for arbitrary product channels analogous to those known for $\varphi$-Divergences. At finite orders, the Rényi-SDPI constants are bounded above and below through comparisons with the $χ^2$-Divergence and Hellinger Divergences, with sharpness established in multiple cases. At order infinity, we instead relate these constants to the contraction of Total Variation Distance. Finally, our findings are applied to local differential privacy (LDP) and the analysis of Markov chains. This yields sharp contraction guarantees for pure-LDP mechanisms, and connects Rényi-LDP to Rényi-SDPI constants. For Markov chains, we derive finite-time convergence bounds and exhibit a family of chains for which Rényi-SDPIs improve on classical $χ^2$-based bounds by arbitrarily large factors.

0.33

On the Cardinality of Optimal Representations in the Binary-Source Information Bottleneck

cs.IT cs.LGmath.OC ·2026-10-05 ·Dier Tang, Jun Chen
similarity 0.57
Abstract · 9 pages. Feedback and comments are welcome!

The information bottleneck (IB) seeks a representation $U$ of a source $X$ that retains as much information as possible about a target $Y$, subject to a constraint on $I(U;X)$. A classical argument shows that it suffices to consider representations with at most $|\mathcal{X}|+1$ symbols, and this bound is known to be tight whenever $|\mathcal{X}| \geq 3$. We show that the binary case behaves differently: if $X$ is binary and $Y$ is finite, then for every joint distribution of $(X,Y)$ and every rate constraint, the IB optimum is attained by a binary $U$. Hence the bound $|\mathcal{U}| \leq |\mathcal{X}|+1$ sharpens to $|\mathcal{U}| \leq |\mathcal{X}|$ for binary sources. The proof combines a separating hyperplane argument with the observation that, for a binary source, the ratio of the second derivatives of the two entropy functions involved is concave.

0.16

Entropic Inequalities and Limit Theorems in Boolean Probability

cs.IT math.PR ·2026-10-14 ·Kewei Pan
theory similarity 0.56
Abstract

We study Boolean entropy and Fisher information in information inequalities and the Boolean central limit theorem. We prove a fractional Stam inequality for the Boolean Fisher information associated with arbitrary fractional partitions of subset sums, and derive an entropy power inequality for sums of Boolean independent random variables and its weighted version. We then establish a Boolean analogue of Barron's entropy convergence result along the Boolean central limit theorem: Boolean entropy converges to the entropy of the Rademacher law if and only if the initial entropy is finite. Furthermore, under a finite fourth moment, in the Boolean central limit theorem, we obtain \[ Γ(μ_n\mid\mathbf b)= \frac{m_4(μ)-1-m_3(μ)^2}{2}\frac{\log n}{n} +o\!\left(\frac{\log n}{n}\right), \] with an explicit example showing sharpness. For symmetric laws with finite fourth moment and finite Boolean Fisher information, \[ Ψ(μ_n\mid\mathbf b) =2\bigl(m_4(μ)-1\bigr)\frac{\log n}{n} +o\left(\frac{\log n}{n}\right). \] Our approach is based on the Nevanlinna representation of Cauchy transforms.

0.27

Bounded Relative Boundary Implies Narrow DNF Approximation

cs.CC cs.DM ·2026-08-31 ·Chenghua Liu, Boning Meng
theory similarity 0.55
Abstract · 13 pages; proof and exposition revised, Lean repository linked

We prove Friedgut's conjecture (J. Amer. Math. Soc. 12 (1999)) that every increasing family in the $p$-biased discrete cube with bounded relative boundary can be approximated arbitrarily well by one whose minimal elements have size bounded independently of the dimension and the bias. For $0<p\le 1/2$, every increasing Boolean function with total resampling influence at most $K$ is $\varepsilon$-close under $μ_p^n$ to a monotone DNF of width $\exp(O((K+1)^2/\varepsilon^2))$, from which the full conjecture follows for all $p\in(0,1)$. We build on Hatami's pseudo-junta theorem (Ann. of Math. 176 (2012)), whose approximator need not be increasing. Our randomized shifts produce an increasing approximator that is measurable with respect to a controlled refinement of Hatami's adaptive partition. Retaining only short positive certificates yields the desired narrow DNF.

0.35

The probabilistic bound for perfect hashing is sharp only for binary alphabets

cs.IT math.CO ·2026-11-05 ·Shitao Li, Minjia Shi
theory similarity 0.55
Abstract

We investigate the optimal asymptotic rate $R_q$ of perfect $q$-hash codes, a fundamental extremal problem in combinatorics, information theory, and computer science. Let $B_q=-\log_2(1-q!/q^q)/(q-1)$ denote the classical probabilistic lower bound. We prove that $R_q=B_q$ if and only if $q=2$, establishing an explicit strict improvement for every $q\ge3$ and settling the first assertion of a conjecture of Xing and Yuan (Combinatorica 43 (2023), 347--366). More significantly, we show that, for every $η>0$, $R_q\ge B_q+\exp(-(2+η)q)$ for all sufficiently large integers $q$. This improves the previously known increments of scale $\exp(-3q+O(\ln q))$ by an exponential factor. We also establish a structural limitation of the concatenation method. For pairwise uniform inner codes of subexponential length, the resulting rate bound exceeds $B_q$ by at most $\exp(-3q+o(q))$. Thus, exponentially long inner codes are necessary to achieve the new improvement scale within this framework. Our proofs combine Fourier analysis, high-dimensional simplex codes, additive permutation counts, and entropy bounds for hypergraph matchings. We further strengthen lower bounds from short inner codes and improve several known numerical estimates.

0.43