CS Paper Radar

Redundancy and synergy in multivariate Gaussians via the Blackwell order

Artemy Kolchinsky

cs.IT stat.ML ·submitted 2026-10-05 20:27 UTC · v1
theory

Abstract

The goal of the partial information decomposition (PID) is to quantify the redundant and synergistic information that multiple sources provide about a target. PID has many applications in machine learning, neuroscience, and other fields, but defining and computing it for high-dimensional continuous systems remains challenging. Here, we define a PID for multivariate Gaussian systems based on the Blackwell order, which formalizes when one channel is more informative than another. We prove that Gaussian channels are optimal for extracting both redundant and union information, yielding an intuitive geometric interpretation and an efficient numerical algorithm for the PID. Our union information and synergy coincide with the well-known BROJA measures, and we derive closed-form expressions for both in the case of two sources. We also argue that Blackwell redundancy (which differs from BROJA) is the only existing redundancy measure that satisfies a set of natural desiderata. We demonstrate the scalability of our method on systems with up to a thousand dimensions or sources. Our approach is illustrated on an optimal control problem, where it identifies redundant and synergistic interactions between sensor and memory.

Closest papers

Nearest neighbours by meaning across the whole index.

Gaussian BROJA Partial Information Decomposition in Closed Form

cs.IT stat.ML ·2026-10-09 ·Ohad Sheelo, Tal Golan, Ari Pakman
code/data similarity 0.71
Abstract · 28 pages total, including 10 pages of main text, 2 main figures, 5 supplementary sections, and 5 supplementary figures

The BROJA partial information decomposition provides an operationally motivated way to separate the information that two sources carry about a target into unique, redundant, and synergistic components. For continuous and high-dimensional variables, however, computing the BROJA decomposition requires solving a difficult optimization problem. Previous work showed that, when each source-target pair is jointly Gaussian, the problem can be reduced to finite-dimensional numerical optimization. Here, we derive a closed-form solution to this optimization problem, enabling direct computation of the BROJA decomposition in the general multivariate Gaussian setting. The resulting closed-form expression enables speedups exceeding two orders of magnitude for high-dimensional Gaussian BROJA PID. Code is available at https://github.com/brainsandmachines/Eigen_PID.

0.35

Between Hi-Fi and MMI: the overlap of redundancy and synergy in partial information decomposition

cs.IT physics.data-anstat.ME ·2026-11-05 ·Niels Van Santen, Yves Rosseel, Daniele Marinazzo
similarity 0.65
Abstract

Partial information decomposition (PID) splits the information that two sources carry about a target into redundant, unique and synergistic parts, with nonnegative decompositions lying on a segment where redundancy measures are bounded by the positive part of the co-information and the smaller source-target information. We parameterize this segment by the overlap, the common excess of redundancy and synergy over their net balance, and show that for many of the named PID measures, their redundancy can be written as the largest positive part of the co-information over a set of reference distributions. This common form induces an ordering of those PIDs on the segment. With the smallest redundancy and synergy and the largest unique atoms of any nonnegative PID, the information-theoretic version of Hi-Fi, a feature-importance method, is the zero-overlap end of the segment for two sources, with the maximum overlap bound given by the minimum mutual information (MMI) PID. We introduce, by splitting the co-information into a pairwise maximum- entropy part and the third-order connected information, a PID that sits at zero overlap except when pairwise redundancy and triadic synergy coexist. For three or more sources the bivariate results hold on every pair of merged source groups, and among all nonnegative solutions of the lattice equations MMI and Hi-Fi still give the smallest and largest unique atoms but zero overlap on all pairs at once is often unattainable. Two inequalities across pairs are necessary in general and sufficient for three sources. For identity targets zero overlap is possible exactly when the sources' information diagram is nonnegative. In random three- and four-source distributions zero overlap on all pairs fails in 21-100% of the cases.

0.15

From Common Information to Canonical Correlation Analysis and Beyond: Bridging Information Theory, Signal Processing, and Machine Learning

cs.IT cs.LGeess.SPstat.ML ·2026-10-20 ·Paris A. Karakasis, Walid Saad
published similarity 0.59
Abstract · Tutorial, 100 pages, 11 figures. Accepted for publication in Entropy, Special Issue "Foundations and Frontiers of Information Theory" in hon…

Many problems in signal processing, machine learning, and neuroimaging provide several jointly observed measurements of one process, each mixing a component common to all of them with a component private to itself, and require the common component to be recovered without supervision. This problem has been studied in several communities, largely independently. Statisticians proposed canonical correlation analysis (CCA), introduced by Hotelling in 1936. Information theorists proposed two notions of common information, due to Gács and Körner and to Wyner. Engineers developed subspace methods for sensor arrays and multiuser communications to design multiview receivers. This tutorial presents the main results and tools each community developed for this problem in a single exposition, and explains the connections that emerge where they meet, distinguishing exact equivalences from special cases and analogies. We develop the information-theoretic foundation from first principles. We then introduce CCA and its extension to several views, followed by the engineering thread from blind multiuser detection to CCA as a blind receiver. Next, we turn to nonlinear CCA, covering kernel, deep, and probabilistic formulations, the conditions under which the common factor is identifiable, and connections to self-supervised learning. We conclude with a discussion of further extensions and open directions. The exposition is self-contained, so that readers from each of these communities can follow the parts developed by the others.

0.39

Minimum Rate For Partially Observable Linear System with Side Information: LQG Plant and Gaussian-Markov Source

cs.IT eess.SY ·2026-08-27 ·Sijie Li, Hyeji Kim
publishedtheory similarity 0.57
Abstract · accepted for CDC 2026, full version, modified a few typos, polished more

This paper studies the minimum rate required for a partially observable linear system with side information. The Linear Quadratic Gaussian(LQG) plant and the Gaussian-Markov source are considered. We show that a class of linear policies is sufficient for optimizing the conditional directed information lower bound. We also show that the resulting optimization problem is convex for the scalar case in both time-varying and time-invariant systems. Our results generalize the past works that consider the case with full or partial observation only, and the case with full observation and side information. Numerical simulations are presented to illustrate the effect of side information for partially observable systems.

0.33

Gaussian entanglement and quantum communication: geodesic optimization and multimode bounds

cs.IT quant-ph ·2026-10-13 ·Hemant K. Mishra, Mark M. Wilde
similarity 0.56
Abstract · 80 pages, 12 figures

Loss and noise limit the use of light to distribute entanglement, transmit quantum information, and establish secret keys. Gaussian states have a compact description in terms of means and covariance matrices, but their entanglement still requires global optimization. We develop a framework for computing Gaussian-restricted versions of the relative entropy of entanglement and Rains entanglement. An integral representation reveals a convex geometry under matrix geometric means and reduces Gaussian Rains entanglement to relative entropy minimized over Gaussian states with positive partial transpose. We obtain unique optimal reference states and additivity for independent Gaussian input states. Our convex-majorization algorithm has an explicit geometric convergence rate, uniform in the faithful regularization parameter and valid for pure input states. Controlled numerical integration and approximate convex subproblem solutions give a bit-complexity bound that is polynomial in accuracy bits for each fixed rational input covariance matrix, with explicit dependence on covariance size. These calculations bound extractable entanglement and, through the relative entropy of entanglement, extractable secret key. Teleportation simulation, energy allocation, and a convex optimization of passive-basis loss cores give bounds on quantum and private communication under an input-energy constraint, allowing arbitrary two-way classically assisted protocols. Two- and three-mode examples compare the bounds with achievable rates, including matching pure-loss benchmarks. Finally, we separate the Gaussian-restricted quantities from their unrestricted counterparts and give a photon-number truncation method with controlled error for computing the latter.

0.58

Beyond Gaussian Assumptions: Distribution-Aware Channel Capacity for Effective Connectivity

cs.LG q-bio.NC ·2026-09-26 ·Jianan Jian, Jacob Kang, Nurahmed Multezem, Benjamin Li, Nan Xu
similarity 0.55
Abstract · 31 pages, 9 figures, 5 tables

Effective-connectivity estimation from brain signals often relies on Gaussian residual modeling, which enables tractable estimation but can discard informative distributional structure and distort inferred directed interactions when empirical residuals are non-Gaussian. We show across multiple modalities, species, and experimental conditions that both brain signals and fitted channel residuals frequently deviate from Gaussianity. We therefore introduce a distribution-aware, information-theoretic measure of effective connectivity based on channel capacity under general residual distributions. To estimate the resulting capacity from empirical, potentially non-Gaussian residuals, we develop a dual-flow min-max estimator based on normalizing flows, in which a generator searches over admissible input distributions under a power constraint while an observer estimates output entropy. We provide a theoretical characterization of the estimator, showing that the observer objective recovers differential entropy up to a KL approximation term, that the formulation reduces to classical Gaussian capacity as a special case, and that residual entropy can alter achievable information rates beyond variance; game-gap and error analyses further characterize optimization and approximation sources. In brain-like simulations with known directed connectivity, Dual-flow achieves the highest AUROC and AUPRC across ten conditions spanning diverse network topologies, hidden drivers, feedback, and heterogeneous hemodynamics, compared with Gaussian capacity, Granger causality, VAR-LiNGAM, and GIMME. Applied to multimodal brain signals, the method reveals time- and condition-resolved directed interactions consistent with known neurobiological circuitry. Together, these results establish a principled distribution-aware framework for effective-connectivity estimation beyond Gaussian residual modeling.

0.32

Exact Common Information and Exact Channel Synthesis for Correlated Gaussian Sources

cs.IT ·2026-08-26 ·Lei Yu
similarity 0.55
Abstract · 16 pages, 2 figures

In this paper, we resolve two conjectures posed by Yu and Tan in 2020 (in two separate papers published in the IEEE Trans. Inf. Theory). Specifically, we establish that: 1) the exact common information for a pair of $ρ$-correlated Gaussian sources is given by the conjectured expression $\frac{1}{2}\log\frac{1+ρ}{1-ρ}+\fracρ{1+ρ}$; and 2) the admissible region for the shared randomness rate and the communication rate in exact channel synthesis is exactly the conjectured one. These results yield two important consequences. First, for any $ρ>0$, the exact common information of a correlated Gaussian pair strictly exceeds Wyner's common information. Second, for $ρ>0$, the exact channel synthesis of such a pair requires strictly higher rates than the total-variation version. The proof combines an exact optimal-transport representation of the worst-case Gaussian cross-entropy, Fathi's Gaussian transport inequality, and a determinant inequality arising from the covariance structure of the conditional means.

0.36

Fundamental Limits of Joint Target Detection and Parameter Estimation - Characterizing Mixed-State Sensing Limits via Posterior Entropy Volume

cs.IT eess.SP ·2026-09-09 ·Dazhuan Xu, Nan Wang, Han Zhang
similarity 0.54
Abstract

The development of integrated sensing and communication calls for a unified theoretical foundation for sensing. This paper models target-presence patterns and continuous physical parameters as a mixed discrete-continuous state $Ξ$ on a branched reference measure, and treats posterior entropy volume and joint mutual information as two complementary representations of the same limit. Entropy volume carries physical units, can be compared with engineering scales such as resolution cells, and remains meaningful when the number of active targets varies; mutual information is dimensionless, invariant to coordinates and units, and additive through the chain rule. We define entropy number, entropy volume, and mixed entropy volume for discrete, continuous, and mixed states, respectively. The maximum-entropy principle for the uniform distribution on a support of fixed measure explains the measure-theoretic meaning of the exponential entropy scale, while the main limit follows from a mixed asymptotic equipartition property and a posterior probability-volume inequality through conditional typical sets. For asymptotically reliable high-probability sensing regions, the minimum achievable first-order posterior mixed entropy volume equals the prior mixed entropy volume multiplied by $2^{-I(Ξ;Y)}$, where $I(Ξ;Y)=I(V;Y)+I(X_V;Y\mid V)$. Thus, detection and estimation contributions multiply in the volume domain and add in the bit domain, and one sensing bit halves the posterior effective measure. We further prove that posterior-preserving cascades attain the direct-inference limit, while arbitrary intermediate compression incurs the exact information loss $I(Ξ;Y\mid Z)$. Numerical results for a single-target presence-range model illustrate the information composition, posterior entropy-volume contraction, and cascade-interface loss.

0.25