CS Paper Radar

Foundations of Independent Component Analysis

Patrick Forré

cs.LG math.STmath.PRstat.ML ·submitted 2026-08-13 13:31 UTC · v1

Abstract

We present the mathematical foundations of linear independent component analysis (ICA) models based on standard literature in a self-contained note. It is aimed at readers with a background in measure-theoretic probability theory. We first develop the theory of the characteristic functions of probability measures on $\mathbb{R}^d$, including their analyticity and the way in which they determine and characterise the distributions. We then focus on several identifiability results of ICA models with successively strengthened assumptions on the sources: from merely non-constant, to non-Gaussian, to Gaussian-free independent sources. Under the strictest assumptions, we show that the independent sources are identifiable up to translation, permutation, scales and signs, and this even in the presence of additive Gaussian noise. Furthermore, we present the online equivariant gradient descent ICA algorithm for recovering the independent sources from data, in the standard complete noiseless non-Gaussian ICA setting.

Closest papers

Nearest neighbours by meaning across the whole index.

Identifying Dependent Components from Multi-domain Linear Mixtures

cs.LG cs.AI ·2026-10-20 ·Danru Xu, Lauri Parkkonen, Sara Magliacane, Aapo Hyvärinen
top venuehighly cited similarity 0.65
Abstract · 29 pages, 3 figures

We study a linear mixing model with dependent latent components, assuming multiple data domains. Most existing models assume that the components are independent or at least uncorrelated, in line with independent component analysis (ICA). Some recent work allows for dependent components, but then makes specific assumptions such as parametric forms of dependencies, multi-view settings, or interventions, or does not recover the individual components. In contrast, we consider a multi-domain setting in which domains differ through domain-specific scalings of the components, while the distribution of the underlying latent components is the same across domains. This approach can model data collected, for example, from different sensors measuring the same process, different laboratories conducting the same experiment, different experimental conditions, or different subjects that might differ in biological or physiological factors. We show that, under sufficient domain variability, latent variables and mixing functions can be identified from second-order statistics alone. We propose the Multi-Domain Covariance Matching (MuDo-CoM) algorithm that generalizes previous methods of joint diagonalization. MuDo-CoM is validated on simulated data and a real-world fMRI dataset.

0.63

Kinks vs. Smoothness: Identifiability of Real Analytic nICA for Laplace-like Sources

cs.LG ·2026-09-18 ·Isaac Manring, Kejun Huang
theory similarity 0.63
Abstract

Many machine learning systems try to explain complex data - like images or financial time series - in terms of hidden, independent factors that generated them. Recovering the true underlying factors, rather than some scrambled version of them, is the central challenge of nonlinear Independent Component Analysis (nICA). We prove identifiability (exact recovery) up to trivial ambiguities for real analytic generating functions when source probability density functions have a finite number of discontinuities in the first derivative. The Laplace distribution is the most prominent example satisfying this assumption. Our proof relies on the contrast between kinks in the source distribution and the smoothness of real analytic functions. Real analytic functions comprise a broad class of generating mechanisms, and can be approximated with Normalizing Flows or Variational Autoencoders with standard activation functions (e.g., tanh, softplus, GELU), so our result applies with minimal changes to existing training pipelines. We perform experiments on real and synthetic data with both Normalizing Flows and Variational Auto-Encoders demonstrating their identifiability properties. In experiments on CelebA data we recover several interpretable latent factors controlling unique attributes across the dataset.

0.27

Towards Identifiable Representations under Misspecified Structure

cs.LG ·2026-09-27 ·Yuke Li, Yujia Zheng, Ziyi Chen, Kun Zhang, Heng Huang
theory similarity 0.59
Abstract

The presence of noise that depends on the latent variables poses a fundamental challenge to identifiability. Existing results rely on conditional independence among the observations given the latent variables. We study a more general \emph{misspecified structure}, where this conditional factorization does not hold, and establish both precise and approximate identifiability guarantees. We characterize structural misspecification as a perturbed factor analysis problem. For precise identifiability, we establish subspace identifiability under spectral separation and controlled perturbation, followed by component-wise identifiability under structural sparsity. When the precise condition is not guaranteed, we derive an approximate subspace-identifiability theorem. Based on these results, we develop an unsupervised variational estimator for recovering latent variables. Experiments demonstrate the effectiveness of the proposed framework.

0.27

Online Covariance Estimation for Averaged Stochastic Gradient Descent on Riemannian Manifolds

cs.IT stat.MLcs.LGmath.OC+1 ·2026-10-24 ·Jingrong Guan, Liwei Jiang, Sen Na
similarity 0.57
Abstract · 60 pqges, 7 figures, 2 tables

We study online inference for averaged Riemannian stochastic gradient descent (RSGD) when the unknown parameter lies on a manifold. Although the averaged RSGD iterate is asymptotically normal, practical inference requires estimating the unknown covariance operator in its limiting distribution. Existing online batch-means methods cannot be applied directly because a general Riemannian manifold lacks an addition operation, and the tangent space containing the Riemannian gradient varies across iterations. Through vector transport, we construct overlapping and non-overlapping covariance estimators that adaptively combine information from successive batches while accounting for the changing tangent spaces. The proposed estimators are online, require neither Riemannian Hessian computations nor a prespecified terminal sample size, and attain the same convergence rates as their Euclidean counterparts (Zhu et al., 2023). Together with asymptotic normality, these estimators enable online statistical inference for RSGD on manifolds. We illustrate the proposed framework through a detailed study of streaming principal component analysis on Grassmann manifolds.

0.14

From Common Information to Canonical Correlation Analysis and Beyond: Bridging Information Theory, Signal Processing, and Machine Learning

cs.IT cs.LGeess.SPstat.ML ·2026-10-20 ·Paris A. Karakasis, Walid Saad
published similarity 0.56
Abstract · Tutorial, 100 pages, 11 figures. Accepted for publication in Entropy, Special Issue "Foundations and Frontiers of Information Theory" in hon…

Many problems in signal processing, machine learning, and neuroimaging provide several jointly observed measurements of one process, each mixing a component common to all of them with a component private to itself, and require the common component to be recovered without supervision. This problem has been studied in several communities, largely independently. Statisticians proposed canonical correlation analysis (CCA), introduced by Hotelling in 1936. Information theorists proposed two notions of common information, due to Gács and Körner and to Wyner. Engineers developed subspace methods for sensor arrays and multiuser communications to design multiview receivers. This tutorial presents the main results and tools each community developed for this problem in a single exposition, and explains the connections that emerge where they meet, distinguishing exact equivalences from special cases and analogies. We develop the information-theoretic foundation from first principles. We then introduce CCA and its extension to several views, followed by the engineering thread from blind multiuser detection to CCA as a blind receiver. Next, we turn to nonlinear CCA, covering kernel, deep, and probabilistic formulations, the conditions under which the common factor is identifiable, and connections to self-supervised learning. We conclude with a discussion of further extensions and open directions. The exposition is self-contained, so that readers from each of these communities can follow the parts developed by the others.

0.39

Distributed Blind Source Separation from Compressed Mixtures

eess.SP ·2026-11-17 ·Amir Weiss
benchmarktheory similarity 0.55
Abstract

Blind source separation (BSS) is increasingly needed in sensing architectures where mixtures are acquired by multiple remote devices but the inference itself is carried out centrally. In such systems, the sensors often cannot transmit their raw observations since the communication links are rate-limited, energy-limited, or both. The fusion center therefore must recover the latent sources from compressed versions of the mixtures. This setting raises questions that classical BSS models do not address directly: whether source identifiability survives compression, how separation accuracy changes under a specified compression model, and how the separation procedure should be modified to account for the statistical model induced by the compression stage. In this work, we develop a framework for distributed BSS from compressed mixtures that addresses these issues at three levels. First, we formalize identifiability under compression and establish a data-processing viewpoint showing that compression cannot introduce distinguishability that is absent before compression. Second, for mixtures of Gaussian sources with general covariance structures, we derive an idealized Gaussian rate-distortion-based Cramér-Rao lower bound benchmark for the interference-to-source ratio. Third, we develop practical compression-aware separation schemes based on subtractively dithered quantization together with debiasing at the fusion center for both fourth-order-cumulant-based and second-order-statistics-based methods. Simulation experiments demonstrate the resulting tradeoffs between communication rate and separation performance and the gains obtained by compression-aware separation in representative distributed sensing scenarios.

0.24

Is $\sqrt{d}$ Separation Necessary for Gradient EM to Learn Gaussian Mixtures in High Dimensions?

cs.LG stat.ML ·2026-10-06 ·Yiran Zhang, Mo Zhou, Weihang Xu, Maryam Fazel, Simon S. Du
theory similarity 0.55
Abstract · 51 pages

Learning Gaussian mixture models (GMMs) using the Expectation-Maximization (EM) algorithm and its gradient-based variants is a fundamental problem in machine learning. It is known that randomly initialized (gradient) EM fails to learn multi-component GMMs in the exact-parameterized setting, where the number of components matches that of the ground-truth GMM. Recently, global convergence of gradient EM has been established in the over-parameterized setting, where more components are used, provided that the ground-truth components are well separated. In particular, the minimum separation between ground-truth components is required to scale as $Ω(\sqrt{d})$, where $d$ is the dimension. In this paper, we show that this dimensional dependence is unavoidable in high-dimensional settings. Specifically, we consider a hybrid EM algorithm that uses standard EM updates for the mixing weights and gradient EM updates for the component means. For any $ε> 0$, we prove that when the dimension is sufficiently large, in the worst case a separation of order $Ω(d^{0.5-ε})$ is insufficient to guarantee global convergence of population gradient EM in sub-exponential time under random initialization, even in the over-parameterized regime. Our result establishes an almost optimal worst-case lower bound on the ground-truth separation required for learning Gaussian mixtures via gradient EM in high dimensions.

0.31

Interpretable intrinsic dimension estimation through componentwise calibration of distance and angle

cs.LG stat.ML ·2026-09-29 ·Chih-Hsuan Huang, Chih-Wei Chen, Szu-Chi Chung
similarity 0.55
Abstract · 35 pages, 5 figure, 2 tables

DANCo (Dimensionality from Angle and Norm Concentration) jointly calibrates nearest-neighbor distance and angular statistics and consistently reaches state-of-the-art accuracy on clean intrinsic-dimension (ID) benchmarks. Practical data, however, introduce neighborhood-relative noise and sample-amplitude heterogeneity that can distort these geometric signals. We reformulate DANCo componentwise, retaining separate distance and angular discrepancy curves so that the source of an estimate can be identified and interpreted. For the distance component, we derive a closed-form Kullback-Leibler divergence for the generic-order ratios of the generalized ratios ID estimator (Gride); when both angular parameters are matched (Full), Gride reduces mean percentage error from $27.7\%$ to $17.6\%$ at noise equal to $40\%$ of typical neighbor spacing on 24 manifolds. For the angular component, two sampling regimes motivate aligning mean direction while retaining concentration matching (Profiled). On a Gaussian scale mixture with generating dimension 70 embedded in 100 dimensions, profiling raises the Minimum Neighbor Distance (MiND) estimate from $22.8$ to $66.7$, while removing the known amplitudes restores MiND-Full to $71.9$; the control thus attributes the Full shortfall to amplitude heterogeneity. On CIFAR-10 and ImageNet, amplitude-reducing normalizations move angular location toward the references and narrow the Full-Profiled gap, an observational counterpart to the controlled mixture. Across four pretrained convolutional neural networks, Gride-Profiled, the two-nearest-neighbor estimator (TWO-NN), and the maximum-likelihood estimator (MLE) exhibit similar rise-and-fall profiles, while Full-Profiled differences identify the layers most sensitive to angular calibration.

0.26