CS Paper Radar

A Geometric Analysis of Initialization Bias in Spherical $K$-means in the Weak Signal Regime

Amnon Balanov, Tamir Bendory

eess.SP cs.IT ·submitted 2026-09-02 07:18 UTC · v1

Abstract

We study initialization bias in spherical $K$-means for weakly informative directional mixtures. We model the observations by a $K$-component von Mises-Fisher mixture with a small concentration parameter $κ$, corresponding to a high-dispersion regime in which the data provide limited information about the underlying directions. Our analysis begins with the limiting case $κ=0$ (corresponding to a uniform distribution over the sphere), where one population spherical $K$-means update is governed entirely by the Voronoi tessellation induced by the initialized templates. For uniformly random initializations in fixed dimension $d$, the updated templates become asymptotically aligned with their initial values as $K\to\infty$: the average squared geodesic error scales as $O(K^{-2/(d-1)})$, while the worst-case error is $O((\log K/K)^{2/(d-1)})$. We then show that, in the weak-signal regime of small positive $κ$, the population update remains an $O(κ)$ perturbation of this limiting map. Thus, in the weak-signal regime, spherical $K$-means can preserve initialization-induced structure despite the presence of a genuine but highly dispersed directional signal.

Closest papers

Nearest neighbours by meaning across the whole index.

When is Uniform Sampling Optimal? Coresets for Distributional Clustering

cs.DS cs.CG ·2026-11-16 ·Junqiu Hu, Lingxiao Huang, Baoduo Xu
theory similarity 0.63
Abstract

Recent work [1] settled the optimal worst-case strong $\varepsilon$-coreset size for Euclidean $k$-means in unrestricted dimension at $\tildeΘ(\min\{k^{3/2}\varepsilon^{-2}, k\varepsilon^{-4}\})$, showing that the natural target $\tilde{O}(k\varepsilon^{-2})$ is unattainable in general. For mixtures of $k$ isotropic Gaussian components with balanced weights and bounded scale imbalance in dimension $d\ge\log k$, we show that uniform sampling yields strong $\varepsilon$-coresets of size $\tilde{O}(k\varepsilon^{-2})$, without cluster separation. This saves a factor of $\min\{\sqrt{k},\varepsilon^{-2}\}$ over the worst-case bound, up to logarithms. For population approximation by independent, equally weighted samples, the near-linear rate is optimal up to logarithmic factors over this class. More generally, for datasets generated from an isotropic Gaussian mixture $μ$ satisfying a mild cost-comparison assumption, uniform sampling yields strong $\varepsilon$-coresets of size $\tilde{O}((kζ_μ+π_{\min}^{-1})\varepsilon^{-2})$, where $π_{\min}$ is the minimum component weight and $ζ_μ$ measures scale imbalance. Our proof uses a base-shift cost decomposition: one query center per component describes the base costs, and graph spanners control the sparse corrections from points switching centers. For general mixtures under cost and tail assumptions, we obtain $\tilde{O}((kζ_μ+π_{\min}^{-1})\varepsilon^{-2}\min\{\sqrt{k},\varepsilon^{-2}\})$ samples. We also prove a matching $\tildeΩ(\min\{k^{3/2}\varepsilon^{-2},k\varepsilon^{-4}\})$ lower bound for population approximation by uniform sampling from anisotropic Gaussian mixtures, even with equal weights and a shared covariance matrix. [1] Junqiu Hu, Lingxiao Huang, Jian Li, and Baoduo Xu. Tight coreset bounds for $(k, z)$-clustering in Euclidean and doubling metrics, 2026. arXiv:2610.34373.

0.17

Polylogarithmic Sparsity of Randomly Reweighted NPMLEs for Gaussian Mixtures

cs.LG stat.MEmath.ST ·2026-10-01 ·Hansheng Jiang
similarity 0.58
Abstract

The nonparametric maximum likelihood estimator (NPMLE) of a Gaussian location mixture maximizes the likelihood over the infinite-dimensional space of mixing distributions. The maximizing mixing distribution can be nonunique, and the classical bound on its number of atoms grows linearly with the sample size $n$. We show that a vanishingly small random perturbation of the likelihood yields exact polylogarithmic sparsity. The resulting randomly reweighted NPMLE maximizes a weighted likelihood whose independent weights, taken to be Gamma in our analysis, concentrate around one as $n$ grows. With high probability, it is unique, has $O\{(\log n/\log\log n)^d+\log n\}$ atoms in dimension $d$, nearly maximizes the ordinary likelihood, and estimates the mixture density at a Hellinger rate that is parametric up to logarithmic factors. This sparsity holds for the estimator itself, not for an approximation of it, and requires no support penalty. The proof rests on an effective-dimension principle for positive kernel mixtures: low-dimensional variation of the fitted values controls the support of every extreme point of the set of maximizers. Numerical illustrations verify that the reweighted NPMLE has Hellinger risk and support size comparable to those of the ordinary NPMLE.

0.47

Dimension-free regimes for unadjusted Langevin in ridge-separable models: A subspace robust Wasserstein analysis

cs.LG stat.MLmath.PRmath.ST+1 ·2026-10-22 ·Yuansi Chen, Francesco Pedrotti, Peter A. Whalley
theory similarity 0.58
Abstract · 30 pages, 4 figures

The Unadjusted Langevin Algorithm (ULA) is widely used for high-dimensional Bayesian inference. Classical analyses in Wasserstein-2 distance predict an asymptotic bias and a mixing time that grow with dimension, but this metric is stronger than necessary for many statistical tasks. We instead analyze the bias in subspace robust Wasserstein distance, which controls the mean and arbitrary low-dimensional marginals and is therefore well suited to observables typically estimated in practice. For ridge-separable potentials formed with i.i.d. data, we prove high-probability bounds showing a bell-shaped mixing profile as the number of data points $N$ varies relative to the dimension $d$. In the overparameterized regime, the effective dimension is the rank $N$ of the data-dependent perturbation, leading to complexity depending on $N$ rather than $d$. In the large-sample regime, Gaussian-perturbation estimates yield dimension-free complexity once $N$ is sufficiently large. In contrast, near the proportional regime $\asymp d$, we prove that dimension-free guarantees are impossible in general.

0.14

$L_1$ versus $L_2$ Clustering: When Does the Choice of Norm Matter?

cs.DS ·2026-11-04 ·Nick Gravin, Jianhao Jia
similarity 0.56
Abstract

We study how well approximation guarantees for clustering under one norm translate into guarantees under another. Specifically, we compare $k$-clustering in $\mathbb{R}^d$ under the $L_1$ and $L_2$ objectives, defined as the sums of Manhattan and Euclidean distances, respectively. In the worst case, the elementary relationship between these norms yields a $\sqrt{d}$ approximation bound, which is already tight for $k=2$. We therefore investigate whether stronger guarantees are possible in a natural average-case model, where the dataset is arbitrary but the coordinate system is chosen uniformly at random among all orthonormal bases. Our main result shows that, when $d=Ω(\log k)$, with high probability over the random rotation, any $β$-approximate $L_1$ clustering, with optimally chosen centers for its clusters, is also an $O(β)$-approximation under the $L_2$ objective. We also establish an expected approximation ratio of $O(β\min\{\sqrt{d},k^{2/d}\})$, yielding a dimension-independent bound of $O(β\sqrt{\log k/\log\log k})$. We obtain analogous guarantees in the reverse direction, from $L_2$ to $L_1$. We complement these results with lower bounds of $Ω(\min\{\sqrt{d},k^{1/d}\})$ in both directions, showing that our dimension-independent guarantees have asymptotically optimal dependence on $k$ for constant $β$. Our proofs overcome the dependence of the algorithm's output on the random rotation using geometric projection arguments and small Euclidean coresets. In particular, our results provide guarantees for existing clustering algorithms without requiring their redesign.

0.13

Optimal random quantisers for spherically symmetric distributions

cs.LG math.STstat.ML ·2026-10-08 ·Luc Pronzato, Anatoly Zhigljavsky
theory similarity 0.55
Abstract

Zador's celebrated theorem is a cornerstone of optimal quantisation: it establishes both the weak limit of the empirical distribution of an optimal $n$-point quantiser in $R^d$ and the decay rate of the associated $L_s$-mean quantisation error. In large dimension, however, observing this asymptotic behaviour requires an astronomically large sample size. We prove that, for spherically symmetric target distributions, optimisation over all spherically symmetric distributions is a convex problem and derive an equivalence theorem that both characterises global optimality and yields a constructive algorithm. We show that, for moderate $n$, random quantisers uniformly distributed on a sphere of suitably chosen radius $R$ perform exceptionally well and, over a broad range of values of $n$, are numerically certified to be optimal among all random quantisers. Their expected distortion has an explicit integral representation that can be evaluated to arbitrary precision, and we prove concentration across random quantisers: the distortion variance tends to zero as $n\to\infty$ for fixed $d$. For $s=2$, both the optimal radius and the associated minimum expected distortion admit exact expressions. For general $s$, the optimal radius can be determined efficiently, and extreme-value theory provides useful approximations when $n$ grows with $d$. Depending on this growth rate, $R$ either converges to zero or approaches a positive limit that is independent of $s$.

0.57

Separation Is a Riesz Energy: How Many Environments Does ICA Need?

cs.LG cs.ITstat.ML ·2026-11-04 ·Yuda Bi, Yubo Guo, Jiaxiao Cai, Xiaonai Li, Zhuolin Li +1 more
theory similarity 0.55
Abstract

How many environments and samples does it take to separate $d$ Gaussian sources whose variances change across environments? Each source has a signature, its vector of variances. The Riesz $2$-energy $E_j$ of source $j$ sums the inverse Fisher informations for rotating it towards other sources. With known whitening, we prove a worst-case squared-angle risk lower bound of order $E_j/(n+E_j)$ for orthogonal estimators at every $n$ and $d$. A quadratic pencil given the signatures attains $E_j/n$ up to a constant above a separation-dependent sample threshold, even with estimated whitening. For well-spread signatures at fixed $K$, the energy grows as $d^2$, $d\log d$ and $d$ for $K=2$, $3$ and $K\ge4$, respectively. A design with $K\asymp\log d$ and pairwise Fisher information bounded below by a constant makes $d\log d$ samples sufficient. The eigenvectors of a single fixed linear combination have an average first-order error coefficient of at least order $d^2$. Simulations up to $d=1000$ support the first-order predictions, and on task fMRI the energy helps rank split-half instability.

0.28

Diffusion Models for High-Dimensional Clustered Data: Intrinsic-Dimension Adaptivity via Bayesian Classification

cs.LG stat.MLmath.ST ·2026-08-19 ·Yuga Iguchi, Paul Fearnhead
theory similarity 0.53
Abstract

The empirical success of diffusion models in generative modelling has motivated theoretical work, including quantitative error bounds and qualitative analyses that characterise the different phases of denoising. We bring these two areas together by studying the adaptivity of diffusion models to the structured geometry of multimodal high-dimensional data that consists of multiple clusters in $\mathbb{R}^D$, each with its own low-dimensional structure, and inter-cluster separation depending on $D$. We employ $K$-mixture Gaussian distributions as a canonical framework to capture this geometry and establish two theoretical results. First, we interpret denoising as a dynamical Bayesian classifier: the mixture score is a posterior-weighted average of cluster-wise scores, and we show that, with high probability, the posterior class probabilities concentrate on a single cluster once the signal-to-noise ratio reaches the scale $Θ(\log (KD)/D)$. Second, by separately analysing the denoising process in its mixing and cluster-commitment phases, we prove that the KL error bound depends linearly on the maximum intrinsic dimension of a cluster, up to a logarithmic factor, even when $K$ grows polynomially with $D$. This improves on ambient-dimensional bounds and extends existing low-dimensional adaptivity analyses to multimodal distributions with heterogeneous, approximately low-rank covariances.

0.26

Fisher Information and Support Complexity of Gaussian Mixtures

cs.IT ·2026-10-15 ·Luca Barletta, Alex Dytso
theory similarity 0.53
Abstract · 24 pages, single-column

We study lower bounds on the Fisher information of Gaussian location mixtures and their applications to estimation problems. First, for a discrete mixing distribution, we derive a universal positive-semidefinite lower bound on the Fisher information matrix in terms of the Rényi entropy of order two. For a mixing distribution supported on $K$ points, this bound scales as $\log(K)/K^2$, improving by a logarithmic factor on the support-size dependence obtained from Stam's inequality. We then develop dimension-aware trace bounds using classical Sobolev inequalities. These bounds are expressed in terms of Sibson's $α$-mutual information and yield the scaling $K^{-2/n}$ in dimension $n$. A complementary bound involving the $α$-mutual information of order $1/2$ is obtained using the sharp Young convolution inequality. The derived inequalities are further specialized to rotationally invariant mixing distributions supported on discrete spherical shells. In this setting, an intrinsic-dimensional bound on Sibson mutual information leads to lower bounds on the Fisher information in terms of the number of occupied shells. As applications, we study least favorable priors for bounded Gaussian mean estimation. In the scalar case, we show that the least favorable prior on $[-R,R]$ has at least $Ω(R\sqrt{\log R})$ support points and that each of its probability masses is of order at most $1/R$. In the vector case, we show that the number of occupied shells grows at least linearly with $R$. Finally, Brown's identity and the I-MMSE relationship are used to translate the Fisher-information inequalities into upper bounds on the MMSE and mutual information.

0.28