CS Paper Radar

Multivariate linear regression without prior assumptions

Mayank S. K. Gupta, Deepanjhan Das, Arun K. Tangirala, Shankar Narasimhan

eess.SY stat.APstat.CO ·submitted 2026-09-09 17:18 UTC · v1
benchmark

Abstract

Recovering the linear relationships that govern a system from noisy measurements is a basic task across the physical and engineering sciences. Because every measured variable may carry an unknown amount of noise, classical regression must commit in advance to a set of structural assumptions: ordinary least squares requires a declared input-output partition with input variables being noise-free, total least squares assumes equal noise variance across all variables, and generalized total least squares additionally requires the noisy-variable partition and variances to be known beforehand. Kalman~\cite{Kalman:1982} showed that any procedure returning a unique linear model from inexact data must rest on such unverifiable a priori assumptions -- ``prejudices'' -- that cannot be checked against the data itself, and that removing them leaves the identification problem fundamentally indeterminate. Whether these prejudices can instead be resolved directly from the data has remain unresolved. Here we show that an iterative generalized-eigenvalue algorithm, QZ-IPCA, recovers the noisy-variable partition, noise variances, number of linear relations, and regression coefficients of a multivariate linear system simultaneously, using only the raw data. Across all possible exhaustive noise configurations of a five-variable benchmark network, QZ-IPCA correctly identifies model structure and recovers coefficients with error below 6.4\%. It outperforms ordinary least squares even when given the best partition, and succeeds in rank identification precisely where standard total least squares falls once noise variances differ across variables. These results show that the assumptions conventionally required for multivariate regression are not necessary, recasting model identification as a problem solvable from data geometry alone.

Comment
The manuscript is under further processing. Important modifications might take place in further versions. There are 32 pages containing 10 figures and 3 tables

Closest papers

Nearest neighbours by meaning across the whole index.

Subspace Based Identification of Errors-in-Variables Linear Descriptor Systems

eess.SY ·2026-08-31 ·Deepanjhan Das, Shankar Narasimhan
published similarity 0.58
Abstract · Accepted for publication in the 8th International Symposium on Advanced Control of Industrial Processes (IEEE AdCONIP 2026), University of A…

The identification of linear descriptor systems (DAEs) from noise-corrupted data makes two critical assumptions: requirement of an \textit{a priori} classification of variables into inputs and outputs, and a pre-specified structural assumption with respect to the index of the system. This paper proposes a data-driven methodology for identifying index-0 and index-1 DAEs within an errors-in-variables framework. We extend a subspace-based iterative PCA (SMI-IPCA) approach to the behavioral setting, treating all measured variables as a unified augmented vector to avoid classification bias. This method enables systematic estimation of the noise variances, the number of algebraic and differential output variables, while simultaneously identifying the algebraic constraints and kernel representation of the dynamic system corresponding to its minimal realization order without prior structural knowledge. Simulation studies on index-0 and index-1 systems demonstrate the effectiveness of the proposed approach and its practical applicability.

0.56

Blind Noise Equalization for Large Data Matrices under General Variance Profiles

cs.LG stat.MLeess.SPmath.ST+1 ·2026-11-05 ·Boris Landa, Francesco Strino, Fabio Parisi, Ruiqi Li, Yuval Kluger
theory similarity 0.56
Abstract

Low-rank approximation is a basic tool for inference from large and noisy data matrices. However, when noise variances differ across entries, a setting known as heteroskedasticity, the noise can distort the observed spectrum, producing spurious components and obscuring weak signals. To address these challenges, we develop BRINE (Blind Resolvent-based Iterative Noise Equalizer), a data-driven procedure that iteratively rescales rows and columns to equalize the average noise variance across both dimensions. Under suitable conditions, this procedure restores the Marchenko-Pastur law for the noise spectrum and can amplify signal components without prior knowledge of the signal or noise structure. A key idea underlying our approach is to equalize the diagonal entries of the data resolvent as a proxy for equalizing the unknown noise variances. At the population level, we identify a potential function that characterizes this equalization and motivates an iterative minimization procedure. We prove convergence of the iterations to the true equalizer for arbitrary positive variance profiles, with a geometric rate under mild conditions. The updates can be estimated directly from noisy data, and we establish probabilistic error bounds for the scaling factors. Simulations and experiments across diverse biological data modalities and preprocessing pipelines illustrate the effectiveness of our approach.

0.30

Large-System Analysis of Sparse Bayesian Learning

eess.SP cs.IT ·2026-09-06 ·Fangqing Xiao, Dirk T. M. Slock
similarity 0.55
Abstract

Sparse Bayesian learning is widely used for sparse linear inverse problems, yet its large-system stationary behavior remains poorly understood because all variance hyperparameters are estimated from the same data. We study classical sparse Bayesian learning, formulated as evidence maximization (type-II maximum likelihood), for underdetermined linear models with sensing matrices having independent and identically distributed Gaussian entries, Gaussian measurement noise, and an unknown deterministic signal sequence. Hyperparameter reoptimization induces a nonvanishing feedback term: a typical coordinate obeys a reoptimization-corrected scalar Gaussian law whose signal coefficient is governed by the normalized adaptive response rather than the frozen resolvent trace. A one-coordinate leave-one-out construction gives an exact conditional Gaussian law, which is transferred to a selected full stationary branch without assuming asymptotic closeness of the reduced and full stationary vectors. Combining this law with the Karush--Kuhn--Tucker conditions of the evidence objective yields a generally set-valued scalar relation and three branchwise large-system consistency relations. If the model noise variance is jointly estimated by evidence maximization, interior joint stationarity yields an exact finite-dimensional equality between the normalized residual energy and normalized resolvent trace. When the limiting signal law has nonzero mass at zero, this identity further yields a parameter-free asymptotic chi-square null law. Under an additional differentiability condition on the selected scalar branch, the large-system characterization also gives a closed relation for the reconstruction error of the posterior mean. The analysis is stationary-point based and permits multiple stationary branches.

0.32

Practical Algebraic Parameter Estimation for Noisy Data via Gaussian Process Regression

eess.SY cs.MSmath.OC ·2026-09-24 ·Oren Bassik, Alexander Demin, Alexey Ovchinnikov
benchmark similarity 0.53
Abstract

Parameter estimation for ordinary differential equation (ODE) models is a fundamental task that is often complicated by the limitations of conventional optimization-based methods. In theory, differential-algebraic approaches offer an appealing alternative: they reduce the problem to polynomial system solving and do not require user-supplied initial guesses for parameter values. In practice, however, algebraic methods have been limited by their sensitivity to measurement noise, because they require accurate derivatives of observed outputs. In this work, we integrate Gaussian Process Regression (GPR) into the differential-algebraic method and derive a first-order error analysis in terms of noise level and algebraic sensitivity. We evaluate the method across several noise levels on a benchmark of 25 dynamical systems arising in applications including mechanical engineering and systems biology. The proposed method achieves the highest aggregate performance among the methods considered, recovering all sought parameter values and initial conditions to within 10% relative error in 88.5% of runs. These results demonstrate that robust derivative estimation can make differential-algebraic parameter estimation practical for dense, noisy synthetic data while retaining key advantages of the algebraic formulation.

0.35

Latent variable models for simultaneous EOV identification and removal in population-based SHM

eess.SP cs.LG ·2026-08-12 ·M. D. Champneys, M. R. Jones, A. J. Hughes, T. J. Rogers, E. J. Cross +1 more
benchmark similarity 0.53
Abstract

The robust treatment of environmental and operational variability (EOV) is an open challenge in population-based structural health monitoring (PBSHM). The difficulty is compounded in the case that the EOV signals are unmeasured. A common approach in conventional SHM is to apply \emph{projection-based} methods that discard subspaces of healthy feature data, reasoning that the EOV signal dominates the variance of the measured features. However, a common pitfall of projection-based approaches is that when damage acts close to the same variance-dominant direction, damage sensitivity is removed along with the EOV. An alternative identifying assumption for the removal of particular unmeasured EOVs is slowness; the latent EOV process is characterised by its long temporal correlation. In this paper, the latent EOV is cast as a state-space Gaussian process, enabling tractable $\mathcal{O}(T)$ inference via a Kalman filter. A robust hierarchical Bayesian identification framework is developed that enables population-level identification of latent EOVs and EOV-free residual features, using a Laplace approximation. The approach is first validated on a single laboratory-scale benchmark structure from the literature, subject to thermal EOVs, demonstrating robust damage detection and EOV recovery. The method is then applied to a simulated nine-turbine offshore wind farm with staggered deployment and damage, where it delivers a substantial true-positive uplift over projection and cointegration-based baselines at matched false-positive rates.

0.04

Identifying Dependent Components from Multi-domain Linear Mixtures

cs.LG cs.AI ·2026-10-20 ·Danru Xu, Lauri Parkkonen, Sara Magliacane, Aapo Hyvärinen
top venuehighly cited similarity 0.52
Abstract · 29 pages, 3 figures

We study a linear mixing model with dependent latent components, assuming multiple data domains. Most existing models assume that the components are independent or at least uncorrelated, in line with independent component analysis (ICA). Some recent work allows for dependent components, but then makes specific assumptions such as parametric forms of dependencies, multi-view settings, or interventions, or does not recover the individual components. In contrast, we consider a multi-domain setting in which domains differ through domain-specific scalings of the components, while the distribution of the underlying latent components is the same across domains. This approach can model data collected, for example, from different sensors measuring the same process, different laboratories conducting the same experiment, different experimental conditions, or different subjects that might differ in biological or physiological factors. We show that, under sufficient domain variability, latent variables and mixing functions can be identified from second-order statistics alone. We propose the Multi-Domain Covariance Matching (MuDo-CoM) algorithm that generalizes previous methods of joint diagonalization. MuDo-CoM is validated on simulated data and a real-world fMRI dataset.

0.63

Towards Identifiable Representations under Misspecified Structure

cs.LG ·2026-09-27 ·Yuke Li, Yujia Zheng, Ziyi Chen, Kun Zhang, Heng Huang
theory similarity 0.51
Abstract

The presence of noise that depends on the latent variables poses a fundamental challenge to identifiability. Existing results rely on conditional independence among the observations given the latent variables. We study a more general \emph{misspecified structure}, where this conditional factorization does not hold, and establish both precise and approximate identifiability guarantees. We characterize structural misspecification as a perturbed factor analysis problem. For precise identifiability, we establish subspace identifiability under spectral separation and controlled perturbation, followed by component-wise identifiability under structural sparsity. When the precise condition is not guaranteed, we derive an approximate subspace-identifiability theorem. Based on these results, we develop an unsupervised variational estimator for recovering latent variables. Experiments demonstrate the effectiveness of the proposed framework.

0.27

Sample Dependence-Aware Blind Source Separation for Linear Causal Discovery

eess.SP ·2026-10-05 ·Cihan Eralp Kumbasar, Emin Erdem Kumbasar, Tulay Adali
similarity 0.51
Abstract · 5 pages, 2 figures. Submitted to the 2027 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Causal discovery seeks to determine directional relationships among variables in complex systems, yet observational data alone generally do not identify causal direction. Complete identification therefore requires assumptions beyond linearity and acyclicity. A key approach is the Linear Non-Gaussian Acyclic Model (LiNGAM), which formulates linear structural equation model estimation as a blind source separation (BSS) problem. LiNGAM treats disturbances as mutually independent random variables under independent sampling, relying on higher-order statistics (HOS) and non-Gaussianity for identifiability. This restricts identifiability when multiple disturbances are Gaussian and ignores potentially informative temporal dependence. To broaden causal identifiability, we formulate linear acyclic causal discovery with mutually independent disturbance processes and transfer established BSS identification conditions to causal structure recovery. Under this formulation, replacing the mutual information cost with mutual information rate enables identification under a broader class of disturbances, including Gaussian processes with nonproportional covariance functions. Synthetic experiments reveal the limitations of methods relying solely on HOS or sample dependence, while their joint exploitation provides near-perfect causal recovery across the tested disturbance settings. Experiments on real functional magnetic resonance imaging (fMRI) data further demonstrate that jointly exploiting HOS and sample dependence improves the bootstrap stability of estimated ancestral directionality.

0.17