CS Paper Radar

Connections Between Instrumental Variable, Pseudo-Linear Regression, and Prediction Error Methods in Terms of Their Minimized Criteria

Bernard Vau, Henri Bourlès

eess.SY ·submitted 2026-11-17 07:59 UTC · v1
unusual

Abstract

This paper explores the connections between the criteria directly or indirectly minimized by Prediction Error (PE), Instrumental Variable (IV), and Pseudo-Linear Regression (PLR) algorithms for the identification of output-error models. By definition, PE methods aim to minimize the variance of the prediction error. In contrast, the correlation functions appearing in the stationarity conditions of IV and PLR algorithms are not the gradients of any underlying loss function involving a signal variance. Nevertheless, if the true system lies in the model set, we show that one IV algorithm minimizes asymptotically the variance of an unmeasurable signal called the equivalent prediction error. This signal was introduced previously to analyze the bias distribution of PLR schemes. It turns out that, regardless of whether the true system lies in the model set, some iterative IV and PLR algorithms aim to minimize the prediction error variance and can therefore be interpreted as indirect PE algorithms.

Closest papers

Nearest neighbours by meaning across the whole index.

Optimal estimation for Functional Linear Regression with Noisy Discretized Data

cs.LG stat.ML ·2026-09-08 ·Sixtine Sphabmixay
similarity 0.52
Abstract · 63 pages, 25 figures

In this paper, we consider the scalar-on-function linear regression model under a realistic sampling scheme in which the functional covariates are observed on a regular grid and contaminated by additive noise. We propose a two-step estimation procedure: first, the underlying curves are reconstructed from the discrete noisy observations using a Fourier-based projection method; second, the slope function is estimated by a penalized least-squares criterion over finite-dimensional trigonometric spaces, with data-driven selection of the model dimension. We establish oracle-type inequalities for the prediction error, both with respect to the reconstructed curves and to the true latent curves. Under regularity assumptions on the slope function and polynomial decay of the eigenvalues of the covariate, we derive convergence rates for the prediction error and show that our estimator attains the minimax rate when the number of grid points is sufficiently large. Finally, the proposed method is illustrated on simulated data and on a real meteorological dataset.

0.24

From PAC bounds on the generalization error to PAC bounds on the parameter estimation error

cs.LG ·2026-11-06 ·Mihaly Petreczky, John Leth, Emma Lukacs, Balint Daroczy
similarity 0.52
Abstract · 10 pages, 0 figures, under submission to SysID2027

Probably Approximately Correct (PAC) bounds on the generalization error (the difference between the empirical prediction error computed on the training data and its expectation) are an established tool in statistical machine learning, and there have been several extensions of these bounds to dynamical systems. In parallel, in recent years, there has been growing interest in finite-sample bounds for parameter estimation error in system identification. The latter bounds tend to be algorithm-specific, relying on reformulating the identification algorithm as a least-squares regression problem. This is in contrast to classical PAC bounds, which are algorithm-agnostic and can be applied to any identification algorithm based on minimizing a cost function involving the empirical loss. In this paper, we show how oracle bounds derived from PAC bounds can be used to derive finite-sample bounds for dynamical systems. As an illustration of this approach, we present finite-sample parameter-estimation bounds for linear and bilinear systems with bounded inputs and noise. These bounds apply to identification algorithms based on minimizing a regularized empirical loss function. The key ingredient is to establish an oracle inequality, i.e., an inequality between the true prediction error of the learned model and the minimum achievable true prediction error, and then use either the convexity of the true loss or system-theoretic properties of the loss function to derive bounds on the parameter estimation error. We also analyze the rates of the obtained bounds and compare them with those of classical asymptotic results.

0.11

Linear Regression with Self-Selection Bias in Polynomial Time

cs.DS math.STcs.LGstat.ML ·2026-11-12 ·Felix Zhou
similarity 0.49
Abstract

We revisit the problem of estimating $k$ linear regressors with self-selection bias in $n$ dimensions up to error $\varepsilon$ with the maximum self-selection criterion, as introduced by Cherapanamjeri, Daskalakis, Ilyas, and Zampetakis [CDIZ23, STOC]. Our main result is a $\mathrm{poly}(n,k,1/\varepsilon)$ time algorithm for the unknown-index variant of this problem, thereby improving upon the running times of the best known algorithms due to Gaitonde and Mossel [GM24] and Kalavasis, Mehrotra, and Zhou [KMZ26, COLT] with exponential dependence on $k$. Our algorithm succeeds in the presence of correlated noise between regressors, which captures the related setting of max-affine regression. Consequently, we also give the first polynomial-time algorithm for this problem, improving on the prior works of Ghosh, Pananjady, Guntuboyina, and Ramchandran [GPGR22, IEEE Transactions on Information Theory] and Gaitonde and Mossel [GM24] with exponential dependence on $k$. Prior works exhaustively search over a $k$-dimensional subspace spanned by the regressors, using conditional moments of the response to test candidates. In contrast, our algorithm is driven by an improved understanding of the moments of the covariates. We show that, restricting to observations with large response values, the degree-$(\log n)$ Hermite moment of the covariates approximately decomposes as a sum of tensor powers of the regressors. The first step of our algorithm can be viewed as a tensor decomposition algorithm for these implicitly represented high-order tensors, which outputs unit vectors parallel to the regressors. A secondary step based on a simple variant of ordinary least squares extracts estimates for the regressors given these unit vectors. We believe our analytic insight and algorithmic components may have applications to designing efficient algorithms for other latent-variable problems.

0.26

Self-Calibration DOA Estimation for Moving Dilated Arrays with Gain and Phase Errors

eess.SP ·2026-10-09 ·Kaiyuan Fang, Chengzhi Ye, Ruoyu Zhang, Shichao Wei, Wen Wu
similarity 0.48
Abstract

Moving dilated arrays (MDAs) can increase the uniform degrees of freedom (uDOFs) while alleviating mutual coupling, but their performance is susceptible to unknown gain and phase errors (GPEs). To deal with this problem, we investigate self-calibration direction-of-arrival (DOA) estimation for MDAs with unknown GPEs. First, multi-position observations are phase aligned to construct an equivalent virtual array with an enlarged aperture. Next, the motion-induced relationships among the GPEs are exploited to establish explicit calibration constraints. By leveraging the orthogonality between the steering vectors and the noise subspace, the joint estimation of the DOAs and GPEs is formulated as a constrained optimization problem. To solve this problem, a two-stage alternating optimization (AO) algorithm is developed. In the first stage, the GPEs are estimated in closed form with the DOAs fixed. In the second stage, the estimated GPEs are used to calibrate the covariance matrix, after which the DOAs are refined using the multiple signal classification (MUSIC) algorithm. To obtain reliable initial estimates, a rotational invariance-like (RIL) initialization method is further developed by exploiting two translated virtual subarrays with identical GPEs. Furthermore, the Cramer-Rao bound (CRB) is derived to characterize the theoretical performance limit of the considered system. Simulation results demonstrate that the proposed method converges reliably and achieves accurate DOA estimation under different array apertures, numbers of snapshots, GPE levels, and numbers of sources.

0.22

Data-Driven LQR with Uncertainty Refinement and Domain Randomization---Part~I: Certified Uncertainty Refinement

eess.SY ·2026-10-24 ·Hamidreza Modares
theory similarity 0.47
Abstract

Learning controllers from trajectories corrupted by process and measurement noise is difficult when the system matrices and the noise covariances are unknown: noisy regressors bias least squares, and model uncertainty cannot be quantified without the noise covariance, which can only be estimated from residuals of the uncertain model. This paper, Part~I of a two-part study, resolves both obstacles with a finite-sample framework that certifies, and iteratively refines, uncertainty sets for the model, the noise in the collected data, and the noise covariance. A closed-loop parametrization based on delayed instrumental variables (IVs) removes the bias and assigns to every candidate model exactly one noise sequence that reproduces the data. Learning this noise breaks the circular dependence: the noise sequences give the covariance estimate directly, models whose noise violates the prior or the statistical certificates are discarded, and the surviving models yield a tighter covariance certificate. Several delayed instruments further tighten the model set and give a certified test of data consistency. We prove finite-sample containment of the true model, noise, and covariance, recursive nesting, and convergence at the IV rate. Part~II uses these certificates for domain-randomized linear quadratic regulator synthesis.

0.52

Multivariate linear regression without prior assumptions

eess.SY stat.APstat.CO ·2026-09-09 ·Mayank S. K. Gupta, Deepanjhan Das, Arun K. Tangirala, Shankar Narasimhan
benchmark similarity 0.46
Abstract · The manuscript is under further processing. Important modifications might take place in further versions. There are 32 pages containing 10 f…

Recovering the linear relationships that govern a system from noisy measurements is a basic task across the physical and engineering sciences. Because every measured variable may carry an unknown amount of noise, classical regression must commit in advance to a set of structural assumptions: ordinary least squares requires a declared input-output partition with input variables being noise-free, total least squares assumes equal noise variance across all variables, and generalized total least squares additionally requires the noisy-variable partition and variances to be known beforehand. Kalman~\cite{Kalman:1982} showed that any procedure returning a unique linear model from inexact data must rest on such unverifiable a priori assumptions -- ``prejudices'' -- that cannot be checked against the data itself, and that removing them leaves the identification problem fundamentally indeterminate. Whether these prejudices can instead be resolved directly from the data has remain unresolved. Here we show that an iterative generalized-eigenvalue algorithm, QZ-IPCA, recovers the noisy-variable partition, noise variances, number of linear relations, and regression coefficients of a multivariate linear system simultaneously, using only the raw data. Across all possible exhaustive noise configurations of a five-variable benchmark network, QZ-IPCA correctly identifies model structure and recovers coefficients with error below 6.4\%. It outperforms ordinary least squares even when given the best partition, and succeeds in rank identification precisely where standard total least squares falls once noise variances differ across variables. These results show that the assumptions conventionally required for multivariate regression are not necessary, recasting model identification as a problem solvable from data geometry alone.

0.26

Fluid Antenna Channel Reconstruction with Non-Negative Least Square Detector

eess.SP cs.IT ·2026-08-24 ·Yang Chen, Jian Dang, Zaichen Zhang
similarity 0.46
Abstract

Reconstructing the full-aperture channel of a fluid antenna system (FAS) from a few activated ports requires inferring unobserved responses from limited spatial samples. This paper proposes a covariance-domain FAS channel-reconstruction method based on nonnegative least squares (NNLS). A nonuniform port arrangement constructs a difference coarray with an enlarged aperture. Vectorizing the sample covariance maps the physical array to a virtual steering dictionary, and the resulting complex covariance-fitting model is recast as a strictly equivalent real-valued NNLS problem. A projected-gradient iteration initialized at zero estimates the nonnegative angular-occurrence coefficients, and the L_s strongest well-separated peaks yield the angles of arrival (AoAs). Simulations show that the proposed FAS-NNLS estimator exploits the enlarged virtual aperture and achieves lower AoA mean squared errors than uniform linear array (ULA) multiple signal classification (ULA-MUSIC) and ULA estimation of signal parameters via rotational invariance techniques (ULA-ESPRIT). Its non-line-of-sight (NLoS) path accuracy matches FAS covariance matching pursuit, while its line-of-sight (LoS) estimation is significantly improved.

0.17

Analysis increments for model bias diagnosis and correction: Theoretical foundations

eess.SY physics.ao-phstat.ME ·2026-11-08 ·Eviatar Bach
similarity 0.46
Abstract · 15 pages, 5 figures

A common method for diagnosing and correcting model bias in weather and climate prediction is based on analysis increments: the difference between the analysis and the forecast at each data assimilation cycle. Their time average is used to estimate and correct the bias, and machine learning is increasingly used to map the forecast state to the increments, giving a state-dependent correction. Despite its widespread use, the method lacks theoretical foundations. We first analyse a Kalman filter whose forecast model has a constant bias. We show what the averaged increment measures, and prove that iterating the correction removes all of the bias seen through the observations; for linear dynamics, no method can identify the rest. We then extend the analysis to state-dependent biases. For linear dynamics, each increment records the model error over the filter's memory, as seen through the observations. We use this to determine what the increments measure when averaged over a long period, averaged over periods with similar slowly varying conditions, and used without averaging, as in machine-learned corrections. Numerical experiments confirm the theory for a linear model and suggest that its main conclusions carry over to the nonlinear Lorenz '96 model, including for a correction learned by a neural network.

0.34