CS Paper Radar

Error Bounds for Statistical Estimators in BTL Model with Parametric Multivariate Utility Functions

Yicheng Li, Huifu Xu

cs.LG stat.ML ·submitted 2026-09-22 12:36 UTC · v1

Abstract

We study preference elicitation under the Bradley-Terry-Luce (BTL) model where the true partworth vector is unknown and has to be estimated as a parameter with elicited preference information. The set of selected pairwise queries is non-uniform, deterministic, and arbitrary over a collection of alternatives, provided that it satisfies a joint identifiability condition. We focus on understanding when the canonical maximum likelihood estimator (MLE) is finite and admits sharp error bounds without explicit compactness constraints on the feasible set or external regularizers. To this end, we derive minimax lower bounds under the standard bounded dynamic range condition, and find that the same Fisher-information geometry in the classic Cramér-Rao lower bounds underpins the finite-sample difficulty of the estimation problem. By combining a non-asymptotic expansion of the likelihood score equation with a fixed-point localization argument, we identify a design-dependent sample size threshold above which the unconstrained canonical MLE exists and is unique with high probability. The same expansion yields a decomposition of the estimation error into a linear stochastic term, an explicit second-order bias, and a higher-order remainder. A refined analysis gives sufficient sample size conditions under which the canonical MLE attains the minimax rates up to logarithmic and constant factors. These results provide a unified non-asymptotic theory for parametric utility elicitation and reveal when the inference is determined by response data alone rather than by external regularization. Preliminary numerical results are consistent with the theoretical findings.

Closest papers

Nearest neighbours by meaning across the whole index.

Statistical Learning Theory for Direct Preference Optimization

cs.LG stat.ML ·2026-10-19 ·Joel Wills, Yuanfan Li, Yiming Ying, Ding-Xuan Zhou
theory similarity 0.64
Abstract · 16 pages, including references and appendices

Direct Preference Optimization (DPO) learns from pairwise preferences through a logistic objective whose score-difference parameterization follows the Bradley--Terry model. We study how failure of this modeling assumption limits DPO, distinguishing structural misspecification from limited model capacity and finite-sample estimation. Our main contribution is a characterization of structural approximation error through the triangle-curl of the preference log-odds, which vanishes exactly when preferences admit a Bradley--Terry representation. We upper bound the achievable excess logistic risk by a score-class approximation term and a triangle-curl term, and establish a lower bound proportional to triangle-curl under additional boundedness assumptions. For ReLU score models, including clipped models, we derive explicit approximation rates showing that increasing capacity can eliminate the score-approximation term while leaving the structural obstruction unchanged. We further establish finite-sample bounds for clipped empirical risk minimizers that separate network approximation, structural misspecification, and estimation error. We place these results within a surrogate-risk framework by identifying contextual preference accuracy (CPA) as a natural measure of agreement between preference judgments and changes in pairwise probability ratios relative to the reference policy. Standard logistic consistency and calibration results connect DPO's objective to this policy property and transfer our excess-risk upper bounds to CPA, without treating CPA as a complete measure of generated-response quality.

0.18

Learning Distributions from Samples and Qualities

cs.LG stat.ML ·2026-10-13 ·Marc Jourdan, Nicolas Flammarion
theoryhot direction similarity 0.63
Abstract · 50 pages, 9 figures, 1 table

We study parameter estimation when each sample from a continuous parametric distribution is accompanied by a binary quality label governed by the same unknown parameter. By imposing hard feasibility constraints, deterministic labels can accelerate estimation from the classical $n^{-1/2}$ rate of smooth quality-aware M-estimators to $n^{-1}$. Combining PAC guarantees with local error-to-parameter conversions, we establish this rate for both identifiable and finitely ambiguous quality classifiers, with likelihood information resolving the latter ambiguity. When the quality model is known only up to an additive constant, we construct threshold-free pairwise preferences from oppositely labeled samples and retain the accelerated rate despite the unknown threshold. We further show that pointwise qualities and pairwise preferences are incomparable: either can be strictly more informative than the other. Our assumptions hold for Gaussian, Laplace, and exponential families under thresholded log-likelihood rewards. Finally, a minimax lower bound establishes the optimal $n^{-1}$ dependence on sample size, up to logarithmic and dimension-dependent factors, while experiments confirm the benefits of constraint-based estimation.

0.35

Minimax-Optimal Robust Identification of Continuous-Time Systems: Handling Narrow-Band Disturbances in the Frequency Domain

eess.SY math.ST ·2026-09-20 ·Jonas Gillberg, Fredrik Gustafsson
similarity 0.61
Abstract · 17 pages, 13 figures

The high-frequency spectral roll-off of continuous-time ARMA (CARMA) models can magnify the effect of narrow-band disturbances when aliasing is weak, making standard maximum-likelihood Whittle estimation sensitive to affected ordinates. We show that a logarithmic transformation $r_k = \log ρ_k$ converts the Whittle scale problem into a Gumbel location problem, connecting robust spectral estimation to the classical minimax theory of Huber and Rieder. A piecewise centering correction---closed-form for small $b$, implicit closed-form for the practitioner range---preserves Fisher consistency for any clipping level without numerical optimisation. Clipping the Gumbel score symmetrically and transforming back yields a two-sided clipped Gumbel score (the standard Rieder--Hampel bounded-influence form) whose normalised influence curve is proved locally asymptotically minimax under shrinking gross-error contamination. The efficiency loss is quantified by a single scalar $K(b)$: at $b = 1.5$, only $16\%$ nominal asymptotic variance overhead. In the stated AR(2) Monte Carlo design, the sample-size trends are compatible with the asymptotic rate, and at $b=1.5$ the bias reductions are about $40\%$, $90\%$, and $96\%$ for $a_1$, $a_2$, and $λ$, respectively.

0.26

Minimax Quantile Bounds via Information Measures

cs.IT math.ST ·2026-08-21 ·Amedeo Roberto Esposito
similarity 0.59
Abstract

We develop a unified information-theoretic framework for lower bounding minimax quantiles. The starting point is a loss-adapted Neyman--Pearson metaconverse that bounds the minimax success probability at every loss threshold and confidence level. The bound separates the small-ball behaviour of the prior under the loss from the statistical distinguishability of the observation model, and is optimised over an auxiliary output distribution. Different relaxations of this Neyman--Pearson bound yield converses based on \(f\)-informativity, Sibson mutual information \(I_α\), Maximal Leakage, and Amemiya norms. Classical Fano and Le Cam lower bounds are recovered as special cases. The framework also clarifies why different information measures are suited to different recovery criteria. Maximal Leakage is exact for a class of symmetric exact-recovery problems. We use this identity to derive finite-sample bounds on the full minimax exact-recovery risk in the balanced Gaussian weighted stochastic block model, as well as two-sided finite-sample minimax-quantile bounds for low-rank matrix estimation under isotropic bounded-energy noise. For approximate Hamming recovery, we exhibit a heterogeneous binary model in which an optimised finite Sibson order yields a strong converse while the Maximal Leakage specialisation is trivial. Finally, for one-coordinate Poisson localisation, a Bennett-type Young function used through its Amemiya norm recovers the exact success-probability scale, whereas classical Fano and fixed-power relaxations are strictly weaker. These results show that sharp converses for minimax quantiles require adapting the information measure to the recovery resolution, whether exact or approximate, and to the tail behaviour of the likelihood ratio.

0.45

A Bayesian-Geometric Reading of Classical Model-Based Optimal Design

cs.IT math.STmath.OCmath.PR+1 ·2026-11-03 ·Mohamed Tarek
similarity 0.59
Abstract

The classical model-based optimal design criteria (D-, T-, L-, C-, and A-optimality) are usually motivated via the Cramér--Rao bound as optimizing scalar functions of the Fisher information matrix (FIM). This paper offers a complementary reading in which each criterion appears asymptotically as $N \to \infty$ as the leading-order term of an expected dissimilarity between prior and posterior, with the FIM entering through the Bernstein--von Mises limit. The Kullback--Leibler (KL) divergence of the posterior relative to the prior recovers D-optimality (classically known); a reverse KL of the prior relative to the posterior recovers T-optimality (here defined as maximization of the trace of the FIM, following one common convention) and its weighted-trace generalization; and a first-order approximation of the squared 2-Wasserstein distance recovers L-, C- and A-optimality. The five criteria emerge as instances of a common Bayesian-geometric template, indexed by three choices: the dissimilarity, the prior parametrization, and the approximation scheme used to evaluate it (e.g.\ Taylor expansions of the matrix square root). For the weighted criteria, the Gaussian prior covariance encodes the criterion's weight matrix, so any choice of weights corresponds to a specific prior in the family. The T-, L-, C-, and A-optimality derivations appear novel to the best of my knowledge. A sixth criterion, E-optimality, is recovered not as an independent expected-dissimilarity limit but as a min-max robustification of C-optimality against uncertainty in the modeler's prior direction. The framework is extended to distinguish between a designer's prior over unknown parameters at design time and a modeler's prior used for posterior inference, recovering the classical expected information gain objective as a special case.

0.17

Lower bounds for Bayesian CVaR in interactive problems with local differential privacy

cs.IT ·2026-10-20 ·Raghav Bongole, Amirreza Zamani, Tobias J. Oechtering, Mikael Skoglund
theory similarity 0.59
Abstract

We develop an information-theoretic method for lower bounding the prior-predictive conditional value-at-risk (CVaR) of loss in interactive statistical decision making (ISDM). A generalized Fano inequality replaces a single error event by a randomized binary statistic constructed from a bounded transform of the loss, reducing the problem to a Bernoulli $f$-divergence inequality. A bounded hinge transform yields a direct lower bound on Bayesian CVaR under a prior fixed in advance. We derive reverse-KL specializations and obtain explicit bounds for Gaussian mean estimation and Gaussian stochastic bandits. We then impose local differential privacy (LDP) at the observation stage of the same ISDM model. LDP contraction inequalities combined with the adaptive KL chain rule give explicit Bayesian CVaR lower bounds for Gaussian mean estimation with heterogeneous privacy parameters and Gaussian $K$-armed bandits.

0.20

Minimax Optimal Estimator and Improved Error Rate for the MLE in Logistic Regression with Gaussian Design

cs.IT math.STstat.ML ·2026-08-18 ·Junren Chen, Arya Mazumdar
theory similarity 0.58
Abstract

We study finite-sample parameter estimation in logistic regression with Gaussian design, where the goal is to estimate $\mathbfθ^*\in \mathbb{R}^d$ with $R=\|\mathbfθ^*\|_2\ge 1$ from i.i.d. samples $\{(\mathbf{x}_i,y_i)\}_{i=1}^n,$ $\mathbf{x}_i \sim N(0,\mathbf{I}_d)$, $y_i\mid \mathbf{x}_i \sim \mathrm{Bernoulli}((1+\exp(-\mathbf{x}_i^\top \mathbfθ^*))^{-1})$. In this paper, we provide the first minimax optimal estimator, and improve on the best known finite-sample error rate for the maximum likelihood estimator (MLE). These two accomplishments are due to a minimax optimal estimator for the parameter norm $R$. First, we establish the minimax lower bound $Ω(\sqrt{R^3/n})$ for norm estimation. We then improve the best known norm estimation error rate of the MLE, i.e., $O(\sqrt{R^3d/n})$ from Chardon, Lerasle and Mourtada (2024), to $\tilde{O}(\sqrt{R^3/n}+R^2d/n)$. The additional term, $R^2d/n$, appears to be the intrinsic bias of the MLE, as evidenced by the high-dimensional asymptotic theory of Zhao, Sur and Candes (2022) and numerical examples. We show that, however, this additional term is not information-theoretically necessary. To this end, we construct an efficient debiased norm estimator that achieves the error rate $O(\sqrt{R^3/n})$ and is therefore minimax optimal. Combining this with the optimal direction estimator given by the MLE, we establish the minimax optimal rate $Θ(\sqrt{Rd/n}+\sqrt{R^3/n})$ for estimating $\mathbfθ^*$, as well as the improved finite-sample error rate $\tilde{O}(\sqrt{Rd/n}+\sqrt{R^3/n}+R^2d/n)$ for the MLE. Numerical experiments demonstrate that the proposed minimax optimal estimators outperform the MLE.

0.31

From PAC bounds on the generalization error to PAC bounds on the parameter estimation error

cs.LG ·2026-11-06 ·Mihaly Petreczky, John Leth, Emma Lukacs, Balint Daroczy
similarity 0.57
Abstract · 10 pages, 0 figures, under submission to SysID2027

Probably Approximately Correct (PAC) bounds on the generalization error (the difference between the empirical prediction error computed on the training data and its expectation) are an established tool in statistical machine learning, and there have been several extensions of these bounds to dynamical systems. In parallel, in recent years, there has been growing interest in finite-sample bounds for parameter estimation error in system identification. The latter bounds tend to be algorithm-specific, relying on reformulating the identification algorithm as a least-squares regression problem. This is in contrast to classical PAC bounds, which are algorithm-agnostic and can be applied to any identification algorithm based on minimizing a cost function involving the empirical loss. In this paper, we show how oracle bounds derived from PAC bounds can be used to derive finite-sample bounds for dynamical systems. As an illustration of this approach, we present finite-sample parameter-estimation bounds for linear and bilinear systems with bounded inputs and noise. These bounds apply to identification algorithms based on minimizing a regularized empirical loss function. The key ingredient is to establish an oracle inequality, i.e., an inequality between the true prediction error of the learned model and the minimum achievable true prediction error, and then use either the convexity of the true loss or system-theoretic properties of the loss function to derive bounds on the parameter estimation error. We also analyze the rates of the obtained bounds and compare them with those of classical asymptotic results.

0.11