In statistics, point estimation involves the use of sample data to calculate a single value (known as a point estimate, since it identifies a point rather than an interval), which serves as a "best guess" or "best estimate" of an unknown quantity, for example, the population mean, the variance of a distribution, or a model parameter (in a parametric model).
Point estimation can be contrasted with interval estimation: interval estimates are typically either confidence intervals, in the case of frequentist inference, or credible intervals, in the case of Bayesian inference. More generally, a point estimator can be contrasted with a set estimator. Examples are given by confidence sets or credible sets. A point estimator can also be contrasted with a distribution estimator. Examples are given by confidence distributions, randomized estimators, and Bayesian posteriors.
Properties of point estimators
Biasedness
The bias is defined as the difference between the expected value of the estimator and the true value of the population parameter being estimated. It can also be described that the closer the expected value of a parameter is to the measured parameter, the lesser the bias. When the estimated number and the true value is equal, the estimator is considered unbiased. This is called an unbiased estimator. The estimator will become a best unbiased estimator if it has minimum variance. However, a biased estimator with a small variance may be more useful than an unbiased estimator with a large variance.Dekking et al. (2005), p. 305
Mathematically speaking, if is an estimator based on a random sample drawn from a distribution , the difference is called the bias of . The estimator is called unbiased for the parameter if the bias is zero, irrespective of the value of .Dekking et al. (2005), p. 290 Otherwise, is called biased. Examples of unbiased estimators are the sample mean and the unbiased sample variance.
The concept of (un-)biasedness can be generalized to other metrics than the mean. An unbiased estimator fulfillsCasella & Berger (2002), p. 58Thus, a more general condition for unbiasedness can be defined byfor some function . For example, if , then the estimator is called median-unbiased, since the median is a minimiser of the mean absolute error.
Consistency
A point estimator is called consistent, if the probability that the estimate is close to the true value tends to 1 as the sample size grows to infinity. If the estimate (almost) surely gets arbitrarily close to the true value, eventually, as the sample size grows to infinity, then the estimator is even called strongly consistent. Intuitively, a consistent estimator will be "probably approximately correct", and a strongly consistent estimator even "surely approximately correct", if the sample size is large enough. In the special case, where an estimator is unbiased, it is already consistent, if its variance decreases to zero as the sample size grows to infinity.
Efficiency
Efficiency is a property used to investigate the variance of unbiased estimators. According to the bias-variance decomposition, the variance of an unbiased estimator is equal to its mean squared error (MSE), which is often used as a measure for the approximation error of an estimator. Thus, it is desirable to seek for (unbiased) estimators that have minimal variance. If the distribution from which the data is drawn is somehow "well-behaved", it can be shown that the variance of "well-behaved" unbiased estimators cannot be smaller than a certain threshold (that is, the Cramér–Rao bound). An estimator whose variance is exactly is then called efficient.
More precisely, for a random sample drawn from a distribution with parameter (efficiency can also be defined in a similar manner for parameter vectors) fulfilling the Cramér–Rao regularity conditions, any regular unbiased estimator of the parameter fulfills the Cramér–Rao bound
where is the Fisher-information of . If for any value of , then is called efficient. In fact, it can be shown that is efficient, if and only if, is an exponential family and (actually ) is the natural sufficient statistic.Lehmann & Casella (1998), p. 120–121Casella & Berger (2002), p. 341
Sometimes, the variance of an estimator is compared to the variance of some other estimator (rather than the Cramér–Rao bound): For two unbiased estimators and of the same parameter , the estimator would be called more efficient than if , irrespective of the value of .Dekking et al. (2005), p. 303 One can also say that the most efficient estimators are the ones with the least variability of outcomes. The notion of (relative) efficiency can be extended to biased estimators by saying that an estimator is more efficient than an estimator (for the same parameter of interest), if the MSE of is smaller than the MSE of .Dekking et al. (2005), p. 305
Uniform minimum variance and unbiasedness (UMVU)
The concept of efficiency is restricted to estimators and distributions, which fulfill the Cramér–Rao regularity conditions. However, there are other settings, which do not fit into this framework, where there still exist unbiased estimators with uniformly minimal variance over all parameters . Such estimators are then called (uniformly) minimum-variance unbiased estimators (UMVUE/MVUE).
Given a distribution with some unknown parameter , an unbiased estimator is an UMVUE if for any other unbiased estimator it holdsLehmann & Casella (1998), p. 85
for any .
Sufficiency
In statistics, the job of a statistician is to interpret the data that they have collected and to draw statistically valid conclusion about the population under investigation. But in many cases the raw data, which are too numerous and too costly to store, are not suitable for this purpose. Therefore, the statistician would like to condense the data by computing some statistics and to base their analysis on these statistics so that there is no loss of relevant information in doing so, that is the statistician would like to choose those statistics which exhaust all information about the parameter, which is contained in the sample.
We define sufficient statistics as follows: Let be a random sample. A statistic is said to be sufficient for (or for the family of distribution) if the conditional distribution of given does not depend on .Sahu et al. (2015), p. 3
Asymptotic properties
Often, one is interested in the behaviour of a statistical estimation procedure as the sample size tends to infinity. Additionally to consistency, desirable asymptotic properties of an estimator for are:
asymptotic unbiasedness: Roughly speaking, the distribution of approaches a distribution with zero mean as tends to infinity. More precisely, converges in distribution to a distribution with zero mean as tends to infinity. If the expectation of does not explode when goes to infinity, then this is the same as saying that the bias converges to zero (though it might be non-zero for finite sample sizes). Every consistent estimator is asymptotically unbiased. asymptotic normality: The distribution of approaches a normal distribution as tends to infinity. That is to say, the rescaled estimator , for some sequences and , converges in distribution to a standard normal distribution. asymptotic efficiency: Assuming that is the parameter of a parametric model and that is both asymptotically normal and asymptotically unbiased, the distribution of converges, as the sample size tends to infinity, to a distribution whose variance is the inverse of the Fisher information, that is, the smallest possible variance for a regular unbiased estimator based on one data point.
Estimation methods Below are some commonly used methods for estimating unknown parameters. The methods vary in their domain of applicability and on the underlying statistical paradigm (frequentist or Bayesian).
Frequentist methods
Least square estimation (LSE)
The method of least squares is a parameter estimation procedure popular for regression, where observations need to be expressed as functions of the covariates via
for some function depending on a parameter vector and a random variable accounting for random noise in the data. The ordinary least squares estimator (OLS) assumes that is a linear function of and estimates the parameter by minimizing the sum of squared differences, called residuals:
In fact, if the noise variables are independent and identically normally distributed with mean zero, then the OLS coincides with the maximum likelihood estimator.
If the rank of the matrix of covariates is equal to , then the OLS is uniquely given by
and it is the best linear unbiased estimator (BLUE) by the Gauß–Markov theorem. If, as above, the noise variables are independent and identically normally distributed, then the OLS is even UMVU for the entire class of unbiased estimators.Sahu et al. (2015), p. 56–57
There are multiple variants of least squares estimation:
weighted least squares (WLS) estimation: The residuals are weighted with different weighting factors . generalized least squares (GLS) estimation: The noise variables are allowed to be correlated. nonlinear least squares estimation: The function can depend non-linearly on the parameter . regularized least squares estimation: The parameter is chosen such that the sum of squared residuals plus a regularization term is minimized. Typical choices for the regularization are the euclidian norm , leading to Ridge regression, or the 1-norm , leading to Lasso regression.
Maximum likelihood estimation (MLE)
The method of maximum likelihood, due to R.A. Fisher, is arguably the most important general method for estimating the parameters of a parametric model.Sahu et al. (2015), p. 48 According to this method, the best estimate for the unknown model parameter is the one that maximizes the probability (or probability density) of observing the data that has been observed. This probability as a function of the model parameters is called the likelihood, giving the method its name.Dodge (2008), p. 334
In mathematical terms, the methods works as follows: Let denote a random data sample with joint probability density function or probability mass function depending on the vector of model parameters. The function is called the likelihood function, often denoted by . Assuming that true model parameter lies in a set , a maximum likelihood estimator fulfills
In practice, the likelihood function is often differentiable, in which case is a solution of the equationsSahu et al. (2015), p. 48–49
Under certain assumptions on the likelihood, the MLE is strongly consistent, asymptotically efficient and asymptotically normal.
Method of moments (MoM)
The method of moments is one of the oldest methods of estimation, which was first introduced by Karl Pearson in 1895.Sahu et al. (2015), p. 47Dodge (2008), p. 349 This method uses the functional relationship between quantities of interest of a distribution (e.g. the variance or model parameters) and the moments of that distribution, leading to a set of equations that are solved for the aforementioned quantities. Motivated by the law of large numbers, the moments of the distribution are estimated by the sample moments, that is, the arithmetic averages of powers of the data values.Dodge (2008), p. 348–349 The method stands out for its simplicity and wide domain of applicatibility, but often do not yield the best estimators and sometimes even no estimator at all.Shao (2003), p. 207 There also exists generalizations of the method of moments.
Mathematically speaking, for a distribution depending on some vector of parameters , the equations of the first moments of are established:
If these equations have a solution, then there exists functions such that the parameters can be expressed by the moments via
Given a random sample from , the moments are then replaced by the sample moments , , yielding the moment estimators: