Point Estimators
The probability distributions met so far are parameterized by unknown quantities (means, variances, covariance
matrices) that must be determined from observed data. The fundamental question of
statistical inference is: given a sample \(\mathcal{D} = \{x_1, \ldots, x_n\}\), how do we estimate
the true parameter \(\theta\) of the underlying population?
Definition: Point Estimator
Let the population parameter \(\theta\) take values in a parameter space
\(\Theta \subseteq \mathbb{R}^k\) (with \(k = 1\) for scalar parameters). A point estimator of
\(\theta\) is a function
\[
\hat{\theta} : \mathbb{R}^n \to \Theta, \quad (X_1, \ldots, X_n) \mapsto \hat{\theta}(X_1, \ldots, X_n)
\]
of the sample random variables that does not itself depend on \(\theta\). Such a function is called a
statistic. The numerical value \(\hat{\theta}(x_1, \ldots, x_n)\) computed from observed data is
the corresponding point estimate.
For example, the sample mean
\[
\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i
\]
is a point estimator of the population mean \(\mu\).
Since \(\hat{\theta}\) is a function of random variables, it is itself a random variable with its own distribution, called
the sampling distribution. A natural question is: how close is \(\hat{\theta}\) to the true parameter
\(\theta\)? Two fundamental properties characterize the quality of an estimator. Its bias measures
systematic error, and its variance measures precision across samples.
Definition: Bias
The bias of an estimator \(\hat{\theta}\) of a parameter \(\theta\) is
\[
\operatorname{Bias}(\hat{\theta}) := \mathbb{E}[\hat{\theta}] - \theta.
\]
An estimator with \(\operatorname{Bias}(\hat{\theta}) = 0\) for every \(\theta \in \Theta\) is called
unbiased.
The variance of an estimator is the standard
variance
of \(\hat{\theta}\) viewed as a random variable:
\[
\operatorname{Var}(\hat{\theta}) = \mathbb{E}\!\left[(\hat{\theta} - \mathbb{E}[\hat{\theta}])^2\right].
\]
An ideal estimator has both low bias and low variance. However, these two goals often conflict. Reducing one may increase
the other. The mean squared error provides a single criterion that balances both considerations.
Definition: Mean Squared Error (MSE)
The mean squared error of an estimator \(\hat{\theta}\) of a parameter \(\theta\) is
\[
\operatorname{MSE}(\hat{\theta}) := \mathbb{E}\!\left[(\hat{\theta} - \theta)^2\right].
\]
A short calculation shows that the MSE decomposes into the variance and the squared bias of the estimator.
Bias-variance decomposition:
\[
\begin{align*}
\operatorname{MSE}(\hat{\theta}) &= \mathbb{E}\left[(\hat{\theta} - \theta)^2\right] \\\\
&= \mathbb{E}\left[(\hat{\theta} - \mathbb{E}[\hat{\theta}] + \mathbb{E}[\hat{\theta}] - \theta)^2\right] \\\\
&= \mathbb{E}\left[(\hat{\theta} - \mathbb{E}[\hat{\theta}])^2
+ 2(\hat{\theta} - \mathbb{E}[\hat{\theta}])(\mathbb{E}[\hat{\theta}] - \theta)
+ (\mathbb{E}[\hat{\theta}] - \theta)^2\right].
\end{align*}
\]
Using the linearity of expectation,
we distribute \(\mathbb{E}\). Note that \(\theta\) is a fixed population parameter, and \(\mathbb{E}[\hat{\theta}]\)
is a constant. Thus, \((\mathbb{E}[\hat{\theta}] - \theta)\) is treated as a constant:
\[
\begin{align*}
\operatorname{MSE}(\hat{\theta}) &= \mathbb{E}\left[(\hat{\theta} - \mathbb{E}[\hat{\theta}])^2\right]
+ 2(\mathbb{E}[\hat{\theta}] - \theta)\mathbb{E}\left[\hat{\theta} - \mathbb{E}[\hat{\theta}]\right]
+ \mathbb{E}\left[(\mathbb{E}[\hat{\theta}] - \theta)^2\right] \\\\
&= \operatorname{Var}(\hat{\theta}) + 2(\mathbb{E}[\hat{\theta}] - \theta)(0) + [\operatorname{Bias}(\hat{\theta})]^2 \\\\
&= \operatorname{Var}(\hat{\theta}) + [\operatorname{Bias}(\hat{\theta})]^2.
\end{align*}
\]
Note: The cross term vanishes because the expected deviation from the mean is zero:
\(\mathbb{E}\left[\hat{\theta} - \mathbb{E}[\hat{\theta}]\right] = \mathbb{E}[\hat{\theta}] - \mathbb{E}[\hat{\theta}] = 0\).
The MSE serves as a criterion for comparison. Among competing estimators, we prefer the one with the smallest MSE. For an
unbiased estimator, the MSE reduces to the variance alone.
Once an estimator is selected, we quantify its precision using the standard error (SE), which is
the standard deviation of the estimator's sampling distribution. For the sample mean of an i.i.d. sample with
finite variance \(\sigma^2\),
\[
\operatorname{SE}(\bar{X}) = \sqrt{\operatorname{Var}(\bar{X})} = \frac{\sigma}{\sqrt{n}}.
\]
Notice that the standard error decreases as \(n\) grows, which confirms the intuition that more data yields more precise
estimates. With the concept of an estimator and its quality in hand, we now ask: what principle should guide our choice of
estimator in the first place?
Likelihood Functions
Before we can choose an optimal estimator, we need a way to measure how well a candidate parameter value \(\theta\)
explains the observed data. The key insight is a change of perspective. The same mathematical expression
that gives the probability (or probability density) of data given a parameter can also be viewed as a function of the
parameter given fixed data. This reversal of roles leads to the likelihood function.
Definition: Likelihood Function
Let \(f(\,\cdot \mid \theta)\) denote the joint p.d.f. (or p.m.f.) of a sample \((X_1, \ldots, X_n)\), parameterized
by an unknown parameter \(\theta \in \Theta\). Once the data \((x_1, \ldots, x_n)\) have been observed, they are no
longer free variables. Only \(\theta\) remains unknown. The likelihood function of \(\theta\) given
the observed data is the function
\[
L(\theta \mid x_1, \ldots, x_n) := f(x_1, \ldots, x_n \mid \theta),
\]
now regarded as a function of \(\theta\) with the observed values held fixed. We often write \(L(\theta)\) when the
data are clear from context.
For an i.i.d. sample, that is, independent observations sharing a common p.d.f. or p.m.f., the joint p.d.f. or
p.m.f. factorizes into the product of the individual ones, and
\[
L(\theta) = \prod_{i=1}^n f(x_i \mid \theta).
\]
For two observations this factorization is the
definition of independence, and for \(n\)
observations we take independence to mean the analogous factorization into all \(n\) individual p.d.f.s or p.m.f.s.
The crucial distinction is one of interpretation. The expression \(\prod_{i=1}^n f(x_i \mid \theta)\) is the same
mathematical formula in both readings. Read as a probability (or density) it is a function of \(x\) with \(\theta\)
fixed, and read as a likelihood it is a function of \(\theta\) with \(x\) fixed. This duality is often expressed as
\[
\underbrace{L(\theta \mid x_1, \ldots, x_n)}_{\text{After sampling: function of } \theta}
= \underbrace{\prod_{i=1}^n f(x_i \mid \theta)}_{\text{Before sampling: function of } x}.
\]
With the likelihood function defined, we can now state the most widely used principle for parameter estimation.
Maximum Likelihood Estimation
In machine learning, model fitting (or training) is the process of estimating unknown parameters
\(\boldsymbol{\theta} = (\theta_1, \ldots, \theta_k)\) from sample data
\(\mathcal{D} = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}\). This is typically framed as the optimization of an objective
(or loss) function over \(\boldsymbol{\theta}\). A widely used choice in frequentist inference is to select the parameter
under which the observed data are most probable or, for continuous data, have the highest density.
Convention. In what follows we allow \(\mathbf{x}_i \in \mathbb{R}^d\) in general. The univariate case
\(x_i \in \mathbb{R}\) used in the examples below is the special case \(d = 1\). Boldface marks vector-valued quantities
and lightface marks scalar ones.
Definition: Maximum Likelihood Estimator
The maximum likelihood estimator (MLE) is defined as
\[
\hat{\boldsymbol{\theta}}_{\text{MLE}} = \arg\max_{\boldsymbol{\theta}}\, L(\boldsymbol{\theta})
\]
where \(L(\boldsymbol{\theta})\) is the likelihood function for the sample data \(\mathcal{D}\).
The argmax need not exist or be unique in general. Compactness of \(\Theta\) with continuous \(L\) gives existence, and
strict concavity of \(\ln L\) on the set where \(L \gt 0\), when that set is convex and nonempty, gives uniqueness. The
binomial example below satisfies both. In the normal example the parameter space is not compact, and existence is settled
directly, by tracking the likelihood at the edges of that space.
Since the logarithm is a strictly increasing function, maximizing \(L\) is equivalent to maximizing \(\ln L\).
Working with the log-likelihood is preferred in practice for two reasons: it converts products
into sums (improving numerical stability) and simplifies differentiation.
If \(L(\boldsymbol{\theta})\) is differentiable in \(\boldsymbol{\theta}\) and the maximum is attained at an interior
point of the parameter space, the MLE satisfies the score equation. For an i.i.d. sample,
\[
\begin{align*}
\nabla_{\boldsymbol{\theta}} \ln L(\boldsymbol{\theta})
&= \nabla_{\boldsymbol{\theta}} \sum_{i=1}^n \ln f(\mathbf{x}_i \mid \boldsymbol{\theta}) \\\\
&= \sum_{i=1}^n \nabla_{\boldsymbol{\theta}} \ln f(\mathbf{x}_i \mid \boldsymbol{\theta}) \\\\
&= \mathbf{0}.
\end{align*}
\]
The score equation provides only a necessary condition. Critical points must be checked against the second-order
condition, a negative-definite Hessian of \(\ln L\), to confirm a local maximum. Boundary maxima and non-differentiable
cases must be handled separately. This gradient of the log-likelihood is called the score function, which
plays a central role in the Fisher information theory.
Example 1: Binomial Distribution \(X \sim b(n, p) \)
We begin with a discrete example. Consider flipping a coin \(n\) times and observing \(k\) heads.
Each flip is a Bernoulli trial, and the number of heads in the \(n\) flips follows \(b(n, \theta)\). We can equivalently
view the data as a single observation \(X \sim b(n, \theta)\) with likelihood \(L(\theta) = P(X = k \mid \theta)\), or as
\(n\) i.i.d. Bernoulli trials with likelihood \(\prod_{i=1}^n \theta^{x_i}(1-\theta)^{1-x_i}\) where \(\sum x_i = k\).
Both views yield the same log-likelihood up to a \(\theta\)-independent constant. We adopt the single-observation view
below, so the symbol \(n\) here denotes the number of trials in the binomial experiment, not the sample size of
the previous section (which is one in this view).
Example:
Let \(X \sim b(n, \theta)\), where \(\theta \in [0, 1]\) is the probability of success. If we observe \(X = k\)
successes in \(n\) trials, the likelihood function is the probability mass function:
\[
L(\theta) = P(X = k \mid \theta) = \binom{n}{k} \theta^k (1-\theta)^{n-k}.
\]
Taking the natural logarithm, we get:
\[
\ln L(\theta) = \ln \binom{n}{k} + k \ln(\theta) + (n-k) \ln(1-\theta).
\]
To find \(\hat{\theta}_{\text{MLE}}\), we take the derivative with respect to \(\theta\) and set it to zero. Note that
the combinatorial term \(\ln \binom{n}{k}\) is a constant with respect to \(\theta\) and vanishes:
\[
\begin{align*}
\frac{d}{d\theta} \ln L(\theta)
&= \frac{k}{\theta} - \frac{n-k}{1-\theta} \\\\
&= \frac{k(1-\theta) - (n-k)\theta}{\theta(1-\theta)} \\\\
&= \frac{k - n\theta}{\theta(1-\theta)} \\\\
&= 0,
\end{align*}
\]
which holds (for \(\theta \in (0,1)\)) if and only if \(k = n\theta\). Therefore,
\[
\hat{\theta}_{\text{MLE}} = \frac{k}{n}.
\]
For \(0 \lt k \lt n\) the second derivative
\(\frac{d^2}{d\theta^2}\ln L(\theta) = -k/\theta^2 - (n-k)/(1-\theta)^2 \lt 0\) on \((0,1)\), so this critical point
is the unique interior maximum. If \(k = 0\) or \(k = n\) the likelihood is monotone on \([0,1]\) and attains its
maximum at the endpoint \(\theta = k/n\), so the same formula holds.
The estimator \(k/n\) is exactly the sample proportion \(\hat{p} = X/n\), so the intuitively natural
choice coincides with the MLE. Writing \(p\) for the success probability \(\theta\), the sampling distribution of
\(\hat{p}\) follows from \(X \sim b(n, p)\):
\[
\begin{align*}
\mathbb{E}[\hat{p}] &= \frac{1}{n}\mathbb{E}[X] = p, \\\\
\operatorname{Var}(\hat{p}) &= \frac{1}{n^2}\operatorname{Var}(X) = \frac{p(1-p)}{n},
\end{align*}
\]
so \(\hat{p}\) is unbiased, with variance shrinking as \(1/n\). We now turn to a continuous example where the MLE must be
found for two parameters simultaneously.
Example 2: Normal Distribution
Given an i.i.d. sample \(X_1, X_2, \ldots, X_n\) from the
normal distribution
\(\mathcal{N}(\mu, \sigma^2)\), we seek the MLEs for both parameters.
Example:
With observed values \(\mathcal{D} = \{x_1, x_2, \ldots, x_n\}\) and per-observation density
\[
f(x \mid \mu, \sigma^2) = \frac{1}{\sigma \sqrt{2\pi}}\exp \left\{- \frac{(x - \mu)^2}{2\sigma^2}\right\},
\]
the likelihood function is
\[
\begin{align*}
L(\mu, \sigma^2) &= \prod_{i=1}^n f(x_i \mid \mu, \sigma^2) \\\\
&= \left (\frac{1}{\sigma \sqrt{2\pi}}\right)^n \exp \left\{-\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2 \right\}.
\end{align*}
\]
The log-likelihood function is given by
\[
\begin{align*}
\ln L(\mu, \sigma^2) &= n \ln \left(\frac{1}{\sigma \sqrt{2\pi}}\right) -\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2 \\\\
&= -n \ln (\sigma) - n \ln (\sqrt{2\pi}) -\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2 \\\\
&= -\frac{n}{2} \ln (\sigma^2) - \frac{n}{2}\ln (2\pi) -\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2. \tag{1}
\end{align*}
\]
At any interior critical point of \(\ln L\), both partial derivatives must vanish simultaneously. Setting the partial
derivative of (1) with respect to \(\mu\) equal to zero:
\[
\begin{align*}
\frac{\partial \ln L(\mu, \sigma^2)}{\partial \mu}
&= \frac{1}{\sigma^2} \sum_{i=1}^n (x_i - \mu) \\\\
&= \frac{1}{\sigma^2}\!\left(\sum_{i=1}^n x_i - n\mu\right) \\\\
&= 0,
\end{align*}
\]
which gives
\[
\hat{\mu}_{\text{MLE}} = \frac{1}{n} \sum_{i=1}^n x_i = \bar{x}.
\]
Similarly, taking the partial derivative of (1) with respect to the variance \(\sigma^2\) (treating \(\sigma^2\) as a
single variable) and setting it to zero:
\[
\begin{align*}
\frac{\partial \ln L(\mu, \sigma^2)}{\partial (\sigma^2)}
&= -\frac{n}{2\sigma^2} + \frac{1}{2(\sigma^2)^2}\sum_{i=1}^n (x_i - \mu)^2 \\\\
&= 0,
\end{align*}
\]
which holds if and only if
\[
\sigma^2 = \frac{1}{n}\sum_{i=1}^n (x_i - \mu)^2.
\]
Substituting the value \(\hat{\mu}_{\text{MLE}} = \bar{x}\) obtained above into this equation gives the
joint critical point
\[
\hat{\sigma}^2_{\text{MLE}} = \frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2.
\]
Assume the observed values are not all equal, so that \(\hat{\sigma}^2_{\text{MLE}} \gt 0\). As \(\sigma^2 \to 0^+\)
or \(\sigma^2 \to \infty\) the likelihood then vanishes, and \((\hat{\mu}_{\text{MLE}}, \hat{\sigma}^2_{\text{MLE}})\)
is the unique interior critical point, so it is the global maximum. If the values are all equal the likelihood is
unbounded and no maximizer exists.
Note that the MLE for the variance divides by \(n\), not \(n - 1\). This means \(\hat{\sigma}^2_{\text{MLE}}\) is a
biased estimator of \(\sigma^2\), with
\(\mathbb{E}[\hat{\sigma}^2_{\text{MLE}}] = \frac{n-1}{n}\sigma^2\). Replacing \(n\) with \(n-1\), a step known as
Bessel's correction, gives the unbiased sample variance
\(s^2 = \frac{1}{n-1}\sum_{i=1}^n (x_i - \bar{x})^2\) for \(n \geq 2\).
Verification of the bias:
Let the sample be i.i.d. with mean \(\mu\) and finite variance \(\sigma^2\). Since \(\mathbb{E}[\sum_i X_i] = n\mu\),
\[
\operatorname{Var}\Bigl(\sum_{i=1}^n X_i\Bigr) = \mathbb{E}\Bigl[\Bigl(\sum_{i=1}^n (X_i - \mu)\Bigr)^2\Bigr].
\]
Multiplying out the square and using linearity of expectation over the finitely many terms gives \(\sum_{i=1}^n \operatorname{Var}(X_i)\)
plus the cross terms \(\mathbb{E}[(X_i - \mu)(X_j - \mu)] = \operatorname{Cov}(X_i, X_j)\) for \(i \neq j\), each finite since
\(|ab| \leq (a^2 + b^2)/2\). Each cross term vanishes because
independence implies zero covariance
and \(X_i\), \(X_j\) are independent. Independence of the sample is the \(n\)-fold factorization in the
likelihood function definition. Summing or integrating that
factorization over the other \(n - 2\) variables, each of whose factors contributes \(1\), leaves the product of the \(i\)-th and
\(j\)-th factors, and the same computation over \(n - 1\) variables shows that each factor is the marginal of its variable. Hence \(\operatorname{Var}\bigl(\sum_{i=1}^n X_i\bigr) = n\sigma^2\),
and the
Variance of a Linear Transformation
with \(a = 1/n\) then gives \(\operatorname{Var}(\bar{X}) = \sigma^2/n\). Expand using
\(\sum_{i=1}^n (X_i - \bar{X})^2 = \sum_{i=1}^n X_i^2 - n\bar{X}^2\) and take expectations. Expanding the square in
the definition of variance and using linearity of expectation gives \(\mathbb{E}[X_i^2] = \sigma^2 + \mu^2\) and
\(\mathbb{E}[\bar{X}^2] = \operatorname{Var}(\bar{X}) + (\mathbb{E}[\bar{X}])^2 = \sigma^2/n + \mu^2\):
\[
\begin{align*}
\mathbb{E}\!\left[\sum_{i=1}^n (X_i - \bar{X})^2\right]
&= \sum_{i=1}^n \mathbb{E}[X_i^2] - n\,\mathbb{E}[\bar{X}^2] \\\\
&= n(\sigma^2 + \mu^2) - n\!\left(\frac{\sigma^2}{n} + \mu^2\right) \\\\
&= (n - 1)\sigma^2.
\end{align*}
\]
Dividing by \(n\) gives \(\mathbb{E}[\hat{\sigma}^2_{\text{MLE}}] = \frac{n-1}{n}\sigma^2\), confirming the bias.
The finite-sample bias of \(\hat{\sigma}^2_{\text{MLE}}\) illustrates a general feature of MLE. An MLE can be biased for
finite \(n\), yet under standard regularity conditions (identifiability, compactness of the parameter
space, continuity and domination of the log-likelihood) it is consistent. Consistency means
\(\hat{\boldsymbol{\theta}}_{\text{MLE}} \xrightarrow{P} \boldsymbol{\theta}_{\text{true}}\) as \(n \to \infty\), in the
sense of convergence in probability.
A classical consistency theorem along these lines is due to Wald (1949).
Connections to Machine Learning
MLE is the default parameter estimation method across machine learning. Training a
logistic regression model is equivalent
to minimizing the negative log-likelihood (cross-entropy loss). Training a
neural network with mean squared error loss
is equivalent to MLE under a Gaussian noise assumption. The
exponential family contains the Bernoulli and Gaussian likelihoods
behind these examples, and when its natural parameter depends linearly on the parameters being estimated, the MLE,
where it exists, is characterized by moment matching. In
Bayesian inference, maximum-a-posteriori estimation under a uniform
(flat) prior reduces to MLE, and adding a squared \(\ell^2\) or an \(\ell^1\) penalty to the negative log-likelihood
is equivalent to choosing a zero-mean Gaussian or Laplace prior, respectively.
MLE provides point estimates, single "best guess" values for the parameters. But how confident should we be in
these estimates? Hypothesis testing and
confidence intervals provide principled frameworks for quantifying the uncertainty inherent in
any statistical estimate.