Convergence

Proof (via Chebyshev's inequality):

First, the variance of the sample mean. By the scaling rule for variance applied to \(\bar{X}_n = \tfrac{1}{n}\sum_{i=1}^n X_i\), \[ \operatorname{Var}(\bar{X}_n) = \frac{1}{n^2}\,\operatorname{Var}\!\left(\sum_{i=1}^n X_i\right). \]

Expanding the variance of the sum, \[ \begin{align*} \operatorname{Var}\!\left(\sum_{i=1}^n X_i\right) &= \mathbb{E}\!\left[\left(\sum_{i=1}^n (X_i - \mu)\right)^{\!2}\right] \\\\ &= \sum_{i=1}^n \operatorname{Var}(X_i) + \sum_{i \neq j} \mathbb{E}[(X_i - \mu)(X_j - \mu)]. \end{align*} \] Since the \(X_i\) are independent, independence factors expectations, so \(\mathbb{E}[X_i X_j] = \mu^2\) for \(i \neq j\). Expanding the product and using linearity of expectation, iterated over the terms of the expansion, \[ \begin{align*} \mathbb{E}[(X_i - \mu)(X_j - \mu)] &= \mathbb{E}[X_i X_j] - \mu\,\mathbb{E}[X_i] - \mu\,\mathbb{E}[X_j] + \mu^2 \\\\ &= \mu^2 - \mu^2 - \mu^2 + \mu^2 = 0. \end{align*} \] The cross terms vanish, leaving \(\operatorname{Var}(\sum_i X_i) = n\sigma^2\), and hence \(\operatorname{Var}(\bar{X}_n) = \sigma^2/n\).

Next, the deviation bound. Set \(Z = (\bar{X}_n - \mu)^2 \geq 0\). For any \(\epsilon \gt 0\), \[ \mathbb{E}[Z] \geq \mathbb{E}\!\left[Z \cdot \mathbb{1}\{Z \geq \epsilon^2\}\right] \geq \epsilon^2 \cdot P(Z \geq \epsilon^2). \]

The first inequality holds because \(Z \cdot \mathbb{1}\{Z \lt \epsilon^2\} \geq 0\), and the second because the indicator restricts to the region \(Z \geq \epsilon^2\). Rearranging and using \(\{Z \geq \epsilon^2\} = \{|\bar{X}_n - \mu| \geq \epsilon\}\), \[ \begin{align*} P(|\bar{X}_n - \mu| \geq \epsilon) &\leq \frac{\mathbb{E}[(\bar{X}_n - \mu)^2]}{\epsilon^2} \\\\ &= \frac{\operatorname{Var}(\bar{X}_n)}{\epsilon^2} \\\\ &= \frac{\sigma^2}{n\epsilon^2} \\\\ &\longrightarrow 0. \end{align*} \] This is Chebyshev's inequality applied to \(\bar{X}_n - \mu\), and the bound \(\mathbb{E}[Z] \geq \epsilon^2 \cdot P(Z \geq \epsilon^2)\) behind it is Markov's inequality for \(Z\). Both are stated in general form on a later page as the Markov and Chebyshev inequalities.

Remark. The finite-variance assumption above is convenient but not necessary. The SLLN guarantees \(\bar{X}_n \xrightarrow{a.s.} \mu\) under the weaker condition \(\mathbb{E}[|X_1|] \lt \infty\) alone, and almost sure convergence implies convergence in probability (established below). So the WLLN in fact holds whenever the SLLN does. Finite variance is needed only for the simple self-contained argument given here.

Proof:

(\(\xrightarrow{a.s.}\) implies \(\xrightarrow{P}\)). Suppose \(X_n \xrightarrow{a.s.} X\) and fix \(\epsilon \gt 0\). Set \(Y_n = \mathbb{1}\{|X_n - X| \lt \epsilon\}\), so \(Y_n \in [0,1]\) for all \(n\). On the convergence set (which has probability \(1\)), each \(\omega\) eventually satisfies \(|X_n(\omega) - X(\omega)| \lt \epsilon\), so \(Y_n(\omega) \to 1\). Hence \(\liminf_n Y_n = 1\) almost surely, and \(\mathbb{E}[\liminf_n Y_n] = 1\). Applying Fatou's Lemma to the nonnegative sequence \(\{Y_n\}\), \[ \begin{align*} 1 &= \mathbb{E}\!\left[\liminf_{n \to \infty} Y_n\right] \\\\ &\leq \liminf_{n \to \infty} \mathbb{E}[Y_n] \\\\ &= \liminf_{n \to \infty} P(|X_n - X| \lt \epsilon). \end{align*} \]

Since \(P(|X_n - X| \lt \epsilon) \leq 1\) for all \(n\), this forces \(P(|X_n - X| \lt \epsilon) \to 1\), equivalently \(P(|X_n - X| \geq \epsilon) \to 0\).

(\(\xrightarrow{P}\) implies \(\xrightarrow{D}\)). Proved below as Convergence in Probability Implies Convergence in Distribution.

(Partial converse: \(\xrightarrow{D} \Longrightarrow \xrightarrow{P}\) when the limit is a constant \(c\)). Suppose \(X_n \xrightarrow{D} c\). The CDF of the constant \(c\) is the step function \(F(x) = \mathbb{1}\{x \geq c\}\), whose only discontinuity is at \(x = c\). Every other point of \(\mathbb{R}\) is a continuity point.

Fix \(\epsilon \gt 0\). The points \(c - \epsilon\) and \(c + \epsilon/2\) are both continuity points of \(F\), and \[ \begin{align*} \{|X_n - c| \geq \epsilon\} &= \{X_n \leq c - \epsilon\} \cup \{X_n \geq c + \epsilon\} \\\\ &\subseteq \{X_n \leq c - \epsilon\} \cup \{X_n \gt c + \epsilon/2\}, \end{align*} \] so \[ \begin{align*} P(|X_n - c| \geq \epsilon) &\leq F_{X_n}(c - \epsilon) + \bigl(1 - F_{X_n}(c + \epsilon/2)\bigr) \\\\ &\longrightarrow F(c - \epsilon) + \bigl(1 - F(c + \epsilon/2)\bigr) \\\\ &= 0 + (1 - 1) = 0, \end{align*} \] which is convergence in probability.

Proof:

Let \(\epsilon \gt 0\). Since \(f\) is continuous at \(a\), there exists \(\delta \gt 0\) such that \[ |x - a| \lt \delta \Longrightarrow |f(x) - f(a)| \lt \epsilon. \] Taking the contrapositive, \[ |f(x) - f(a)| \geq \epsilon \Longrightarrow |x - a| \geq \delta. \]

Substituting \(X_n\) for \(x\), we obtain \[ P(|f(X_n) - f(a)| \geq \epsilon) \leq P(|X_n - a| \geq \delta). \] As \(n \to \infty\), the right-hand side vanishes by assumption, hence \(f(X_n) \xrightarrow{P} f(a)\).

Proof:

Suppose \(X_n \xrightarrow{P} X\), and let \(x\) be a point of continuity of \(F_X\).

Upper bound:
For any \(\epsilon \gt 0\), partition the event \(\{X_n \leq x\}\) according to whether \(|X_n - X| \lt \epsilon\): \[ \begin{align*} F_{X_n}(x) &= P(X_n \leq x) \\\\ &= P(\{X_n \leq x\} \cap \{|X_n - X| \lt \epsilon\}) + P(\{X_n \leq x\} \cap \{|X_n - X| \geq \epsilon\}) \\\\ &\leq P(X \leq x + \epsilon) + P(|X_n - X| \geq \epsilon) \\\\ &= F_X(x + \epsilon) + P(|X_n - X| \geq \epsilon). \end{align*} \]

The first inequality holds because \(\{X_n \leq x\} \cap \{|X_n - X| \lt \epsilon\}\) implies \(X \lt x + \epsilon\). Since \(X_n \xrightarrow{P} X\), the second term vanishes, giving \[ \limsup_{n \to \infty} F_{X_n}(x) \leq F_X(x + \epsilon). \tag{1} \]

Lower bound:
Similarly, partition the complementary event \(\{X_n \gt x\}\): \[ \begin{align*} 1 - F_{X_n}(x) &= P(X_n \gt x) \\\\ &= P(\{X_n \gt x\} \cap \{|X_n - X| \lt \epsilon\}) + P(\{X_n \gt x\} \cap \{|X_n - X| \geq \epsilon\}) \\\\ &\leq P(X \gt x - \epsilon) + P(|X_n - X| \geq \epsilon) \\\\ &= 1 - F_X(x - \epsilon) + P(|X_n - X| \geq \epsilon). \end{align*} \]

The first inequality holds because \(\{X_n \gt x\} \cap \{|X_n - X| \lt \epsilon\}\) implies \(X \gt X_n - \epsilon \gt x - \epsilon\). Rearranging, \[ F_{X_n}(x) \geq F_X(x - \epsilon) - P(|X_n - X| \geq \epsilon), \] and since \(X_n \xrightarrow{P} X\), the second term vanishes, giving \[ \liminf_{n \to \infty} F_{X_n}(x) \geq F_X(x - \epsilon). \tag{2} \]

Combining (1) and (2): \[ \begin{align*} F_X(x - \epsilon) &\leq \liminf_{n \to \infty} F_{X_n}(x) \\\\ &\leq \limsup_{n \to \infty} F_{X_n}(x) \\\\ &\leq F_X(x + \epsilon). \end{align*} \]

Since \(x\) is a point of continuity of \(F_X\), letting \(\epsilon \to 0\) gives \(F_X(x - \epsilon) \to F_X(x)\) and \(F_X(x + \epsilon) \to F_X(x)\), so \[ \begin{align*} F_X(x) &\leq \liminf_{n \to \infty} F_{X_n}(x) \\\\ &\leq \limsup_{n \to \infty} F_{X_n}(x) \\\\ &\leq F_X(x), \end{align*} \] forcing \(\liminf = \limsup = F_X(x)\), hence \(\lim_{n \to \infty} F_{X_n}(x) = F_X(x)\). Since this holds for every continuity point of \(F_X\), we conclude \(X_n \xrightarrow{D} X\).

Proof:

Step 1: MGF of centered variable.
Throughout this proof, write \(X\) for a representative copy of the common distribution (so \(\mathbb{E}[X] = \mu\), \(\operatorname{Var}(X) = \sigma^2\), and \(M(t) = \mathbb{E}[e^{tX}]\) is the common MGF). Define \(W = X - \mu\). The MGF of \(W\) is: \[ m(t) = \mathbb{E}[e^{t(X-\mu)}] = e^{-\mu t} M(t), \quad t \in (-h, h). \]

Applying the moment-extraction property \(m^{(k)}(0) = \mathbb{E}[W^k]\) (established above for MGFs in their domain of analyticity), together with the linearity of expectation and the definition of variance: \[ \begin{align*} m(0) &= 1, \\\\ m'(0) &= \mathbb{E}[X - \mu] = \mathbb{E}[X] - \mu = 0, \\\\ m''(0) &= \mathbb{E}[(X-\mu)^2] = \operatorname{Var}(X) = \sigma^2. \end{align*} \]

Step 2: Taylor expansion.
The MGF \(m\) is infinitely differentiable on \((-h, h)\), since its power series \(\sum_{k=0}^\infty t^k \mathbb{E}[W^k]/k!\) converges on this interval. Taylor's theorem with Lagrange remainder, a standard result of one-variable calculus, therefore gives for any \(t \in (-h, h)\) a point \(\xi\) between \(0\) and \(t\) such that: \[ m(t) = m(0) + m'(0)t + \frac{m''(\xi)}{2}t^2 = 1 + \frac{m''(\xi)}{2}t^2. \]

We rewrite this as: \[ m(t) = 1 + \frac{\sigma^2 t^2}{2} + \frac{[m''(\xi) - \sigma^2]t^2}{2}. \tag{3} \]

Step 3: MGF of the standardized sum.
Let \(M_n\) denote the MGF of \(Y_n\). Writing the exponential of a sum as a product of exponentials, we obtain \[ \begin{align*} M_n(t) &= \mathbb{E}\!\left[\exp\!\left(t \cdot \frac{\sum_{i=1}^{n}(X_i - \mu)}{\sigma\sqrt{n}}\right)\right] \\\\ &= \mathbb{E}\!\left[\prod_{i=1}^n \exp\!\left(\frac{t(X_i - \mu)}{\sigma\sqrt{n}}\right)\right]. \end{align*} \]

Since the \(X_i\) are independent, so are the functions \(\exp(t(X_i - \mu)/(\sigma\sqrt{n}))\) of them. We take this closure of independence under measurable maps for granted here. Each factor has finite expectation because \(t/(\sigma\sqrt{n})\) lies in \((-h, h)\), so the product rule for expectations of independent variables applies and the expectation factors: \[ M_n(t) = \prod_{i=1}^n \mathbb{E}\!\left[\exp\!\left(\frac{t(X_i - \mu)}{\sigma\sqrt{n}}\right)\right]. \] Each factor equals \(m(t/(\sigma\sqrt{n}))\) by the i.i.d. assumption (the \(X_i\) all share the common centered MGF \(m\)), so \[ M_n(t) = \left[m\!\left(\frac{t}{\sigma\sqrt{n}}\right)\right]^n. \]

Step 4: Substitute and take limit.
Replacing \(t\) with \(\frac{t}{\sigma\sqrt{n}}\) in equation (3): \[ m\left(\frac{t}{\sigma\sqrt{n}}\right) = 1 + \frac{t^2}{2n} + \frac{[m''(\xi_n) - \sigma^2]t^2}{2n\sigma^2} \] where \(\xi_n\) lies between \(0\) and \(t/(\sigma\sqrt{n})\) (the precise sign depending on that of \(t\)). Thus: \[ M_n(t) = \left\{1 + \frac{t^2}{2n} + \frac{[m''(\xi_n) - \sigma^2]t^2}{2n\sigma^2}\right\}^n. \]

As \(n \to \infty\), we have \(|\xi_n| \leq |t|/(\sigma\sqrt{n}) \to 0\), so \(\xi_n \to 0\). By continuity of \(m''\) at \(0\) (which follows from the smoothness of \(m\) on \((-h, h)\) established in Step 2): \[ m''(\xi_n) \to m''(0) = \sigma^2 \Longrightarrow m''(\xi_n) - \sigma^2 \to 0. \]

To close the limit, we use the following generalization of the classical exponential limit \((1 + a_n/n)^n \to e^a\) (valid for any sequence \(a_n \to a\), a standard result of one-variable calculus). If \(c_n \to c\) and \(\epsilon_n \to 0\), then \(\left(1 + \frac{c_n}{n} + \frac{\epsilon_n}{n}\right)^n \to e^c\). (Proof: take logarithms and use the Taylor expansion \(\ln(1 + x) = x + O(x^2)\) for small \(x\), giving \(n \ln(1 + (c_n + \epsilon_n)/n) = (c_n + \epsilon_n) + O((c_n + \epsilon_n)^2/n) \to c\). Then exponentiate.) Applying this with \(c_n \equiv \frac{t^2}{2}\) and \(\epsilon_n = \frac{[m''(\xi_n) - \sigma^2]t^2}{2\sigma^2} \to 0\): \[ \lim_{n \to \infty} M_n(t) = e^{t^2/2}. \]

Step 5: Identify the limit.
The function \(e^{t^2/2}\) is the MGF of \(\mathcal{N}(0,1)\) (verified below). To apply Continuity Theorem for MGFs, we check its hypotheses. Each \(M_n\) is defined on the common open interval \((-h\sigma, h\sigma)\), since \(M_n(t) = [m(t/(\sigma\sqrt{n}))]^n\) requires \(t/(\sigma\sqrt{n}) \in (-h, h)\) and \(\sqrt{n} \geq 1\). The limit function \(e^{t^2/2}\) is an MGF defined on all of \(\mathbb{R}\), and Step 4 established \(M_n(t) \to e^{t^2/2}\) for every \(t \in (-h\sigma, h\sigma)\). Taking \(\delta = h\sigma\) in the theorem yields: \[ Y_n = \frac{\sqrt{n}(\bar{X}_n - \mu)}{\sigma} \xrightarrow{D} \mathcal{N}(0,1). \]

Verification: MGF of \(\mathcal{N}(0,1)\)

We confirm that \(e^{t^2/2}\) is the MGF of the standard normal distribution \(\mathcal{N}(0,1)\), whose density is \(\frac{1}{\sqrt{2\pi}} e^{-x^2/2}\) (see the definition of the normal distribution): \[ \begin{align*} M(t) &= \int_{-\infty}^{\infty} e^{tx} \frac{1}{\sqrt{2\pi}} e^{-x^2/2} \, dx \\\\ &= \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-(x^2 - 2tx)/2} \, dx \\\\ &= \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-[(x-t)^2 - t^2]/2} \, dx \\\\ &= e^{t^2/2} \underbrace{\int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}} e^{-(x-t)^2/2} \, dx}_{= 1} \\\\ &= e^{t^2/2}. \end{align*} \]

The key step is completing the square in the exponent. The remaining integral equals \(1\) because it is the total probability of \(\mathcal{N}(t, 1)\), which (by the same definition) is a probability density integrating to \(1\) over \(\mathbb{R}\).