Convergence

Proof (via Chebyshev's inequality):

First, the variance of the sample mean. By the scaling rule for variance applied to \(\bar{X}_n = \tfrac{1}{n}\sum_{i=1}^n X_i\), \[ \operatorname{Var}(\bar{X}_n) \;=\; \frac{1}{n^2}\,\operatorname{Var}\!\left(\sum_{i=1}^n X_i\right). \] Expanding the variance of the sum, \[ \operatorname{Var}\!\left(\sum_{i=1}^n X_i\right) \;=\; \mathbb{E}\!\left[\left(\sum_{i=1}^n (X_i - \mu)\right)^{\!2}\right] \;=\; \sum_{i=1}^n \operatorname{Var}(X_i) \;+\; \sum_{i \neq j} \mathbb{E}[(X_i - \mu)(X_j - \mu)]. \] Since the \(X_i\) are independent, independence factors expectations, so for \(i \neq j\), \[ \mathbb{E}[(X_i - \mu)(X_j - \mu)] \;=\; \mathbb{E}[X_i - \mu] \cdot \mathbb{E}[X_j - \mu] \;=\; 0. \] The cross terms vanish, leaving \(\operatorname{Var}(\sum_i X_i) = n\sigma^2\), and hence \(\operatorname{Var}(\bar{X}_n) = \sigma^2/n\).

Next, the deviation bound. Set \(Z = (\bar{X}_n - \mu)^2 \geq 0\). For any \(\epsilon > 0\), \[ \mathbb{E}[Z] \;\geq\; \mathbb{E}\!\left[Z \cdot \mathbb{1}\{Z \geq \epsilon^2\}\right] \;\geq\; \epsilon^2 \cdot P(Z \geq \epsilon^2), \] the first inequality because \(Z \cdot \mathbb{1}\{Z < \epsilon^2\} \geq 0\) and the second because the indicator restricts to the region \(Z \geq \epsilon^2\). Rearranging and using \(\{Z \geq \epsilon^2\} = \{|\bar{X}_n - \mu| \geq \epsilon\}\), \[ P(|\bar{X}_n - \mu| \geq \epsilon) \;\leq\; \frac{\mathbb{E}[(\bar{X}_n - \mu)^2]}{\epsilon^2} \;=\; \frac{\operatorname{Var}(\bar{X}_n)}{\epsilon^2} \;=\; \frac{\sigma^2}{n\epsilon^2} \;\longrightarrow\; 0. \] This is Chebyshev's inequality applied to \(\bar{X}_n - \mu\).

Remark. The finite-variance assumption above is convenient but not necessary. The SLLN guarantees \(\bar{X}_n \xrightarrow{a.s.} \mu\) under the weaker condition \(\mathbb{E}[|X_1|] < \infty\) alone, and almost sure convergence implies convergence in probability (established below). So the WLLN in fact holds whenever the SLLN does — finite variance is needed only for the simple self-contained argument given here.

Proof:

(\(\xrightarrow{a.s.}\) implies \(\xrightarrow{P}\)). Suppose \(X_n \xrightarrow{a.s.} X\) and fix \(\epsilon > 0\). Set \(Y_n = \mathbb{1}\{|X_n - X| < \epsilon\}\), so \(Y_n \in [0,1]\) for all \(n\). On the convergence set (which has probability \(1\)), each \(\omega\) eventually satisfies \(|X_n(\omega) - X(\omega)| < \epsilon\), so \(Y_n(\omega) \to 1\). Hence \(\liminf_n Y_n = 1\) almost surely, and \(\mathbb{E}[\liminf_n Y_n] = 1\). Applying Fatou's Lemma to the nonnegative sequence \(\{Y_n\}\), \[ 1 \;=\; \mathbb{E}\!\left[\liminf_{n \to \infty} Y_n\right] \;\leq\; \liminf_{n \to \infty} \mathbb{E}[Y_n] \;=\; \liminf_{n \to \infty} P(|X_n - X| < \epsilon). \] Since \(P(|X_n - X| < \epsilon) \leq 1\) for all \(n\), this forces \(P(|X_n - X| < \epsilon) \to 1\), equivalently \(P(|X_n - X| \geq \epsilon) \to 0\).

(\(\xrightarrow{P}\) implies \(\xrightarrow{D}\)). Proved as Convergence in Probability Implies Convergence in Distribution in the section that follows.

(Partial converse: \(\xrightarrow{D} \Longrightarrow \xrightarrow{P}\) when the limit is a constant \(c\)). Suppose \(X_n \xrightarrow{D} c\). The CDF of the constant \(c\) is the step function \(F(x) = \mathbb{1}\{x \geq c\}\), whose only discontinuity is at \(x = c\); every other point of \(\mathbb{R}\) is a continuity point. Fix \(\epsilon > 0\). The points \(c - \epsilon\) and \(c + \epsilon/2\) are both continuity points of \(F\), and \[ \{|X_n - c| \geq \epsilon\} \;=\; \{X_n \leq c - \epsilon\} \cup \{X_n \geq c + \epsilon\} \;\subseteq\; \{X_n \leq c - \epsilon\} \cup \{X_n > c + \epsilon/2\}, \] so \[ P(|X_n - c| \geq \epsilon) \;\leq\; F_{X_n}(c - \epsilon) + \bigl(1 - F_{X_n}(c + \epsilon/2)\bigr) \;\longrightarrow\; F(c - \epsilon) + \bigl(1 - F(c + \epsilon/2)\bigr) \;=\; 0 + (1 - 1) \;=\; 0, \] which is convergence in probability.

Proof:

Let \(\epsilon > 0\). Since \(f\) is continuous at \(a\), \(\, \exists \delta > 0 \) such that \[ |x - a| < \delta \Longrightarrow |f(x) - f(a)| < \epsilon. \] Taking the contrapositive, \[ |f(x) - f(a)| \geq \epsilon \Longrightarrow |x - a| \geq \delta. \] Substituting \(X_n\) for \(x\), we obtain \[ P(|f(X_n) - f(a)| \geq \epsilon) \leq P(|X_n - a| \geq \delta). \] As \(n \to \infty\), the right-hand side vanishes by assumption, hence \(f(X_n) \xrightarrow{P} f(a)\).

Proof:

Suppose \(X_n \xrightarrow{P} X\), and let \(x\) be a point of continuity of \(F_X\).

Upper bound.
For any \(\epsilon > 0\), partition the event \(\{X_n \leq x\}\) according to whether \(|X_n - X| < \epsilon\): \[ \begin{align*} F_{X_n}(x) &= P(X_n \leq x) \\\\ &= P(\{X_n \leq x\} \cap \{|X_n - X| < \epsilon\}) + P(\{X_n \leq x\} \cap \{|X_n - X| \geq \epsilon\}) \\\\ &\leq P(X \leq x + \epsilon) + P(|X_n - X| \geq \epsilon) \\\\ &= F_X(x + \epsilon) + P(|X_n - X| \geq \epsilon). \end{align*} \] The first inequality holds because \(\{X_n \leq x\} \cap \{|X_n - X| < \epsilon\}\) implies \(X < x + \epsilon\). Since \(X_n \xrightarrow{P} X\), the second term vanishes, giving \[ \limsup_{n \to \infty} F_{X_n}(x) \leq F_X(x + \epsilon). \tag{1} \]

Lower bound:
Similarly, partition the complementary event \(\{X_n > x\}\): \[ \begin{align*} 1 - F_{X_n}(x) &= P(X_n > x) \\\\ &= P(\{X_n > x\} \cap \{|X_n - X| < \epsilon\}) + P(\{X_n > x\} \cap \{|X_n - X| \geq \epsilon\}) \\\\ &\leq P(X > x - \epsilon) + P(|X_n - X| \geq \epsilon) \\\\ &= 1 - F_X(x - \epsilon) + P(|X_n - X| \geq \epsilon). \end{align*} \] The first inequality holds because \(\{X_n > x\} \cap \{|X_n - X| < \epsilon\}\) implies \(X > X_n - \epsilon > x - \epsilon\). Rearranging, \[ F_{X_n}(x) \geq F_X(x - \epsilon) - P(|X_n - X| \geq \epsilon), \] and since \(X_n \xrightarrow{P} X\) the second term vanishes, giving \[ \liminf_{n \to \infty} F_{X_n}(x) \geq F_X(x - \epsilon). \tag{2} \]

Combining (1) and (2): \[ F_X(x - \epsilon) \leq \liminf_{n \to \infty} F_{X_n}(x) \leq \limsup_{n \to \infty} F_{X_n}(x) \leq F_X(x + \epsilon). \] Since \(x\) is a point of continuity of \(F_X\), letting \(\epsilon \to 0\) gives \(F_X(x - \epsilon) \to F_X(x)\) and \(F_X(x + \epsilon) \to F_X(x)\), so \[ F_X(x) \leq \liminf_{n \to \infty} F_{X_n}(x) \leq \limsup_{n \to \infty} F_{X_n}(x) \leq F_X(x), \] forcing \(\liminf = \limsup = F_X(x)\), hence \(\lim_{n \to \infty} F_{X_n}(x) = F_X(x)\). Since this holds for every continuity point of \(F_X\), we conclude \(X_n \xrightarrow{D} X\).

Proof:

Step 1: MGF of centered variable.
Throughout this proof, write \(X\) for a representative copy of the common distribution (so \(\mathbb{E}[X] = \mu\), \(\operatorname{Var}(X) = \sigma^2\), and \(M(t) = \mathbb{E}[e^{tX}]\) is the common MGF). Define \(W = X - \mu\). The MGF of \(W\) is: \[ m(t) = \mathbb{E}[e^{t(X-\mu)}] = e^{-\mu t} M(t), \qquad t \in (-h, h). \] Applying the moment-extraction property \(m^{(k)}(0) = \mathbb{E}[W^k]\) (established above for MGFs in their domain of analyticity), together with the linearity of expectation and the definition of variance: \[ m(0) = 1, \quad m'(0) = \mathbb{E}[X - \mu] = \mathbb{E}[X] - \mu = 0, \quad m''(0) = \mathbb{E}[(X-\mu)^2] = \operatorname{Var}(X) = \sigma^2. \]

Step 2: Taylor expansion.
By Taylor's theorem with Lagrange remainder (a standard result of one-variable calculus, applicable here because the MGF \(m\) is infinitely differentiable on \((-h, h)\) — its power series \(\sum_{k=0}^\infty t^k \mathbb{E}[W^k]/k!\) converges on this interval), for any \(t \in (-h, h)\) there exists \(\xi\) between \(0\) and \(t\) such that: \[ m(t) = m(0) + m'(0)t + \frac{m''(\xi)}{2}t^2 = 1 + \frac{m''(\xi)}{2}t^2. \] We rewrite this as: \[ m(t) = 1 + \frac{\sigma^2 t^2}{2} + \frac{[m''(\xi) - \sigma^2]t^2}{2}. \tag{3} \]

Step 3: MGF of the standardized sum.
Writing the exponential of a sum as a product of exponentials, \[ M_n(t) = \mathbb{E}\!\left[\exp\!\left(t \cdot \frac{\sum_{i=1}^{n}(X_i - \mu)}{\sigma\sqrt{n}}\right)\right] = \mathbb{E}\!\left[\prod_{i=1}^n \exp\!\left(\frac{t(X_i - \mu)}{\sigma\sqrt{n}}\right)\right]. \] Since the \(X_i\) are independent, so are the functions \(\exp(t(X_i - \mu)/(\sigma\sqrt{n}))\) of them. By the product rule for expectations of independent variables, the expectation factors: \[ M_n(t) = \prod_{i=1}^n \mathbb{E}\!\left[\exp\!\left(\frac{t(X_i - \mu)}{\sigma\sqrt{n}}\right)\right]. \] Each factor equals \(m(t/(\sigma\sqrt{n}))\) by the i.i.d. assumption (the \(X_i\) all share the common centered MGF \(m\)), so \[ M_n(t) = \left[m\!\left(\frac{t}{\sigma\sqrt{n}}\right)\right]^n. \]

Step 4: Substitute and take limit.
Replacing \(t\) with \(\frac{t}{\sigma\sqrt{n}}\) in equation (3): \[ m\left(\frac{t}{\sigma\sqrt{n}}\right) = 1 + \frac{t^2}{2n} + \frac{[m''(\xi_n) - \sigma^2]t^2}{2n\sigma^2} \] where \(\xi_n\) lies between \(0\) and \(t/(\sigma\sqrt{n})\) (the precise sign depending on that of \(t\)). Thus: \[ M_n(t) = \left\{1 + \frac{t^2}{2n} + \frac{[m''(\xi_n) - \sigma^2]t^2}{2n\sigma^2}\right\}^n. \] As \(n \to \infty\), we have \(|\xi_n| \leq |t|/(\sigma\sqrt{n}) \to 0\), so \(\xi_n \to 0\). By continuity of \(m''\) at \(0\) (which follows from the smoothness of \(m\) on \((-h, h)\) established in Step 2): \[ m''(\xi_n) \to m''(0) = \sigma^2 \Longrightarrow m''(\xi_n) - \sigma^2 \to 0. \] To close the limit, we use the following generalization of the classical exponential limit \((1 + a_n/n)^n \to e^a\) (valid for any sequence \(a_n \to a\), a standard result of one-variable calculus): if \(c_n \to c\) and \(\epsilon_n \to 0\), then \(\left(1 + \frac{c_n}{n} + \frac{\epsilon_n}{n}\right)^n \to e^c\). (Proof: take logarithms and use the Taylor expansion \(\ln(1 + x) = x + O(x^2)\) for small \(x\), giving \(n \ln(1 + (c_n + \epsilon_n)/n) = (c_n + \epsilon_n) + O((c_n + \epsilon_n)^2/n) \to c\); then exponentiate.) Applying this with \(c_n \equiv \frac{t^2}{2}\) and \(\epsilon_n = \frac{[m''(\xi_n) - \sigma^2]t^2}{2\sigma^2} \to 0\): \[ \lim_{n \to \infty} M_n(t) = e^{t^2/2}. \]

Step 5: Identify the limit.
The function \(e^{t^2/2}\) is the MGF of \(\mathcal{N}(0,1)\) (verified below). To apply Lévy's Continuity Theorem (MGF Version), note that each \(M_n\) is defined on the common open interval \((-h\sigma, h\sigma)\) (since \(M_n(t) = [m(t/(\sigma\sqrt{n}))]^n\) requires \(t/(\sigma\sqrt{n}) \in (-h, h)\), and \(\sqrt{n} \geq 1\)), the limit MGF \(M(t) = e^{t^2/2}\) is defined on all of \(\mathbb{R}\), and Step 4 established \(M_n(t) \to e^{t^2/2}\) for every \(t \in \mathbb{R}\). Taking \(\delta = h\sigma\) in the theorem yields: \[ Y_n = \frac{\sqrt{n}(\bar{X}_n - \mu)}{\sigma} \xrightarrow{D} \mathcal{N}(0,1). \]

Verification: MGF of \(\mathcal{N}(0,1)\)

We confirm that \(e^{t^2/2}\) is the MGF of the standard normal distribution \(\mathcal{N}(0,1)\), whose density is \(\frac{1}{\sqrt{2\pi}} e^{-x^2/2}\) (see the definition of the normal distribution): \[ \begin{align*} M(t) &= \int_{-\infty}^{\infty} e^{tx} \frac{1}{\sqrt{2\pi}} e^{-x^2/2} \, dx \\\\ &= \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-(x^2 - 2tx)/2} \, dx \\\\ &= \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-[(x-t)^2 - t^2]/2} \, dx \\\\ &= e^{t^2/2} \underbrace{\int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}} e^{-(x-t)^2/2} \, dx}_{= 1} \\\\ &= e^{t^2/2}. \end{align*} \] The key step is completing the square in the exponent; the remaining integral equals \(1\) because it is the total probability of \(\mathcal{N}(t, 1)\), which (by the same definition) is a probability density integrating to \(1\) over \(\mathbb{R}\).