Measure-Theoretic Probability

From Two Integrals to One

In our early study of probability, we were forced to treat discrete and continuous random variables as two separate worlds, one ruled by summation \(\sum\) and the other by integration \(\int\). In Random Variables, we defined a function \(X: S \to \mathbb{R}\) that attaches a number to each outcome. Here \(S\) was the sample space, which we now rewrite as \(\Omega\) to conform with measure-theoretic notation. We computed expectations by summation for discrete variables and by integration against a density for continuous ones.

But nature does not divide data into such neat categories. A mixed distribution, partly discrete and partly continuous, cannot be handled cleanly by either formula alone. The discrete-continuous dichotomy is not a feature of probability. It is an artifact of having two separate integration theories. We need a single framework in which all random variables and all expectations are treated uniformly.

The tools to build that framework already exist. In Measure Theory, we constructed the \(\sigma\)-algebra and probability measure as the mathematical substrate of randomness. In Lebesgue Integration, we built an integral powerful enough to handle highly irregular functions on arbitrary measure spaces. In \(L^p\) Spaces, we turned the resulting function spaces into normed spaces.

This page brings these tools to bear on probability. We do not repeat the definitions of \(\sigma\)-algebras, measures, or the Lebesgue integral. Those foundations are in place. Instead, we translate. Each concept from classical probability receives its measure-theoretic formulation, and each measure-theoretic theorem receives its probabilistic name. The result is a unified framework in which discrete sums, continuous integrals, and mixed distributions become instances of one operation: integration against a measure.

Random Variables as Measurable Functions

We defined a random variable as a function that assigns a numerical value to each outcome in the sample space. This definition is correct in spirit but incomplete in one critical respect. It imposes no condition on which functions qualify. In the measure-theoretic framework, not every function from \(\Omega\) to \(\mathbb{R}\) deserves to be called a random variable. The ones that qualify are those compatible with the \(\sigma\)-algebra structure that determines which events can be assigned probabilities.

The Measurability Condition

Let \((\Omega, \mathcal{F}, \mathbb{P})\) be a probability space, and let \((\mathbb{R}, \mathcal{B}(\mathbb{R}))\) be the real line equipped with its Borel \(\sigma\)-algebra, the \(\sigma\)-algebra generated by all open intervals.

The condition asks that for every "reasonable" subset \(B\) of the real line (every Borel set), the set of outcomes \(\omega\) for which \(X(\omega)\) lands in \(B\) be an event. An event is a member of \(\mathcal{F}\), the only kind of set to which \(\mathbb{P}\) assigns a probability. Without this condition, the expression \(\mathbb{P}(X \in B)\) might be undefined. We would be asking for the probability of a set that our \(\sigma\)-algebra does not recognize.

In practice, measurability is verified via a useful shortcut. Since \(\mathcal{B}(\mathbb{R})\) is generated by half-lines \((-\infty, a]\), it suffices to check preimages of these generators:

The half-line criterion connects directly to the familiar CDF. The measurability condition is precisely the requirement that the cumulative distribution function \(F_X(a) = \mathbb{P}(X \leq a)\) is well-defined for every \(a \in \mathbb{R}\).

The Distribution of a Random Variable

Given a random variable \(X\), we constantly ask questions of the form "\(\mathbb{P}(X \in B)\) for various Borel sets \(B\)." This collection of probabilities is itself a measure on \((\mathbb{R}, \mathcal{B}(\mathbb{R}))\), and it encodes everything we need to know about \(X\) from a probabilistic standpoint.

The intuition is geometric. The function \(X\) transports the probability mass from the abstract space \(\Omega\) onto the real line \(\mathbb{R}\). Whatever probability \(\mathbb{P}\) assigns to a region of \(\Omega\), the pushforward \(P_X\) assigns the same probability to the corresponding region of \(\mathbb{R}\) that \(X\) maps it into. The distribution \(P_X\) is simply the result of "weighing" the real line according to how \(X\) redistributes the original probability mass.

That \(P_X\) is indeed a probability measure follows from the properties of \(\mathbb{P}\). First, \(P_X(\mathbb{R}) = \mathbb{P}(X^{-1}(\mathbb{R})) = \mathbb{P}(\Omega) = 1\). For countable additivity, let \(B_1, B_2, \ldots \in \mathcal{B}(\mathbb{R})\) be disjoint. Since preimages commute with countable unions and \(X^{-1}(B_j) \cap X^{-1}(B_k) = X^{-1}(B_j \cap B_k) = \emptyset\) for \(j \neq k\), the preimages \(\{X^{-1}(B_k)\}\) are disjoint measurable sets, and \[ \begin{align*} P_X\Bigl(\bigsqcup_k B_k\Bigr) &= \mathbb{P}\Bigl(\bigsqcup_k X^{-1}(B_k)\Bigr) \\\\ &= \sum_k \mathbb{P}(X^{-1}(B_k)) \\\\ &= \sum_k P_X(B_k), \end{align*} \] using countable additivity of \(\mathbb{P}\).

The pushforward perspective reveals that all the information about \(X\) that matters probabilistically is encoded in the single measure \(P_X\) on \(\mathbb{R}\). Its CDF, its moments, and its tail behavior are all determined by \(P_X\). Two random variables defined on entirely different probability spaces can have the same distribution, and from the perspective of any question answerable by \(P_X\) alone, they are indistinguishable.

Recovering PMFs and PDFs

The pushforward framework unifies the discrete and continuous cases that were treated separately in Random Variables.

Discrete case. If \(X\) takes values in a countable set \(\{x_1, x_2, \ldots\}\), then \(P_X\) is concentrated on these points: \[ \begin{align*} P_X(B) &= \sum_{k:\, x_k \in B} \mathbb{P}(X = x_k) \\\\ &= \sum_{k:\, x_k \in B} p(x_k). \end{align*} \] Here \(p(x_k) = \mathbb{P}(X = x_k)\) is the probability mass function. Formally, \(P_X\) is the weighted sum of point masses: \(P_X = \sum_k p(x_k) \, \delta_{x_k}\), where \(\delta_{x_k}(B) = 1\) if \(x_k \in B\) and \(0\) otherwise.

The measure \(P_X\) is singular with respect to the Lebesgue measure \(\lambda\). It is supported on a countable set, which has \(\lambda\)-measure zero.

Continuous case. If \(P_X\) is absolutely continuous with respect to \(\lambda\), meaning that \(\lambda(B) = 0\) implies \(P_X(B) = 0\), then there exists a nonnegative measurable function \(f_X\) such that \[ P_X(B) = \int_B f_X(x) \, d\lambda(x) \quad \text{for every } B \in \mathcal{B}(\mathbb{R}). \]

The function \(f_X\) is the probability density function (PDF), and the relationship is written compactly as \[ f_X = \frac{dP_X}{d\lambda}, \] the Radon-Nikodym derivative of \(P_X\) with respect to \(\lambda\).

The fractional notation is not merely symbolic. On a small interval \([x, x + dx]\), when \(f_X\) is continuous at \(x\), it encodes the approximation \(P_X([x, x+dx]) \approx f_X(x) \cdot \lambda([x, x+dx])\), that is, "probability \(\approx\) density \(\times\) length." The density \(f_X(x)\) measures the local rate at which \(P_X\) accumulates probability relative to the rate at which Lebesgue measure \(\lambda\) accumulates length.

The Radon-Nikodym theorem guarantees the existence of \(f_X\) whenever \(P_X \ll \lambda\) (absolute continuity) and the dominating measure \(\lambda\) is \(\sigma\)-finite, a condition satisfied here, since \(\lambda\) is \(\sigma\)-finite on \(\mathbb{R}\). The proof is developed on a later page. For now we use the result freely. This perspective elevates the PDF from "the derivative of the CDF" to a precise statement about how one measure relates to another.

Random Vectors and Measurability

Everything above extends to vector-valued random variables. A random vector \(\mathbf{X} = (X_1, \ldots, X_d) : \Omega \to \mathbb{R}^d\) is measurable with respect to \((\mathcal{F}, \mathcal{B}(\mathbb{R}^d))\), where \(\mathcal{B}(\mathbb{R}^d)\) is the Borel \(\sigma\)-algebra on \(\mathbb{R}^d\). A convenient fact simplifies verification:

The proof relies on the identity \(\mathcal{B}(\mathbb{R}^d) = \mathcal{B}(\mathbb{R}) \otimes \cdots \otimes \mathcal{B}(\mathbb{R})\), which says that the Borel \(\sigma\)-algebra of the product space coincides with the product of the Borel \(\sigma\)-algebras. The inclusion \(\supseteq\) is immediate. Each measurable rectangle \(B_1 \times \cdots \times B_d\) with \(B_i \in \mathcal{B}(\mathbb{R})\) is a Borel set in \(\mathbb{R}^d\). The reverse inclusion uses the second-countability of \(\mathbb{R}^d\) (a countable base of open sets given by rectangles with rational corners), so every open set is a countable union of measurable rectangles and hence lies in \(\mathcal{B}(\mathbb{R}) \otimes \cdots \otimes \mathcal{B}(\mathbb{R})\).

The product \(\sigma\)-algebra construction itself is developed on the next page. That \(\sigma\)-algebra is generated by measurable rectangles \(B_1 \times \cdots \times B_d\) with each \(B_i \in \mathcal{B}(\mathbb{R})\). Since \(\mathbf{X}^{-1}(B_1 \times \cdots \times B_d) = \bigcap_{i=1}^d X_i^{-1}(B_i)\), component-wise measurability ensures that all preimages of generators belong to \(\mathcal{F}\), which suffices because the sets whose preimage lies in \(\mathcal{F}\) form a \(\sigma\)-algebra, exactly as in the half-line proposition. Conversely, each coordinate projection \(\pi_i : \mathbb{R}^d \to \mathbb{R}\) is continuous, so preimages of open sets under \(\pi_i\) are open and hence Borel, and the same reasoning shows that \(\pi_i\) is Borel-measurable. If \(\mathbf{X}\) is measurable, then \(X_i^{-1}(B) = \mathbf{X}^{-1}(\pi_i^{-1}(B)) \in \mathcal{F}\) for every \(B \in \mathcal{B}(\mathbb{R})\), so each \(X_i = \pi_i \circ \mathbf{X}\) is measurable.

The distribution of a random vector is the pushforward \(P_{\mathbf{X}} = \mathbb{P} \circ \mathbf{X}^{-1}\) on \((\mathbb{R}^d, \mathcal{B}(\mathbb{R}^d))\). When \(P_{\mathbf{X}}\) has a density \(f_{\mathbf{X}}\) with respect to the Lebesgue measure on \(\mathbb{R}^d\), we recover exactly the joint density studied in Multivariate Distributions. For instance, the multivariate normal density with mean \(\boldsymbol{\mu}\) and covariance \(\boldsymbol{\Sigma}\) is the Radon-Nikodym derivative \(f_{\mathbf{X}} = dP_{\mathbf{X}}/d\lambda^d\).

Expectation as Lebesgue Integral

In Random Variables, we defined the expected value of \(X\) using two separate formulas: a sum \(\sum x \, p(x)\) for discrete variables and an integral \(\int x \, f(x) \, dx\) for continuous ones. This bifurcation is awkward. Every theorem about expectations had to be stated (or proved) twice, once for each case. The Lebesgue integral eliminates this entirely.

The Unified Definition

When the underlying measure space is a probability space \((\Omega, \mathcal{F}, \mathbb{P})\), the Lebesgue integral of a random variable \(X\) is written \[ \mathbb{E}[X] = \int_\Omega X(\omega) \, d\mathbb{P}(\omega). \] This single expression replaces both the discrete sum and the continuous integral.

The construction of the Lebesgue integral, from indicator functions through simple functions to general measurable functions, is already in place. What we do here is not repeat that construction but show how it absorbs the two classical formulas as special cases.

Recovering the Classical Formulas

The bridge between integration over the abstract space \(\Omega\) and integration over \(\mathbb{R}\) (where we actually compute) is the change-of-variable formula for pushforward measures.

The first equality is the definition of \(\mathbb{E}[g(X)]\), and the second is the content of the theorem. The theorem says that to compute the expectation of \(g(X)\), we do not need access to the abstract space \(\Omega\). We can integrate \(g\) directly against the distribution \(P_X\) on \(\mathbb{R}\). This is precisely what we always did in practice. No one computes expectations by integrating over "all outcomes of the experiment." We integrate over the range of the random variable, weighted by its distribution.

From here, the two classical formulas are immediate:

Discrete case. If \(P_X = \sum_k p(x_k) \, \delta_{x_k}\) (a sum of point masses), then integration against \(P_X\) reduces to summation: \[ \begin{align*} \mathbb{E}[g(X)] &= \int_{\mathbb{R}} g(x) \, dP_X(x) \\\\ &= \sum_k g(x_k) \, p(x_k). \end{align*} \] Setting \(g(x) = x\) recovers the formula \(\mathbb{E}[X] = \sum_k x_k \, p(x_k)\).

Continuous case. If \(P_X\) has density \(f_X = dP_X/d\lambda\), then for an indicator \(g = \mathbf{1}_B\) the identity \(\int_{\mathbb{R}} g \, dP_X = \int_{\mathbb{R}} g \, f_X \, d\lambda\) is the definition of the Radon-Nikodym derivative, and it extends to general \(g\) by the same simple-function approximation as in the proof above: \[ \begin{align*} \mathbb{E}[g(X)] &= \int_{\mathbb{R}} g(x) \, dP_X(x) \\\\ &= \int_{\mathbb{R}} g(x) \, f_X(x) \, dx. \end{align*} \] Setting \(g(x) = x\) recovers the formula \(\mathbb{E}[X] = \int x \, f_X(x) \, dx\).

The "Law of the Unconscious Statistician," traditionally presented as a computational trick, is revealed as a theorem about the relationship between a measure and its pushforward. What seemed like two separate definitions were always a single integral \(\int g \, dP_X\), specialized to different types of measures.

Properties of Expectation: One Proof for All

The Lebesgue integral inherits linearity, monotonicity, and compatibility with limits from its construction. These properties now hold for all random variables simultaneously, with no case splitting:

Recall that in our first treatment, linearity of expectation was argued once for discrete and once for continuous variables. Here it is a single application of the integral's linearity, valid for discrete, continuous, and mixed distributions alike.

Moments and \(L^p\) Spaces

The measure-theoretic framework reveals a direct link between moments of a random variable and the \(L^p\) spaces studied in analysis.

The remark is not a new theorem but a direct restatement of the definition of \(L^p\) in the special case where the measure is a probability measure. The translation is nevertheless illuminating:

• \(X \in L^1(\mathbb{P})\): the mean \(\mathbb{E}[X]\) exists and is finite.
• \(X \in L^2(\mathbb{P})\): the variance \(\operatorname{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2\) is finite. The space \(L^2(\Omega, \mathbb{P})\) is a Hilbert space with inner product \(\langle X, Y \rangle = \mathbb{E}[XY]\).
• \(X \in L^4(\mathbb{P})\): the kurtosis (fourth standardized moment) is defined and finite whenever \(\operatorname{Var}(X) \gt 0\).

The \(L^p\) inclusion hierarchy \(L^q \subseteq L^p\) for \(q \geq p\) (valid on probability spaces, where \(\mathbb{P}(\Omega) = 1\)) translates to: finiteness of a higher moment implies finiteness of all lower moments. If \(X \in L^4(\mathbb{P})\), then automatically \(X \in L^2(\mathbb{P}) \subseteq L^1(\mathbb{P})\), so the mean and variance are guaranteed to be finite.

This inclusion is a privilege of finite measure spaces. On \((\mathbb{R}, \mathcal{B}(\mathbb{R}), \lambda)\), where \(\lambda(\mathbb{R}) = \infty\), the inclusion fails. Functions can be \(L^p\)-integrable without being \(L^q\)-integrable for \(q \gt p\), and vice versa. The proof above breaks down precisely because the factor \(\lambda(\mathbb{R})^{(q-p)/q}\) is infinite instead of \(1\). On a probability space, a "closed world" of total mass \(1\), a finite higher moment forces all lower moments to be finite.

"Almost Surely" Revisited

In Lebesgue integration, we introduced the phrase "almost everywhere" (a.e.) to mean "for all \(\omega\) outside a set of measure zero." On a probability space, the same concept is called "almost surely" (a.s.). An event holds a.s. if the set of outcomes where it fails has probability zero.

The equivalence is literal: \[ \begin{align*} X = Y \quad \text{a.s.} &\Longleftrightarrow \mathbb{P}\bigl(\{\omega : X(\omega) \neq Y(\omega)\}\bigr) = 0 \\\\ &\Longleftrightarrow X = Y \quad \text{a.e. with respect to } \mathbb{P}. \end{align*} \]

This equivalence is why \(L^p(\Omega, \mathbb{P})\) consists of equivalence classes of random variables that agree a.s. Two random variables that differ only on a null event are probabilistically identical, and we should not (and do not) distinguish them.

The "a.e. vs. a.s." translation may seem trivial, but it matters when reading across disciplines. An analyst writes "\(f_n \to f\) a.e." A probabilist writes "\(X_n \to X\) a.s." They mean the same thing. In Limit Theorems and Product Measures, we exploit this translation systematically. We give the great limit theorems of Lebesgue integration their probabilistic names, and we formalize the product measure construction that makes "i.i.d. sampling" mathematically precise.

Looking Ahead

We have completed the first half of the measure-theoretic translation: random variables are measurable functions, distributions are pushforward measures, and expectations are Lebesgue integrals. The discrete-continuous dichotomy that split our earlier treatment has been dissolved. PMFs, PDFs, and mixed distributions are all special cases of a single integral \(\int g \, dP_X\).

But the translation is not yet complete. Two fundamental questions remain open:

• When can we interchange limits and expectations? The Monotone Convergence Theorem, Fatou's Lemma, and the Dominated Convergence Theorem need their probabilistic interpretations. These are the tools that license, under stated regularity hypotheses, the operations of differentiating under the integral sign in stochastic gradient descent and of passing from almost sure convergence to convergence of expectations. Treated in the Convergence Theorems section.
• What does "independent" really mean? The condition \(\mathbb{P}(A \cap B) = \mathbb{P}(A)\mathbb{P}(B)\) must be elevated to independence of \(\sigma\)-algebras, and the product measure construction must be built to give "i.i.d. sampling" its mathematical content. Treated in the Independence and Product Measures section.

These two threads are taken up in Limit Theorems and Product Measures, which completes the core measure-theoretic foundation for probability.