Introduction
On the previous page, we introduced completeness as a property ensuring that a metric space has "no holes." We stated
that complete spaces are those where every Cauchy sequence converges. But what exactly is convergence in a metric space?
And what is a Cauchy sequence?
Traditional calculus often presents these concepts through the lens of pointwise \(\varepsilon\)-\(N\) inequalities, a
"micro-arithmetic" approach that can sometimes obscure the underlying geometric truth. On this page, we shift our
perspective from individual point-to-point distances to structural topology. We define convergence and Cauchy properties
by looking at the behavior of the tail of a sequence: whether the infinite remainder of our process eventually "settles"
into an arbitrarily small neighborhood.
This shift is fundamental to understanding why optimization algorithms work. When gradient descent produces a
sequence of iterate \(\mathbf{x}_0, \mathbf{x}_1, \mathbf{x}_2, \ldots\), we need precise language to describe
how this sequence "concentrates" its energy toward a solution. By treating convergence as a structural event
rather than a mere numerical coincidence, we gain a clearer blueprint for analyzing stability and consistency in
high-dimensional ML-driven landscapes.
Convergence of Sequences
Here we want to generalize the familiar \(\varepsilon\)-\(\delta\) notion from elementary calculus to arbitrary metric
spaces. Why does this matter for modern ML? Because ML operates in spaces far richer than \(\mathbb{R}^n\): probability
distributions (Wasserstein space), functions (reproducing kernel Hilbert spaces), and even
neural network weights when the parameter space
is given a non-Euclidean geometry (for example, the one induced by the Fisher information).
Definition: Tails
Suppose \(X\) is a nonempty set and \(\{x_n\}\) is a sequence in \(X\). For each \(m \in \mathbb{N}\), the set
\[
\operatorname{tail}_m \{x_n\} = \{x_n \mid n \in \mathbb{N}, n \geq m\}
\]
is called the \(m\)th tail of the sequence \(\{x_n\}\).
The tail captures what happens after we discard finitely many initial terms.
In algorithm analysis, we care about asymptotic behavior: whether training eventually stabilizes,
not whether the first few epochs are noisy.
Definition: Convergence and Limit
Suppose \(X\) is a metric space, \(z \in X\) and \(\{x_n\}\) is a sequence in \(X\). We say that \(\{x_n\}\)
converges to \(z\) in \(X\), written \(x_n \to z\) if and only if every open subset of \(X\) that contains \(z\)
includes a tail of \(\{x_n\}\). Furthermore, \(z\) is called the limit of \(\{x_n\}\) denoted by \(\lim x_n\).
Insight: Convergence in ML Contexts
The abstract definition of convergence unifies diverse phenomena in machine learning, shifting the focus from individual
points to the behavior of systems in different spaces:
-
Parameter Convergence.
In optimization, we expect iterates \(\theta_t \to \theta^*\) under proper conditions. For SGD these combine the
Robbins-Monro step-size conditions with assumptions on the objective and the gradient noise, and the convergence
then holds almost surely, that is, along almost every realization of the random iterates. Along such a
realization, every \(\varepsilon\)-ball around the optimum \(\theta^*\) is eventually entered and
never left.
Without these conditions, the stochastic noise might prevent the sequence from settling.
-
Distributional Convergence.
In Generative Models (like GANs or Diffusion Models), we aim for the generated distribution \(p_G\) to approach
the data distribution \(p_{\text{data}}\). This convergence is measured in probability spaces using
"statistical" distances (for example,
KL divergence) or metrics (for example, the
Wasserstein distance). (Note that KL divergence is not metric.)
-
Function Space Convergence.
The Universal Approximation Theorem states that, for a suitable activation (for example, a continuous
non-polynomial one), neural networks are dense in the space of continuous functions on a compact set
\(K \subseteq \mathbb{R}^n\) under the \(L^{\infty}\) (uniform) norm. As we increase the model capacity, the
approximation can therefore converge uniformly on \(K\) to any continuous target function \(f\). Since
continuous functions are in turn dense in \(L^2(K)\), it can also converge in the \(L^2\) norm to
any square-integrable target.
Convergence in a metric space can be characterized through multiple equivalent lenses:
Theorem: Criteria for Convergence
Suppose \(X\) is a metric space, \(z \in X\) and \(\{x_n\}\) is a sequence in \(X\). The following statements
are logically equivalent:
-
\[
\bigcap \left\{\overline{\{x_n \mid n \in S\}} \mid S \subseteq \mathbb{N}, S \text{ infinite}\right\} = \{z\}.
\]
-
\[
z \in \bigcap \left\{\overline{\{x_n \mid n \in S\}} \mid S \subseteq \mathbb{N}, S \text{ infinite}\right\}.
\]
-
\[
\operatorname{dist}(z, \{x_n \mid n \in S\}) = 0
\]
for every infinite subset \(S\) of \(\mathbb{N}\).
- Every open ball centered at \(z\) includes a tail of \(\{x_n\}\).
- Every open subset of \(X\) that contains \(z\) includes a tail of \(\{x_n\}\).
Proof. See the proof sketch below.
Proof (sketch):
We sketch the operationally important equivalences (3) \(\Leftrightarrow\) (4) \(\Leftrightarrow\) (5). The
equivalence (1) \(\Leftrightarrow\) (2) \(\Leftrightarrow\) (3) passes through the distance-characterization of
closure.
(5) \(\Rightarrow\) (4). An open ball \(\mathcal{B}[z; r)\) is itself an open subset of \(X\) that
contains \(z\). Applying (5) yields that it includes a tail.
(4) \(\Rightarrow\) (5). Let \(U\) be an open subset of \(X\) containing \(z\). Since
open balls form a basis for the metric topology,
there exists \(r \gt 0\) with \(\mathcal{B}[z; r) \subseteq U\). By (4), some tail lies in \(\mathcal{B}[z; r)\),
hence in \(U\).
(4) \(\Rightarrow\) (3). Let \(S \subseteq \mathbb{N}\) be infinite and \(\varepsilon \gt 0\). By
(4) applied with radius \(\varepsilon\), some tail \(\{x_n \mid n \geq N\}\) lies in
\(\mathcal{B}[z; \varepsilon)\). Since \(S\) is infinite, it contains some \(n \geq N\), giving
\(d(x_n, z) \lt \varepsilon\). As \(\varepsilon\) was arbitrary,
\(\operatorname{dist}(z, \{x_n \mid n \in S\}) = 0\).
(3) \(\Rightarrow\) (4). Suppose (4) fails, so that some \(r \gt 0\) admits no tail inside
\(\mathcal{B}[z; r)\). Then for every \(N\), some \(n \geq N\) satisfies \(d(x_n, z) \geq r\), and the set \(S\) of
such indices is infinite. For every \(n \in S\), \(d(x_n, z) \geq r\), hence
\(\operatorname{dist}(z, \{x_n \mid n \in S\}) \geq r \gt 0\), contradicting (3).
Criterion 4 is the familiar \(\varepsilon\)-ball formulation. For every \(\varepsilon \gt 0\), there exists \(N\) such
that \(d(x_n, z) \lt \varepsilon\) for all \(n \geq N\). This is the condition that a convergence tolerance in a
numerical algorithm is meant to certify, although the limit \(z\) is usually not available to check against.
Criterion 5 reveals the topological perspective. Beyond numerical distances,
\(z\) is the limit because the sequence eventually enters and never leaves any open
"neighborhood" of \(z\). This formulation generalizes to spaces where we may not have an explicit
metric but do have a notion of "open sets" (general topological spaces).
Remember, convergent sequences of real numbers converge to "at most" one point. This is because \(\mathbb{R}\) is a
metric space, and uniqueness of limits holds in any metric space. This uniqueness of limits guarantees that a convergent
sequence of iterates cannot settle on two different candidates. It does not prevent iterates that fail to converge from
oscillating between candidates indefinitely.
Theorem: The Uniqueness of Limits
Suppose \(X\) is a metric space, and \(\{x_n\}\) is a sequence in \(X\) that converges in \(X\).
Then \(\{x_n\}\) converges to exactly one point in \(X\).
Proof:
Suppose for contradiction that \(\{x_n\}\) converges to both \(w\) and \(z\) with \(w \neq z\). Set
\(r := d(w, z)/2 \gt 0\). By the ball form of convergence (Criterion 4), there exist \(N_w, N_z \in \mathbb{N}\)
such that
\[
x_n \in \mathcal{B}[w; r) \quad \text{for all } n \geq N_w, \quad
x_n \in \mathcal{B}[z; r) \quad \text{for all } n \geq N_z.
\]
Let \(N := \max\{N_w, N_z\}\). Then \(x_N\) lies in both balls, so
\[
d(w, z) \leq d(w, x_N) + d(x_N, z) \lt r + r = d(w, z),
\]
a contradiction. Therefore \(w = z\).
In the real world, it is usually impossible to observe "everything." So we are more interested in "portions" of the
whole sequence. Fortunately, every subsequence of a convergent sequence converges to the same limit as the parent
sequence, which is because every tail of the parent sequence includes a tail of any of its subsequences.
Theorem: Convergence of Subsequences
Suppose \(\{x_n\}\) is a sequence in a metric space \(X\). If \(\{x_n\}\) converges to \(z \in X\),
then every subsequence \(\{x_{m_n}\}\) of \(\{x_n\}\) also converges to \(z\).
Side Note. In this case, \(z\) is in the closure of every tail of \(\{x_n\}\). Or, equivalently,
every ball centered at \(z\) contains an infinite number of terms of \(\{x_n\}\).
This theorem states that convergence is invariant under any infinite sub-sampling.
Whether we monitor every single iterate or only record the state every 100 epochs, the identified limit
remains the same.
Insight: Consistency Across Observations
In practical machine learning, we rarely inspect every single update of a high-frequency optimizer.
Instead, we "sample" the model's performance at specific intervals.
-
Robustness to Checkpointing.
Provided the full sequence of iterates converges, this property guarantees that the "limit" we observe through
periodic checkpoints is identical to the limit of the training process.
-
Avoiding Inconsistency.
If a sequence were to have subsequences converging to different limits, our evaluation results would depend
entirely on when we measured the performance, which is a nightmare for reproducibility. Convergence
prevents this "mode-switching" instability.
Convergence in a metric subspace is not exactly the same notion as convergence of subsequences in a metric space.
Theorem: Convergence in Subspaces
Let \(X\) be a metric space, and \(Y\) be a metric subspace of \(X\). Suppose \(\{x_n\}\) is a sequence in \(Y\)
that converges to \(w \in X\). Then \(\{x_n\}\) converges in \(Y\) if and only if \(w \in Y\), and in that case, its
limit in \(Y\) is \(w\).
Insight: Why Subspaces Matter in ML
We often constrain parameters to a subset \(Y\) (for example, via weight clipping). The critical question is whether
the limit \(L\) stays within our allowed region \(Y\).
-
Closed Subsets. If our constraint set \(Y\) is closed in \(\mathbb{R}^n\), any
sequence in \(Y\) that converges in the larger space \(\mathbb{R}^n\) is guaranteed to have its limit in \(Y\).
-
Numerical Instability. The floating-point numbers form a finite subset of \(\mathbb{R}\). A
sequence of real numbers can converge to a limit, such as \(\sqrt{2}\), that is not a floating-point number, so
computed iterates can only approximate it to within rounding error, and these errors can accumulate
into numerical instability.
Cauchy Sequences
How do we know a sequence "should" converge without knowing the limit? In practice, this is exactly the situation we
face. Gradient descent produces iterates, and we need a stopping criterion that does not require oracle knowledge of the
optimal solution.
The key insight is to measure whether the sequence elements are getting closer to each other,
rather than closer to some unknown target.
Definition: Cauchy Sequence
Suppose \(X\) is a metric space and \(\{x_n\}\) is a sequence in \(X\). We say that \(\{x_n\}\) is a
Cauchy sequence in \(X\) if and only if for every \(r \in \mathbb{R}^+\), there is
a ball of \(X\) of radius \(r\) that includes a tail of \(\{x_n\}\).
Theorem: Convergent Sequences are Cauchy
Let \((X, d)\) be a metric space. If a sequence \(\{x_n\}\) converges in \(X\), then
\(\{x_n\}\) is a Cauchy sequence.
The converse is not always true. Recall from the previous page the sequence of rationals \(x_1 = 1\),
\(x_{n+1} = \tfrac{1}{2}(x_n + 2/x_n)\). It is Cauchy in \(\mathbb{Q}\) but converges to
\(\sqrt{2} \notin \mathbb{Q}\). Whether every Cauchy sequence converges in \(X\) is precisely the question of
completeness.
Proof:
Suppose \(x_n \to z\) and let \(r \in \mathbb{R}^+\). Applying the ball form of convergence
(Criterion 4) with radius \(r\), there exists \(N \in \mathbb{N}\) such that
\(x_n \in \mathcal{B}[z; r)\) for all \(n \geq N\). Thus the tail \(\{x_n \mid n \geq N\}\) is
contained in \(\mathcal{B}[z; r)\), a ball of \(X\) of radius \(r\). Since \(r\) was arbitrary,
\(\{x_n\}\) is Cauchy.
Proof:
(Ball-tail \(\Rightarrow\) \(\varepsilon\)-\(N\)). Given \(\varepsilon \gt 0\), apply the ball-tail
condition with \(r = \varepsilon/2\) to obtain \(c \in X\) and \(N \in \mathbb{N}\) such that
\(x_n \in \mathcal{B}[c; \varepsilon/2)\) for all \(n \geq N\). For any \(m, n \geq N\), the triangle inequality
gives
\[
d(x_m, x_n) \leq d(x_m, c) + d(c, x_n) \lt \tfrac{\varepsilon}{2} + \tfrac{\varepsilon}{2} = \varepsilon.
\]
(\(\varepsilon\)-\(N\) \(\Rightarrow\) Ball-tail). Given \(r \gt 0\), apply the
\(\varepsilon\)-\(N\) form with \(\varepsilon = r\) to obtain \(N \in \mathbb{N}\) such that \(d(x_m, x_n) \lt r\)
for all \(m, n \geq N\). In particular, setting \(m = N\) yields \(d(x_N, x_n) \lt r\) for all \(n \geq N\), so the
tail \(\{x_n \mid n \geq N\}\) is contained in \(\mathcal{B}[x_N; r)\), a ball of \(X\) of radius \(r\).
Insight: The Practical Dilemma
In optimization, the target point \(x^*\) is unknown, so detecting convergence by checking
\(d(x_n, x^*) \lt \varepsilon\) is a circular problem. The Cauchy criterion provides a rigorous
theoretical escape. In a complete space such as \(\mathbb{R}^n\), it ensures convergence by examining whether all
future terms in a sequence remain arbitrarily close to one another, independent of the limit itself.
However, a gap exists between theory and implementation. Verifying the formal Cauchy condition requires monitoring
an infinite tail of the sequence, which is computationally impossible. In practice, we use stagnation-based
stopping criteria (such as \(|\theta_{t+1} - \theta_t| \lt \varepsilon\) over a fixed window). While these
are inspired by Cauchy's logic, they are heuristic approximations. They detect when an algorithm
has slowed down, but do not mathematically guarantee that it has reached a global optimum.
Boundedness
In pure mathematics, boundedness is a property that a sequence possesses as a
necessary consequence of convergence. In machine learning, we invert this logic. We enforce boundedness
as a mechanism to maintain numerical stability and prevent divergence.
Definition: Bounded
A subset \(S\) of a metric space \(X\) is called a bounded subset of \(X\) if and only if
\(S = \emptyset\) or \(S\) is included in some ball of \(X\). A metric space \(X\) is said to be
bounded if and only if it is a bounded subset of itself.
Definition: Diameter
Suppose \((X, d)\) is a metric space and \(S\) is a subset of \(X\). The diameter of \(S\) is defined
as
\[
\operatorname{diam}(S) = \sup \{d(r, s) \mid r, s \in S\}.
\]
Theorem: Diameter of the Closure
Suppose \((X, d)\) is a metric space and \(S \subseteq X\). Then the
closure of \(S\) has the same diameter as \(S\):
\[
\operatorname{diam}(\overline{S}) = \operatorname{diam}(S).
\]
Proof:
Since \(\overline{S} = S \cup \partial S \supseteq S\) and the diameter is a supremum taken over pairs of points,
enlarging the set cannot decrease that supremum. Hence
\(\operatorname{diam}(S) \leq \operatorname{diam}(\overline{S})\).
For the reverse inequality, fix \(r \gt 0\) and take any \(a, b \in \overline{S}\). We first note that each point of
\(\overline{S}\) is approximated arbitrarily well by points of \(S\). If \(a \in S\) this is immediate, and if
\(a \in \partial S\) then the definition of a
boundary point gives
\(\operatorname{dist}(a, S) = 0\), so some \(x \in S\) satisfies \(d(a, x) \lt r/2\). Choosing such \(x, y \in S\) for
\(a, b\) respectively, the triangle inequality yields
\[
d(a, b) \leq d(a, x) + d(x, y) + d(y, b) \lt \frac{r}{2} + \operatorname{diam}(S) + \frac{r}{2} = \operatorname{diam}(S) + r,
\]
where \(d(x, y) \leq \operatorname{diam}(S)\) because \(x, y \in S\). Since \(a, b\) were arbitrary in \(\overline{S}\),
we obtain \(\operatorname{diam}(\overline{S}) \leq \operatorname{diam}(S) + r\). Since \(r \gt 0\) was arbitrary,
\(\operatorname{diam}(\overline{S}) \leq \operatorname{diam}(S)\). The two inequalities give the stated equality.
Theorem: Criteria for Boundedness
Suppose \(X\) is a nonempty metric space, \(z \in X\), and \(S \subseteq X\). The following statements
are logically equivalent:
- \(\operatorname{diam}(S) \lt \infty\).
- There is a ball of \(X\) centered at \(z\) that includes \(S\).
- There is a ball of \(X\) that includes \(S\).
Theorem: Cauchy Sequences are Bounded
Suppose \(X\) is a metric space and \(\{x_n\}\) is a sequence in \(X\). If \(\{x_n\}\) is Cauchy, then
\(\{x_n\}\) is bounded in \(X\). Since every convergent sequence is Cauchy, it follows in particular
that every convergent sequence in \(X\) is bounded.
Proof:
By the Cauchy condition applied with \(r = 1\), some ball \(\mathcal{B}[c; 1)\) of \(X\) includes a tail
\(\{x_n \mid n \geq N\}\) of \(\{x_n\}\), for some \(c \in X\) and \(N \in \mathbb{N}\). In particular,
\(d(x_n, c) \lt 1\) for all \(n \geq N\).
Set
\[
R := 1 + \max\{d(x_1, c), d(x_2, c), \ldots, d(x_N, c)\}.
\]
The maximum is over a finite, nonempty set, so \(R\) is a well-defined positive real number.
For each \(i \in \{1, \ldots, N\}\), we have \(d(x_i, c) \leq R - 1 \lt R\), so \(x_i \in \mathcal{B}[c; R)\). For each
\(n \gt N\), we have \(d(x_n, c) \lt 1 \leq R\), so \(x_n \in \mathcal{B}[c; R)\).
Therefore the entire sequence \(\{x_n\}\) is included in the ball \(\mathcal{B}[c; R)\), so \(\{x_n\}\) is bounded.
Important. The converse is false. Boundedness alone does not guarantee convergence.
Consider \(x_n = (-1)^n\) in \(\mathbb{R}\). The sequence is bounded (contained in \([-1, 1]\)) but oscillates forever
without converging.
Insight
While enforcing a bound does not mathematically guarantee convergence (as the sequence could still oscillate), a
bound placed on the iterates themselves keeps them within a bounded region. In the
finite-dimensional parameter spaces typical of machine learning, such a region, when combined with appropriate
closure, forms a compact set. This "structural containment" is a prerequisite for the Cauchy
property to even be possible in practice.
-
Gradient Clipping.
By enforcing \(\|\nabla L\| \leq M\), we bound the length of each update step. This is a heuristic
countermeasure against the exploding gradient problem, preventing a single step from throwing
the iterates arbitrarily far, although it does not by itself keep the iterates in a bounded region.
-
Weight Clipping (for example, WGAN).
Forcing weights into a compact interval \([-c, c]\) ensures Lipschitz continuity. This
mathematical constraint is required to bound the slope of the function, stabilizing the dual form of the
Wasserstein distance.
-
Regularization (\(L^2\)).
Adding a penalty \(\lambda\|\theta\|^2\) with \(\lambda \gt 0\) to a loss that is bounded below makes the
objective function coercive. It ensures that the set of parameters with low objective value
remains bounded, effectively "trapping" the optimization process in a region where, for a continuous loss, a
minimum is guaranteed to exist.
-
Trust Regions.
By restricting each step to a ball of radius \(\Delta\), we keep the step where the
Taylor approximation (which is only locally valid) is trusted as a model of the loss surface,
shrinking \(\Delta\) whenever the model predicts the actual decrease poorly.
While boundedness ensures that our sequence does not escape to infinity, it does not guarantee that the set is
structurally "solid" enough to contain its own limit points. To bridge this gap, we consider a property that connects
distance to existence in the most fundamental way.
Definition: Nearest-Point Property
Suppose \(X\) is a metric space. We say that \(X\) has the nearest-point property
if and only if \(X = \emptyset\) or \(X\) admits a nearest point to each point in every metric superspace of
\(X\).
Intuitively, this property ensures that the space has no "holes" for bounded sequences to fall into. From any vantage
point in a larger ambient space, there is always a closest point actually lying in \(X\).
Concrete examples help anchor this abstract definition:
-
Compact metric spaces have the nearest-point property. The argument is short. For any superspace
\(Y\) and point \(y \in Y\), the function \(f(x) = d(y, x)\) is continuous on \(X\) (in fact \(1\)-Lipschitz, by the
reverse triangle inequality), and a continuous real-valued function on a compact metric space attains its infimum.
In \(\mathbb{R}^n\), this covers all closed and bounded sets via Heine-Borel.
-
Open intervals such as \((0, 1) \subset \mathbb{R}\) do not have the nearest-point
property. The ambient point \(0 \in \mathbb{R}\) has no nearest point inside \((0, 1)\), since the distances
\(d(0, y) = y\) approach but never attain zero as \(y\) ranges over the interval. This is the same phenomenon
flagged in the discussion of nearest points on the previous page.
A word of caution is needed for Hilbert spaces. The celebrated
Hilbert projection theorem guarantees a unique nearest point in a nonempty closed convex subset
\(C \subseteq \mathcal{H}\), but only for ambient points \(z \in \mathcal{H}\), not for points in an arbitrary
superspace of \(C\). Thus closed convex subsets of a Hilbert space inherit a restricted version of the
nearest-point property within \(\mathcal{H}\), which is what drives orthogonal projection, least-squares regression, and
feasibility projections in convex optimization. The unrestricted nearest-point property (over every superspace) asks for
more, namely that every bounded sequence in the set have a subsequence converging to a point of the set. Closed subsets
of \(\mathbb{R}^n\) meet this demand, while the closed unit ball of an infinite-dimensional Hilbert space does not.
The precise connection to bounded sequences and convergent subsequences, and ultimately to compactness,
will be made rigorous on a later page.
Insight: Structural Guarantee in Optimization
In optimization, simply knowing that our search space is limited (bounded) is only half the battle. The
nearest-point property ensures that even if we cannot reach an "ideal" point outside our set, we
can always find a best approximation (a projection) within it.
For instance, in Constrained SGD or SVM, we rely on this structural "solidness" so
that every projection step has a nearest point to land on. We will explore how this combines with boundedness to
form the broader concept of compactness on a later page.
As we move toward more complex spaces, especially when dealing with infinite-dimensional spaces,
we need a refined version of boundedness that accounts for the "density" of the set, leading us to
the concept of total boundedness.
Definition: Total Boundedness
A subset \(S\) of a metric space \(X\) is called a totally bounded subset of \(X\)
if and only if for each \(r \in \mathbb{R}^+\), there is a finite collection of balls of \(X\) of
radius \(r\) that covers \(S\). The metric space \(X\) is said to be totally bounded
if and only if it is a totally bounded subset of itself.
Insight: Total Boundedness in Function Spaces
In the space of neural network functions, total boundedness directly relates to covering numbers,
the minimum number of balls of radius \(\varepsilon\) needed to cover a function class.
-
Capacity Control.
In statistical learning theory, the logarithm of the covering number (metric entropy) measures the complexity of
a model.
-
Generalization.
A totally bounded function class ensures that we can approximate any function in the class with a finite set of
"representative" networks. Smaller covering numbers imply a less complex hypothesis space, which typically leads
to tighter generalization bounds.
Completeness Revisited
On the previous page, we defined a metric space \(X\) to be
complete if and only if \(X\)
is closed in every metric superspace of \(X\). This formulation prioritizes the structural integrity of the metric space
itself. The space has "no holes" that could be exposed by embedding it in a larger space.
Now that we have developed the Cauchy criterion, we can state an equivalent characterization that is often more useful
in practice. A metric space \(X\) is
complete if and only if every Cauchy sequence in \(X\) converges in \(X\).
The proof of this equivalence (superspace-closed \(\Leftrightarrow\) every Cauchy sequence converges) is developed on
the page devoted to completeness. For now, the sequential characterization gives us an operational test. To verify
completeness, we must show that any sequence whose terms get arbitrarily close together eventually reaches a limit
within the space.
This equivalence carries a deeper structural message. The original definition of completeness quantifies over
every metric superspace \(Y\), an unwieldy and ambient-dependent condition. The Cauchy criterion, in
contrast, is purely intrinsic. Being Cauchy and converging are both conditions on sequences within \(X\)
itself, referring only to the metric \(d\) on \(X\) and making no appeal to any ambient space. The theorem above shows
these two viewpoints coincide. This is precisely why the Cauchy formulation is the workhorse in practice. We never need
to imagine \(X\) embedded in something larger, only to examine its own sequences.
Example:
\(\mathbb{R}^n\) (with the Euclidean metric) is complete. This follows from the completeness of \(\mathbb{R}\) itself by
a coordinate-wise argument. If \(\{x_k\}\) is Cauchy in \(\mathbb{R}^n\), then each coordinate sequence
\(\{x_k^{(i)}\}\) is Cauchy in \(\mathbb{R}\) (since \(|x_k^{(i)} - x_\ell^{(i)}| \leq \|x_k - x_\ell\|_2\)), hence
converges to some \(z^{(i)} \in \mathbb{R}\). The vector \(z = (z^{(1)}, \ldots, z^{(n)})\) is then the limit of
\(\{x_k\}\) in \(\mathbb{R}^n\). In contrast, \(\mathbb{Q}^n\) (rationals) is not complete, since Cauchy sequences can
converge to irrational limits, as in the example on the previous page.
Insight
Completeness is the "floor" of our mathematical world. It ensures that if an algorithm suggests a limit exists (via
the Cauchy property), that limit is a valid point within our space. In machine learning, we rely on the following
complete spaces:
-
\(\mathbb{R}^d\) (Parameter Space).
As a finite-dimensional Euclidean space, \(\mathbb{R}^d\) is complete. While this does not guarantee that SGD
will converge (which depends on learning rates and loss geometry), it ensures that any Cauchy sequence
of weights \(\{\theta_t\}\) has a unique, reachable limit \(\theta^* \in \mathbb{R}^d\).
-
Hilbert Spaces (for example,
RKHS).
By definition, a Hilbert space is a complete inner product space. The convergence of kernel-based algorithms
depends on the fact that the optimization occurs in a space where "holes" do not exist.
-
\(L^p\) Spaces (Function
Spaces).
The Riesz-Fischer Theorem
proves that \(L^p\) spaces are complete. In particular, they are
Banach spaces. This matters
for approximation by neural networks. It guarantees that any sequence of networks in \(L^p\) that is Cauchy in
the \(L^p\) norm converges to some limit function in \(L^p\).