Open Covers and Compactness
While completeness focuses on the
"internal" stability of sequences, compactness addresses the "external" finiteness of the space. The definition via open
covers is the most robust because it relies purely on topological structure rather than specific distance values. This
allows us to analyze global properties by "stitching together" local coordinate patches.
Definition: Open Cover and Subcover
Suppose \(X\) is a metric space and \(S\) is a subset of \(X\). A collection \(\mathcal{C}\) of open
subsets of \(X\) is called an open cover for \(S\) in \(X\) if and only if
\[
S \subseteq \bigcup \mathcal{C}.
\]
If \(\mathcal{C}\) is an open cover for \(S\), then any subset \(\mathcal{A}\) of \(\mathcal{C}\)
that also covers \(S\) is called an open subcover of \(\mathcal{C}\) for \(S\) in
\(X\).
Definition: Finite Intersection Property
Suppose \(X\) is a set and \(\mathcal{S}\) is a collection of subsets of \(X\). We say that \(\mathcal{S}\)
has the finite intersection property if and only if for every non-empty finite subset
\(\mathcal{F}\) of \(\mathcal{S}\), we have
\[
\bigcap \mathcal{F} \neq \emptyset.
\]
With these tools, we can now formally define compactness. The definition takes the "open cover" formulation
as primary, following the standard topological convention.
Definition: Compact Metric Space
A metric space \(X\) is called compact if and only if every open cover of \(X\) has a finite subcover. A subset
\(K \subseteq X\) is called compact if \(K\), viewed as a metric subspace with the restricted metric, is a compact metric space.
Equivalently, \(K\) is compact if and only if every collection of
open subsets of \(X\) whose
union contains \(K\) admits a finite subcollection still covering \(K\). This ambient characterization is the form most
often used in proofs.
Compactness is a multifaceted property. In metric spaces, several distinct conditions converge to describe the
same underlying structure of "finiteness within infinity." Understanding these equivalences is essential for
moving toward advanced fields like Manifold Theory and Information Geometry.
Theorem: Equivalence of Compactness
Suppose \(X\) is a non-empty metric space. The following statements are equivalent, and \(X\) is compact
if and only if it satisfies any one of the following:
-
Open Cover Criterion (Topological Definition)
Every open cover for \(X\) has a finite subcover.
Mathematical Significance: The foundational definition. It ensures that global properties can be derived from finite local
data. Partition of unity constructions in Manifold Theory and Differential Geometry run on exactly this
mechanism, stitching local charts into global objects.
-
Ball Cover Criterion
Every cover of \(X\) by open balls
has a finite subcover.
Mathematical Significance: A specialization for metric spaces. It suffices to check finite subcovers for ball covers rather
than arbitrary open covers. This simplification is the starting point for many classical proofs (the
Lebesgue number lemma,
total boundedness arguments).
-
Finite Intersection Criterion (Duality)
Every collection of closed subsets with the finite intersection property has a non-empty intersection.
Mathematical Significance: The De Morgan dual of the open cover definition. It provides the logic for existence
proofs. If a family of closed feasible sets has every finite subfamily with common points, then the entire family
shares a common point. In constrained optimization, this justifies taking limits of intersecting constraint sets.
-
Cantor's Criterion (Nested Intervals)
Every nest of non-empty closed subsets of \(X\) has
a non-empty intersection.
Mathematical Significance: The nest may be uncountable, in contrast with criterion (5). Unlike the Cantor Intersection
Theorem for complete spaces (which requires diameters \(\to 0\)), compactness removes this size condition, a strictly
stronger guarantee. This underlies branch-and-bound and interval methods. As the search
tree refines, nested candidate regions cannot collapse to emptiness.
-
Cantor's Sequence Criterion
Every sequence \(\{F_n\}\) of nested non-empty closed subsets has a non-empty intersection.
Mathematical Significance: A countable version of (4). Any nested sequence \(F_1 \supseteq F_2 \supseteq \cdots\) of
non-empty closed sets has non-empty intersection. This is useful whenever candidate regions are indexed by iteration count,
for example when proving that the intersection of feasible sets in successive relaxation hierarchies remains non-empty.
-
Convergence Criterion (Sequential Compactness)
Every sequence in \(X\) has a subsequence that
converges in \(X\).
Mathematical Significance: Convergent subsequences do not by themselves imply that an algorithm's iterates converge, but
they guarantee the existence of accumulation points within the feasible set. In non-convex optimization,
such accumulation points are identified with stationary/KKT points by separate arguments, making sequential compactness the
topological backbone of Bolzano-Weierstrass-type convergence analyses.
-
Spatial Criterion (Structural Decomposition)
\(X\) is complete and
totally bounded.
Mathematical Significance: A decomposition of compactness into "no holes" (completeness) and "finite size" (total
boundedness). Total boundedness is exactly the existence of finite \(\varepsilon\)-nets for every
\(\varepsilon \gt 0\), which powers covering-number bounds in statistical learning theory (Rademacher
complexity, PAC bounds) and Arzelà-Ascoli-type precompactness arguments in Functional Analysis.
-
BBW Criterion (Nearest-point Property)
\(X\) is bounded and has the nearest-point property.
Mathematical Significance: A compact set, viewed as a subset of any ambient metric space, admits
a nearest point from every external point. This is the foundation of projection operators
onto compact sets and of Vector Quantization (k-means-type algorithms rely on a
nearest-centroid assignment always being well-defined).
-
Cantor Set Criterion
\(X\) is a continuous image of the Cantor set.
Mathematical Significance: The Hausdorff-Alexandroff Theorem. It reveals that every non-empty compact metric space,
no matter how exotic, admits a sampling from a universal fractal template. That perspective is used in descriptive
set theory, in constructing space-filling curves, and in surjective encodings for symbolic dynamics.
-
Attained Bound Criterion (Operational View)
Every real continuous function defined on \(X\) is bounded and attains its bounds.
Mathematical Significance: A functional characterization. Compactness is exactly what makes every continuous real-valued
function attain its supremum and infimum. This is the Extreme Value Theorem in disguise, the single most
invoked consequence of compactness in analysis and optimization, and it guarantees that minimizers of continuous objectives
on compact feasible sets always exist.
Proof:
A proof traversing all ten criteria requires a web of implications. We establish the central bridge
(1) \(\Rightarrow\) (6) first, then close the remaining connections cluster by cluster, and state the
forward direction of (9) without proof. That bridge exposes the deep link between the covering and sequential viewpoints.
Core implication: (1) \(\Rightarrow\) (6).
Suppose every open cover of \(X\) admits a finite subcover. Let
\(\{x_n\}\) be a sequence in \(X\). Assume for contradiction that \(\{x_n\}\) has no convergent subsequence. Then for every
\(p \in X\) there exists \(r_p \gt 0\) such that the ball \(\mathcal{B}[p; r_p)\) contains \(x_n\) for only finitely many indices \(n\).
Otherwise the balls of radius \(1/k\) around \(p\) would each contain infinitely many terms, and we could inductively select
indices \(n_1 \lt n_2 \lt \cdots\) with \(x_{n_k} \in \mathcal{B}[p; 1/k)\), producing a subsequence that converges to \(p\) and
contradicts our assumption.
The family \(\{\mathcal{B}[p; r_p) : p \in X\}\) is an open cover of \(X\), since
open balls are open subsets, so by (1) a
finite subcollection \(\mathcal{B}[p_1; r_{p_1}), \ldots, \mathcal{B}[p_m; r_{p_m})\) covers \(X\). Then
\[
\mathbb{N} = \bigcup_{i=1}^{m} \{\, n : x_n \in \mathcal{B}[p_i; r_{p_i}) \,\},
\]
a finite union of finite sets. Yet \(\mathbb{N}\) is infinite, a contradiction. Hence a convergent subsequence exists.
Sequential \(\leftrightarrow\) covering cluster: (6) \(\Longleftrightarrow\) (7) \(\Longleftrightarrow\)
(1).
For (6) \(\Rightarrow\) (7): a Cauchy sequence with a convergent subsequence converges to the same limit, so
every Cauchy sequence in \(X\) converges
and \(X\) is complete. Suppose \(X\) were not totally bounded. Then some \(\varepsilon \gt 0\) admits no finite cover of \(X\)
by balls of radius \(\varepsilon\), so picking any \(y_1 \in X\) and then each \(y_{n+1}\) outside
\(\mathcal{B}[y_1; \varepsilon) \cup \cdots \cup \mathcal{B}[y_n; \varepsilon)\) produces a sequence \(\{y_n\}\) with
pairwise distances \(\ge \varepsilon\). Such a sequence admits no convergent subsequence, since any
convergent subsequence is Cauchy and would force \(d(y_{n_k}, y_{n_l}) \lt \varepsilon\) for sufficiently large \(k \ne l\),
contradicting the pairwise \(\ge \varepsilon\) condition. The existence of \(\{y_n\}\) therefore contradicts (6).
For (7) \(\Rightarrow\) (6): given any sequence, total boundedness provides, at each scale \(1/k\), a ball of radius
\(1/k\) containing infinitely many terms. Iterating this inside successive subsequences and extracting the diagonal produces a
subsequence in which any two terms beyond index \(k\) lie in a common ball of radius \(1/k\), hence within distance \(2/k\) of
each other. That subsequence is Cauchy, and completeness delivers its limit.
For (6) \(\Rightarrow\) (1): let \(\mathcal{U}\) be an open cover of \(X\) and let \(\delta \gt 0\) be a
Lebesgue number for it, which (6) supplies on its
own. Total boundedness is already in hand from (6) \(\Rightarrow\) (7), so finitely many balls of radius
\(\delta/3\) cover \(X\). Each of them is non-empty and has diameter \(\le 2\delta/3 \lt \delta\), hence lies in a single
member of \(\mathcal{U}\), and those finitely many members form a finite subcover.
Covering-style criteria: (1) \(\Longleftrightarrow\) (2) \(\Longleftrightarrow\) (3).
(1) \(\Rightarrow\) (2) because ball covers are a special case of open covers, so the finite-subcover property
transfers directly. (2) \(\Rightarrow\) (1): given any open cover \(\mathcal{U}\), for each \(x \in X\) pick
\(U_x \in \mathcal{U}\) with \(x \in U_x\) and, since \(U_x\) is open,
a ball \(\mathcal{B}[x; r_x) \subseteq U_x\). The
family \(\{\mathcal{B}[x; r_x)\}\) is a ball cover, so (2) yields a finite subcollection \(\mathcal{B}[x_1; r_{x_1}), \ldots, \mathcal{B}[x_m; r_{x_m})\)
covering \(X\), and the corresponding \(\{U_{x_1}, \ldots, U_{x_m}\}\) is a finite subcover of \(\mathcal{U}\).
(1) \(\Longleftrightarrow\) (3) is De Morgan duality. A family of open sets covers \(X\) if and only if the
complementary family of closed sets has empty intersection, and a finite subcover corresponds to a finite subfamily with empty
intersection, which is exactly the negation of the finite intersection property.
Nested-set criteria: (3) \(\Rightarrow\) (4) \(\Rightarrow\) (5).
(3) \(\Rightarrow\) (4) \(\Rightarrow\) (5) by specialization. A nest (totally ordered family under \(\subseteq\)) of
non-empty closed sets has the finite intersection property (any finite subfamily has a minimum element among its members, which
is non-empty), and any nested sequence \(F_1 \supseteq F_2 \supseteq \cdots\) is a countable nest. The converse directions,
restoring (3) from (4) or (5), are obtained indirectly, by passing through (6).
The cleanest closure is sequential. Given a sequence \(\{x_n\}\), the nested closed sets
\(F_n = \overline{\{x_k : k \ge n\}}\) have non-empty intersection by any of (3), (4), (5). If \(x \in \bigcap_n F_n\), every
ball about \(x\) meets \(\{x_k : k \ge n\}\) for every \(n\), so we may choose indices \(n_1 \lt n_2 \lt \cdots\) with
\(x_{n_k} \in \mathcal{B}[x; 1/k)\), and \(\{x_{n_k}\}\) converges to \(x\). Combined with (6) \(\Rightarrow\) (1) above, this
closes the equivalence loop.
Geometric and functional characterizations: (1) \(\Longleftrightarrow\) (8), (1) \(\Longleftrightarrow\) (10), (9).
(1) \(\Rightarrow\) (8): compactness implies boundedness, since for any fixed \(x_0 \in X\) the cover
\(\{\mathcal{B}[x_0; n) : n \in \mathbb{N}\}\) admits a finite subcover, placing \(X\) inside a single ball. For the
nearest-point property: for any
metric superspace \((Y, e)\) of \(X\) and
any \(y \in Y\), the function \(e(\cdot, y): X \to \mathbb{R}\) is continuous (Lipschitz with constant 1). The Extreme Value
Theorem, proved below as a consequence of (1), then shows that this function attains its minimum on the compact \(X\), and any
minimizer is a nearest point.
(8) \(\Rightarrow\) (6): suppose \(X\) is bounded and has the nearest-point property, and suppose for contradiction
that some sequence \(\{x_n\}\) in \(X\) has no convergent subsequence. Put \(D = \operatorname{diam}(X)\), which is finite
because \(X\) lies in some ball \(\mathcal{B}[a; R)\) and then \(D \le 2R\). Adjoin a new point \(z \notin X\), and define
\(e\) on \(X \cup \{z\}\) by \(e = d\) on \(X \times X\), \(e(z, z) = 0\), and \(e(x, z) = e(z, x)\) for \(x \in X\), where
\[
e(z, x) = \inf_{n \in \mathbb{N}} \Bigl( d(x, x_n) + D + 1 + \frac{1}{n} \Bigr) \quad (x \in X).
\]
Every term of the infimum exceeds \(D + 1\), so \(e(z, x) \ge D + 1 \gt 0\). Taking the infimum over \(n\) in
\(d(x, x_n) \le d(x, y) + d(y, x_n)\) gives \(e(z, x) \le d(x, y) + e(z, y)\) for \(x, y \in X\). In the other direction,
\(d(x, y) \le d(x, x_n) + d(x_n, x_m) + d(x_m, y) \le d(x, x_n) + D + d(x_m, y)\) for all \(n, m\), and the right side is
smaller than the sum of the \(n\)-th and \(m\)-th terms defining \(e(z, x)\) and \(e(z, y)\), so
\(d(x, y) \le e(z, x) + e(z, y)\). Hence \(e\) is a metric and \(X \cup \{z\}\) is a metric superspace of \(X\). Since
\(e(z, x_m) \le D + 1 + 1/m\) for every \(m\), we get \(\operatorname{dist}(z, X) = D + 1\). A nearest point \(x\) would
satisfy \(e(z, x) = D + 1\), that is, \(\inf_n \bigl(d(x, x_n) + 1/n\bigr) = 0\). Each term is at least \(1/n\), so a term
below \(1/m\) forces \(n \gt m\). We may therefore choose \(n_1\) with term below \(1\), and inductively \(n_{k+1}\) with term
below \(\min\{1/(k+1), 1/n_k\}\), which forces \(n_{k+1} \gt n_k\) and gives \(d(x, x_{n_k}) + 1/n_k \lt 1/k\) for every
\(k\). Then \(\{x_{n_k}\}\) converges to \(x\), which is excluded. So \(z\) has no nearest point in \(X\), contradicting (8).
Together with (6) \(\Rightarrow\) (1) above, this closes (8) \(\Rightarrow\) (1).
(1) \(\Rightarrow\) (10) is the Extreme Value Theorem, proved below. (10) \(\Rightarrow\) (7): we exhibit a
continuous function violating (10) whenever (7) fails. If \(X\) is not complete, then by the
Cauchy criterion for completeness
some Cauchy sequence \(\{x_n\}\) has no limit in \(X\). For each \(x \in X\), \(|d(x, x_n) - d(x, x_m)| \le d(x_n, x_m)\)
makes \(\{d(x, x_n)\}\) a Cauchy sequence of real numbers, so, \(\mathbb{R}\) being complete, \(f(x) = \lim_n d(x, x_n)\)
exists, and \(|f(x) - f(y)| \le d(x, y)\) makes \(f\) continuous. Each value \(f(x)\) is positive, since \(f(x) = 0\) would
mean \(x_n \to x\). Yet \(f(x_m) \le \sup_{n \ge m} d(x_m, x_n) \to 0\) as \(m \to \infty\) because the sequence is Cauchy (in
the
\(\varepsilon\)-\(N\) form), so
\(f\) does not attain its infimum \(0\). If instead \(X\) is not totally bounded, some \(r \gt 0\) admits no finite cover of
\(X\) by balls of radius \(r\), and picking any \(y_1 \in X\) and then each \(y_{n+1}\) outside
\(\mathcal{B}[y_1; r) \cup \cdots \cup \mathcal{B}[y_n; r)\) produces points with \(d(y_n, y_m) \ge r\) for \(n \ne m\). Put
\[
f(x) = \sum_{n=1}^{\infty} n \max\Bigl\{0, 1 - \frac{4\, d(x, y_n)}{r}\Bigr\}.
\]
The \(n\)-th term vanishes unless \(d(x, y_n) \lt r/4\), and a ball \(\mathcal{B}[x; r/8)\) meets at most one of the sets
\(\{w : d(w, y_n) \le r/4\}\), since meeting two of them would give \(d(y_n, y_m) \le r/4 + r/8 + r/8 + r/4 \lt r\). In
particular at most one term is non-zero at any point, so the sum is finite, and on each such ball \(f\) coincides with a
single continuous term (or with \(0\)), so \(f\) is continuous, while \(f(y_n) = n\) makes it unbounded. Either way (10)
fails, and together with (7) \(\Rightarrow\) (6) \(\Rightarrow\) (1) above, this closes (10) \(\Rightarrow\) (1).
(9): the Hausdorff-Alexandroff Theorem states that every non-empty compact metric space is a
continuous image of the Cantor set. This is a classical result in descriptive set theory. Its proof, which uses the
separability of compact metric spaces and a binary encoding of a countable base, is beyond the scope of this page. The reverse
direction (continuous images of the Cantor set are compact) follows from the Cantor set being compact (it is
\(\bigcap_k C_k\), where \(C_k\) is the union of the \(2^k\) closed intervals left after \(k\) rounds of removing open middle
thirds from \([0, 1]\), hence closed and bounded in \(\mathbb{R}\), and compact by the Heine-Borel theorem below) and from
continuous images of compact sets being compact, established separately below.
The Deferred Lemma: Lebesgue Numbers before Compactness
One step of the proof above rests on a statement that none of the ten criteria supplies directly. In passing from (6) to (1) we invoked a
Lebesgue number for an arbitrary open cover, and the
Lebesgue number lemma
cannot be used there: it takes compactness as its hypothesis, and compactness is exactly the conclusion under construction. We
therefore prove the sequential form on its own terms.
Lemma: Lebesgue Number from Sequential Compactness
Suppose \(X\) is a metric space in which every sequence has a subsequence that
converges in \(X\), and let
\(\mathcal{U}\) be an open cover of \(X\). Then there exists \(\delta \gt 0\), called a Lebesgue number for
\(\mathcal{U}\), such that \(\mathcal{B}[x; \delta)\) is contained in a single member of \(\mathcal{U}\) for every \(x \in X\).
Consequently every non-empty subset of \(X\) whose
diameter is smaller than \(\delta\) is
contained in a single member of \(\mathcal{U}\).
Proof:
Suppose no such \(\delta\) exists. In particular \(\delta = 1/n\) fails for each \(n \in \mathbb{N}\), which produces a point
\(x_n \in X\) for which \(\mathcal{B}[x_n; 1/n)\) is contained in no single member of \(\mathcal{U}\). By hypothesis the sequence
\(\{x_n\}\) has a subsequence \(\{x_{n_k}\}\) converging to some \(x \in X\).
Since \(\mathcal{U}\) covers \(X\), there is \(U \in \mathcal{U}\) containing \(x\), and since \(U\) is open,
a ball \(\mathcal{B}[x; \varepsilon) \subseteq U\).
Since \(x_{n_k} \to x\) and \(n_k \ge k\), we may fix \(k\) with both \(d(x_{n_k}, x) \lt \varepsilon/2\) and
\(1/n_k \lt \varepsilon/2\). For every \(y \in \mathcal{B}[x_{n_k}; 1/n_k)\) the triangle inequality then gives
\[
d(y, x) \le d(y, x_{n_k}) + d(x_{n_k}, x) \lt \frac{\varepsilon}{2} + \frac{\varepsilon}{2} = \varepsilon,
\]
so \(\mathcal{B}[x_{n_k}; 1/n_k) \subseteq \mathcal{B}[x; \varepsilon) \subseteq U\). That contradicts the choice of \(x_{n_k}\), and the
first assertion follows.
For the second assertion, let \(A\) be a non-empty subset with \(\operatorname{diam}(A) \lt \delta\) and fix
\(a \in A\). Every \(y \in A\) satisfies \(d(y, a) \le \operatorname{diam}(A) \lt \delta\), so
\(A \subseteq \mathcal{B}[a; \delta)\), and that ball lies in a single member of \(\mathcal{U}\).
Once the equivalence is closed, criteria (1) and (6) describe the same spaces, so this lemma and its compactness version say
the same thing about the same objects. The distinction is one of availability rather than of strength: inside the proof of the
equivalence, (6) may be assumed and (1) may not.
While all ten criteria are logically equivalent in non-empty metric spaces, they offer different practical advantages depending
on our objective:
-
(1) & (6) Theoretical Foundations:
Open Covers (1) and Sequential Convergence (6) are the universal languages of topology and analysis. Use these
for fundamental proofs of continuity and for transitioning between local and global properties.
-
(4) & (5) Algorithmic Assurance:
Cantor's Criteria power branch-and-bound and interval methods, where, provided the nested
regions are closed and each contains a point of a closed solution set, their intersection still contains a solution.
-
(7) & (8) Approximation & Search:
Total Boundedness (7) ensures a space can be represented by a finite "\(\varepsilon\)-net," while the Nearest-point property (8)
guarantees that a "best approximation" actually exists within the set. Both properties are crucial for
Vector Quantization and Signal
Processing.
-
(10) Optimization & Global Extrema:
The Attained Bound property is the engine behind the Extreme Value Theorem. It is the core reason why we
can guarantee the existence of global minima in constrained optimization.
-
(2), (3) & (9) Analytical Tools:
Ball Covers (2) simplify proofs in metric-specific contexts, Duality (3) allows us to switch between open and closed set logic,
and the Cantor Set image (9) reveals a universal template from which every non-empty compact metric space arises as
a continuous image.
Among these, Sequential Compactness (6) is particularly operational. In the context of the
Bolzano-Weierstrass Theorem, it ensures that even if we cannot find the exact global minimum in finite steps, any
sequence of parameters generated by our algorithm within a closed and bounded feasible set in \(\mathbb{R}^n\) contains a
subsequence that converges to a limit within that set.
Intuitively, if a subset of \(\mathbb{R}^n\) is not compact, it has "holes" (missing limit points) or "ends" that stretch to
infinity, allowing us to define a cover that requires infinitely many sets to handle the boundary or the unbounded tail. Compactness
asserts that no matter how small our open sets are, we only need a finite number of them to capture the entire set.
While "holes" often refer to a lack of completeness, compactness is a much stronger constraint. Completeness ensures that sequences
do not "fall through" internal gaps (for example, the limit of \(1, 1.4, 1.41...\) exists in \(\mathbb{R}\) but not in
\(\mathbb{Q}\)). However, compactness prevents sequences from "escaping" to infinity as well.
This difference is captured in Cantor's Intersection Theorem.
In a complete space, nested sets must shrink in size \(\operatorname{diam}(A_n) \to 0\) to guarantee a limit point. In a
compact space, the nested intersection of non-empty closed sets is
always non-empty, regardless of their size. Compactness leaves no room for any sequence
to drift away or vanish, providing the ultimate structural guarantee for stability.
The Heine-Borel Theorem
While the open cover definition is powerful, checking "every" open cover is difficult. Fortunately,
in the familiar Euclidean space \(\mathbb{R}^n\), compactness has a simple geometric characterization.
Theorem: Heine-Borel
A subset \(K \subseteq \mathbb{R}^n\) is compact if and only if it is:
- Closed: It contains all its limit points.
- Bounded: It can be contained within a ball of finite radius.
Proof Sketch:
(\(\Rightarrow\)) Compactness implies closed and bounded in any metric space, so we let \(K\) be a compact subset of
a metric space \(X\) (for the theorem, \(X = \mathbb{R}^n\)). The case \(K = \emptyset\) is trivial in both directions, so we
assume \(K \neq \emptyset\) throughout. Boundedness:
the cover \(\{\mathcal{B}[x; 1) : x \in K\}\) admits
a finite subcover \(\mathcal{B}[x_1; 1), \ldots, \mathcal{B}[x_m; 1)\). Since the finite set \(\{x_1, \ldots, x_m\}\) has
bounded diameter \(M := \max_{i,j} d(x_i, x_j)\), any \(y \in K\) lies in some \(\mathcal{B}[x_i; 1)\) and satisfies
\(d(y, x_1) \le d(y, x_i) + d(x_i, x_1) \lt 1 + M\), placing \(K\) inside \(\mathcal{B}[x_1; 1 + M)\).
Closedness:
if \(p\) is a limit point of \(K\) not in \(K\), consider the sets
\(X \setminus \mathcal{B}[p; 1/k]\) for \(k \in \mathbb{N}\). Each is
open, since whenever \(d(y, p) \gt 1/k\)
the triangle inequality gives \(\mathcal{B}[y; d(y, p) - 1/k) \subseteq X \setminus \mathcal{B}[p; 1/k]\). They cover \(K\)
(every \(x \in K\) has positive distance to \(p\), so \(1/k \lt d(x, p)\) for some \(k\)), but no finite subcover exists. A
finite subfamily, indexed by \(k_1, \ldots, k_m\), would leave \(K \cap \mathcal{B}[p; 1/\max_i k_i]\) uncovered, and that set
is non-empty. Since \(p \notin K\), the limit-point assumption gives \(\operatorname{dist}(p, K) = 0\), so the open ball
\(\mathcal{B}[p; 1/\max_i k_i)\) meets \(K\), and it lies inside the closed ball. This contradicts compactness of \(K\), so
\(p \in K\).
(\(\Leftarrow\)) This direction is the distinctively Euclidean content. By the equivalence above, it suffices to show
closed-and-bounded subsets of \(\mathbb{R}^n\) are complete and totally bounded (criterion (7)).
Completeness:
\(\mathbb{R}^n\) is complete, and a closed subset of a complete metric space is complete (see
complete subset theorem).
Total boundedness:
a bounded subset \(K \subseteq \mathbb{R}^n\) sits inside some cube \([-R, R]^n\). Given
\(\varepsilon \gt 0\), partition this cube into finitely many sub-cubes of side length at most \(\varepsilon / (2\sqrt{n})\),
each having diameter at most \(\varepsilon / 2 \lt \varepsilon\). From each sub-cube that meets \(K\) choose one point of \(K\).
Any other point of \(K\) in the same sub-cube is within \(\varepsilon / 2\) of that chosen point, so the finitely many balls of
\(K\) of radius \(\varepsilon\) about the chosen points cover \(K\).
It is a common pitfall to assume that all "closed and bounded" sets are compact. While the Heine-Borel Theorem guarantees this in
Euclidean space \(\mathbb{R}^n\), it does not hold in infinite-dimensional spaces, such as function spaces used in
Kernel Methods or Deep Learning theory.
In an infinite-dimensional Hilbert space, the unit ball is closed and bounded, but not compact
(compactness of the unit ball).
This is why optimization in infinite-dimensional settings requires additional structure to recover the existence guarantees that
Heine-Borel freely provides in Euclidean space. Two standard routes are weak compactness and
reduction to finite dimension via the representer theorem in reproducing kernel Hilbert spaces (RKHS). For the
first, a closed norm ball of positive radius in an infinite-dimensional Hilbert space, such as an L2 ball, is compact, and even
sequentially compact, in the weak topology,
though not in the norm topology, so every weakly lower semicontinuous objective attains its
minimum on it.
CS Insight: Feasible Regions
In optimization, L2 regularization comes in two forms, and the Heine-Borel theorem applies directly to only one of them. The
constrained form restricts the weights to \( \|w\|_2 \le C \) with \(C \geq 0\), a closed and bounded feasible
set in \(\mathbb{R}^n\). By the Heine-Borel theorem this set is compact, so by the Extreme Value Theorem a continuous loss
attains its minimum over it, and the constraint itself keeps the weights bounded. The penalized form, used in
ridge regression,
instead adds \(\lambda \|w\|_2^2\) with \(\lambda \gt 0\) to the loss and optimizes over all of \(\mathbb{R}^n\), which is not
compact. The two forms are nevertheless linked. If \(w^*\) minimizes the penalized objective, then \(w^*\) also minimizes the
loss over the ball \(\|w\|_2 \le \|w^*\|_2\), since a point of that ball with a smaller loss would also have a
smaller penalized objective.
Continuous Functions on Compact Sets
The power of compactness becomes fully evident when we examine continuous functions. It guarantees
not only that solutions exist, but also that the function behaves predictably
across the entire domain.
Theorem: Preservation of Compactness
If \(f: X \to Y\) is a continuous function between metric spaces and \(K \subseteq X\) is compact,
then the image \(f(K)\) is compact in \(Y\).
Proof:
Let \(\mathcal{C} = \{V_i\}_{i \in I}\) be any open cover of \(f(K)\) in \(Y\). By continuity of \(f\), each preimage
\(f^{-1}(V_i)\) is open in \(X\), and since \(\mathcal{C}\) covers \(f(K)\), the family \(\{f^{-1}(V_i)\}_{i \in I}\) covers
\(K\). Compactness of \(K\) yields a finite subcollection \(f^{-1}(V_{i_1}), \ldots, f^{-1}(V_{i_n})\) covering \(K\). Applying
\(f\) gives \(V_{i_1}, \ldots, V_{i_n}\) covering \(f(K)\). Thus every open cover of \(f(K)\) has a finite subcover, so \(f(K)\)
is compact.
The Heine-Borel characterization of compact subsets of \(\mathbb{R}\) turns this preservation property into a statement about
extreme values.
Theorem: Extreme Value Theorem (EVT)
Let \(K\) be a non-empty compact metric space and \(f: K \to \mathbb{R}\) be a continuous function.
Then \(f\) is bounded and attains its maximum and minimum values. That is, there exist
\(x_{\min}, x_{\max} \in K\) such that for all \(x \in K\):
\[
f(x_{\min}) \le f(x) \le f(x_{\max}).
\]
Proof:
By the Preservation of Compactness theorem,
\(f(K) \subseteq \mathbb{R}\) is compact. By the
Heine-Borel theorem, \(f(K)\) is closed
and bounded in \(\mathbb{R}\). Since \(K \ne \emptyset\) we have \(f(K) \ne \emptyset\), so boundedness
of \(f(K)\) means \(\sup f(K)\) and \(\inf f(K)\) exist and are finite.
Closedness of \(f(K)\) means these supremum and infimum belong to \(f(K)\) itself. Indeed, if \(\sup f(K) \notin f(K)\), then by
the defining property of the supremum every \(\varepsilon\)-neighborhood contains some \(y \in f(K)\) with
\(y \gt \sup f(K) - \varepsilon\), giving \(\operatorname{dist}(\sup f(K), f(K)) = 0\). Combined with \(\sup f(K) \in f(K)^c\)
(which gives \(\operatorname{dist}(\sup f(K), f(K)^c) = 0\) trivially), this places \(\sup f(K)\) in
\(\partial f(K) \subseteq \overline{f(K)} = f(K)\), a contradiction. Hence \(\sup f(K) \in f(K)\), and analogously
\(\inf f(K) \in f(K)\).
Therefore there exist \(x_{\max}, x_{\min} \in K\) with \(f(x_{\max}) = \sup f(K)\) and
\(f(x_{\min}) = \inf f(K)\), and for every \(x \in K\) we have \(f(x) \in f(K)\),
so \(f(x_{\min}) \le f(x) \le f(x_{\max})\).
Insight: Existence of Optima
The EVT is the bedrock of loss minimization. If our parameter space is compact (for example, via regularization constraints) and
our loss function is continuous, we are mathematically guaranteed that a global minimum exists. Without compactness (for
example, on an open interval or unbounded set), the function might only approach an infimum asymptotically (like \(f(x) = 1/x\)
on \((1, \infty)\)), leaving us chasing a solution that is not there.
Beyond existence of optima, compactness forces a uniform modulus of continuity across the entire domain (proved in the
Heine-Cantor theorem below). On a non-compact space, a continuous function can vary arbitrarily fast near a missing boundary point
or at infinity. The function \(f(x) = 1/x\) near \(x = 0\) illustrates how continuity alone gives no uniform control. Compactness
eliminates this pathology.
Proof Sketch:
Fix \(\varepsilon \gt 0\). By continuity, for each \(x \in K\) there exists \(\delta_x \gt 0\) such that
\(f(\mathcal{B}[x; \delta_x)) \subseteq \mathcal{B}[f(x); \varepsilon/2)\). The family \(\{\mathcal{B}[x; \delta_x / 2) : x \in K\}\), whose radii are
deliberately halved, is an open cover of \(K\), so by compactness it admits a finite subcover
\(\{\mathcal{B}[x_1; \delta_{x_1}/2), \ldots, \mathcal{B}[x_n; \delta_{x_n}/2)\}\).
Set \(\delta = \min_{i} \delta_{x_i} / 2 \gt 0\). This choice ensures \(\delta + \delta_{x_i}/2 \le \delta_{x_i}\) for every
\(i\), so that any point within distance \(\delta\) of a half-size ball lies inside the corresponding full-size ball where \(f\)
is controlled. Concretely, for any \(x, y \in K\) with \(d(x, y) \lt \delta\), the point \(x\) lies in some
\(\mathcal{B}[x_i; \delta_{x_i}/2)\), so \(d(x, x_i) \lt \delta_{x_i}/2\). Then
\(d(y, x_i) \le d(y, x) + d(x, x_i) \lt \delta + \delta_{x_i}/2 \le \delta_{x_i}\), placing \(y\) in \(\mathcal{B}[x_i; \delta_{x_i})\).
Both \(f(x)\) and \(f(y)\) therefore lie in \(\mathcal{B}[f(x_i); \varepsilon/2)\), so \(d(f(x), f(y)) \lt \varepsilon\). This \(\delta\)
depends only on \(\varepsilon\), proving uniform continuity.
Insight: Optimization Stability via Uniform Continuity
While the EVT guarantees that a solution exists, the Heine-Cantor Theorem gives our objective a uniform
behavior. On a compact domain, the modulus of continuity \(\omega_f(\delta) = \sup\{|f(x) - f(y)| : d(x, y) \lt \delta\}\)
tends to zero as \(\delta \to 0\), with a rate that does not vary across the domain.
A uniform modulus does not by itself bound the gradient. Uniform continuity is strictly weaker than Lipschitz continuity
(\(f(x) = \sqrt{x}\) on \([0, 1]\) is uniformly continuous but not Lipschitz near \(x = 0\)). To obtain a bounded gradient one
additionally applies the Extreme Value Theorem to \(\|\nabla f\|\) under a \(C^1\) assumption. The two pieces together, a
uniform modulus plus a bounded gradient, give uniform control of \(f\) across the entire compact search space. The classical
fixed step-size rule for gradient descent rests on one further property, a Lipschitz gradient. The same reasoning supplies it on
a compact convex domain when \(f\) is \(C^2\), by applying the Extreme Value Theorem to the norm of the Hessian.