Compactness

Introduction Open Covers and Compactness Heine-Borel Theorem Continuous Functions on Compact Sets

Introduction

In constrained optimization, we intuitively expect to find an optimal solution. Having explored the fundamental mathematical structures in earlier pages, we can now articulate exactly when this intuition holds. This page brings us to a pivotal milestone: Compactness.

While the Extreme Value Theorem guarantees that a continuous function on a non-empty closed and bounded subset of \(\mathbb{R}^n\) attains its extrema, this is merely a special case of a deeper truth. In general metric spaces, compactness captures the essence of being "topologically finite." It ensures that sequences cannot wander indefinitely without accumulating, and that local properties can be stitched together to control the global behavior of a system with finite resources.

Open Covers and Compactness

While completeness focuses on the "internal" stability of sequences, compactness addresses the "external" finiteness of the space. The definition via open covers is the most robust because it relies purely on topological structure rather than specific distance values. This allows us to analyze global properties by "stitching together" local coordinate patches.

Definition: Open Cover and Subcover

Suppose \(X\) is a metric space and \(S\) is a subset of \(X\). A collection \(\mathcal{C}\) of open subsets of \(X\) is called an open cover for \(S\) in \(X\) if and only if \[ S \subseteq \bigcup \mathcal{C}. \] If \(\mathcal{C}\) is an open cover for \(S\), then any subset \(\mathcal{A}\) of \(\mathcal{C}\) that also covers \(S\) is called an open subcover of \(\mathcal{C}\) for \(S\) in \(X\).

Definition: Finite Intersection Property

Suppose \(X\) is a set and \(\mathcal{S}\) is a collection of subsets of \(X\). We say that \(\mathcal{S}\) has the finite intersection property if and only if for every non-empty finite subset \(\mathcal{F}\) of \(\mathcal{S}\), we have \[ \bigcap \mathcal{F} \neq \emptyset. \]

With these tools, we can now formally define compactness. The definition takes the "open cover" formulation as primary, following the standard topological convention.

Definition: Compact Metric Space

A metric space \(X\) is called compact if and only if every open cover of \(X\) has a finite subcover. A subset \(K \subseteq X\) is called compact if \(K\), viewed as a metric subspace with the restricted metric, is a compact metric space. Equivalently, \(K\) is compact if and only if every collection of open subsets of \(X\) whose union contains \(K\) admits a finite subcollection still covering \(K\). This ambient characterization is the form most often used in proofs.

Compactness is a multifaceted property. In metric spaces, several distinct conditions converge to describe the same underlying structure of "finiteness within infinity." Understanding these equivalences is essential for moving toward advanced fields like Manifold Theory and Information Geometry.

Theorem: Equivalence of Compactness

Suppose \(X\) is a non-empty metric space. The following statements are equivalent, and \(X\) is compact if and only if it satisfies any one of the following:

  1. Open Cover Criterion (Topological Definition)
    Every open cover for \(X\) has a finite subcover.
    Mathematical Significance: The foundational definition. It ensures that global properties can be derived from finite local data. Partition of unity constructions in Manifold Theory and Differential Geometry run on exactly this mechanism, stitching local charts into global objects.
  2. Ball Cover Criterion
    Every cover of \(X\) by open balls has a finite subcover.
    Mathematical Significance: A specialization for metric spaces. It suffices to check finite subcovers for ball covers rather than arbitrary open covers. This simplification is the starting point for many classical proofs (the Lebesgue number lemma, total boundedness arguments).
  3. Finite Intersection Criterion (Duality)
    Every collection of closed subsets with the finite intersection property has a non-empty intersection.
    Mathematical Significance: The De Morgan dual of the open cover definition. It provides the logic for existence proofs. If a family of closed feasible sets has every finite subfamily with common points, then the entire family shares a common point. In constrained optimization, this justifies taking limits of intersecting constraint sets.
  4. Cantor's Criterion (Nested Intervals)
    Every nest of non-empty closed subsets of \(X\) has a non-empty intersection.
    Mathematical Significance: The nest may be uncountable, in contrast with criterion (5). Unlike the Cantor Intersection Theorem for complete spaces (which requires diameters \(\to 0\)), compactness removes this size condition, a strictly stronger guarantee. This underlies branch-and-bound and interval methods. As the search tree refines, nested candidate regions cannot collapse to emptiness.
  5. Cantor's Sequence Criterion
    Every sequence \(\{F_n\}\) of nested non-empty closed subsets has a non-empty intersection.
    Mathematical Significance: A countable version of (4). Any nested sequence \(F_1 \supseteq F_2 \supseteq \cdots\) of non-empty closed sets has non-empty intersection. This is useful whenever candidate regions are indexed by iteration count, for example when proving that the intersection of feasible sets in successive relaxation hierarchies remains non-empty.
  6. Convergence Criterion (Sequential Compactness)
    Every sequence in \(X\) has a subsequence that converges in \(X\).
    Mathematical Significance: Convergent subsequences do not by themselves imply that an algorithm's iterates converge, but they guarantee the existence of accumulation points within the feasible set. In non-convex optimization, such accumulation points are identified with stationary/KKT points by separate arguments, making sequential compactness the topological backbone of Bolzano-Weierstrass-type convergence analyses.
  7. Spatial Criterion (Structural Decomposition)
    \(X\) is complete and totally bounded.
    Mathematical Significance: A decomposition of compactness into "no holes" (completeness) and "finite size" (total boundedness). Total boundedness is exactly the existence of finite \(\varepsilon\)-nets for every \(\varepsilon \gt 0\), which powers covering-number bounds in statistical learning theory (Rademacher complexity, PAC bounds) and Arzelà-Ascoli-type precompactness arguments in Functional Analysis.
  8. BBW Criterion (Nearest-point Property)
    \(X\) is bounded and has the nearest-point property.
    Mathematical Significance: A compact set, viewed as a subset of any ambient metric space, admits a nearest point from every external point. This is the foundation of projection operators onto compact sets and of Vector Quantization (k-means-type algorithms rely on a nearest-centroid assignment always being well-defined).
  9. Cantor Set Criterion
    \(X\) is a continuous image of the Cantor set.
    Mathematical Significance: The Hausdorff-Alexandroff Theorem. It reveals that every non-empty compact metric space, no matter how exotic, admits a sampling from a universal fractal template. That perspective is used in descriptive set theory, in constructing space-filling curves, and in surjective encodings for symbolic dynamics.
  10. Attained Bound Criterion (Operational View)
    Every real continuous function defined on \(X\) is bounded and attains its bounds.
    Mathematical Significance: A functional characterization. Compactness is exactly what makes every continuous real-valued function attain its supremum and infimum. This is the Extreme Value Theorem in disguise, the single most invoked consequence of compactness in analysis and optimization, and it guarantees that minimizers of continuous objectives on compact feasible sets always exist.
Proof:

A proof traversing all ten criteria requires a web of implications. We establish the central bridge (1) \(\Rightarrow\) (6) first, then close the remaining connections cluster by cluster, and state the forward direction of (9) without proof. That bridge exposes the deep link between the covering and sequential viewpoints.

Core implication: (1) \(\Rightarrow\) (6).
Suppose every open cover of \(X\) admits a finite subcover. Let \(\{x_n\}\) be a sequence in \(X\). Assume for contradiction that \(\{x_n\}\) has no convergent subsequence. Then for every \(p \in X\) there exists \(r_p \gt 0\) such that the ball \(\mathcal{B}[p; r_p)\) contains \(x_n\) for only finitely many indices \(n\). Otherwise the balls of radius \(1/k\) around \(p\) would each contain infinitely many terms, and we could inductively select indices \(n_1 \lt n_2 \lt \cdots\) with \(x_{n_k} \in \mathcal{B}[p; 1/k)\), producing a subsequence that converges to \(p\) and contradicts our assumption.

The family \(\{\mathcal{B}[p; r_p) : p \in X\}\) is an open cover of \(X\), since open balls are open subsets, so by (1) a finite subcollection \(\mathcal{B}[p_1; r_{p_1}), \ldots, \mathcal{B}[p_m; r_{p_m})\) covers \(X\). Then \[ \mathbb{N} = \bigcup_{i=1}^{m} \{\, n : x_n \in \mathcal{B}[p_i; r_{p_i}) \,\}, \] a finite union of finite sets. Yet \(\mathbb{N}\) is infinite, a contradiction. Hence a convergent subsequence exists.

Sequential \(\leftrightarrow\) covering cluster: (6) \(\Longleftrightarrow\) (7) \(\Longleftrightarrow\) (1).
For (6) \(\Rightarrow\) (7): a Cauchy sequence with a convergent subsequence converges to the same limit, so every Cauchy sequence in \(X\) converges and \(X\) is complete. Suppose \(X\) were not totally bounded. Then some \(\varepsilon \gt 0\) admits no finite cover of \(X\) by balls of radius \(\varepsilon\), so picking any \(y_1 \in X\) and then each \(y_{n+1}\) outside \(\mathcal{B}[y_1; \varepsilon) \cup \cdots \cup \mathcal{B}[y_n; \varepsilon)\) produces a sequence \(\{y_n\}\) with pairwise distances \(\ge \varepsilon\). Such a sequence admits no convergent subsequence, since any convergent subsequence is Cauchy and would force \(d(y_{n_k}, y_{n_l}) \lt \varepsilon\) for sufficiently large \(k \ne l\), contradicting the pairwise \(\ge \varepsilon\) condition. The existence of \(\{y_n\}\) therefore contradicts (6).

For (7) \(\Rightarrow\) (6): given any sequence, total boundedness provides, at each scale \(1/k\), a ball of radius \(1/k\) containing infinitely many terms. Iterating this inside successive subsequences and extracting the diagonal produces a subsequence in which any two terms beyond index \(k\) lie in a common ball of radius \(1/k\), hence within distance \(2/k\) of each other. That subsequence is Cauchy, and completeness delivers its limit.

For (6) \(\Rightarrow\) (1): let \(\mathcal{U}\) be an open cover of \(X\) and let \(\delta \gt 0\) be a Lebesgue number for it, which (6) supplies on its own. Total boundedness is already in hand from (6) \(\Rightarrow\) (7), so finitely many balls of radius \(\delta/3\) cover \(X\). Each of them is non-empty and has diameter \(\le 2\delta/3 \lt \delta\), hence lies in a single member of \(\mathcal{U}\), and those finitely many members form a finite subcover.

Covering-style criteria: (1) \(\Longleftrightarrow\) (2) \(\Longleftrightarrow\) (3).
(1) \(\Rightarrow\) (2) because ball covers are a special case of open covers, so the finite-subcover property transfers directly. (2) \(\Rightarrow\) (1): given any open cover \(\mathcal{U}\), for each \(x \in X\) pick \(U_x \in \mathcal{U}\) with \(x \in U_x\) and, since \(U_x\) is open, a ball \(\mathcal{B}[x; r_x) \subseteq U_x\). The family \(\{\mathcal{B}[x; r_x)\}\) is a ball cover, so (2) yields a finite subcollection \(\mathcal{B}[x_1; r_{x_1}), \ldots, \mathcal{B}[x_m; r_{x_m})\) covering \(X\), and the corresponding \(\{U_{x_1}, \ldots, U_{x_m}\}\) is a finite subcover of \(\mathcal{U}\).

(1) \(\Longleftrightarrow\) (3) is De Morgan duality. A family of open sets covers \(X\) if and only if the complementary family of closed sets has empty intersection, and a finite subcover corresponds to a finite subfamily with empty intersection, which is exactly the negation of the finite intersection property.

Nested-set criteria: (3) \(\Rightarrow\) (4) \(\Rightarrow\) (5).
(3) \(\Rightarrow\) (4) \(\Rightarrow\) (5) by specialization. A nest (totally ordered family under \(\subseteq\)) of non-empty closed sets has the finite intersection property (any finite subfamily has a minimum element among its members, which is non-empty), and any nested sequence \(F_1 \supseteq F_2 \supseteq \cdots\) is a countable nest. The converse directions, restoring (3) from (4) or (5), are obtained indirectly, by passing through (6).

The cleanest closure is sequential. Given a sequence \(\{x_n\}\), the nested closed sets \(F_n = \overline{\{x_k : k \ge n\}}\) have non-empty intersection by any of (3), (4), (5). If \(x \in \bigcap_n F_n\), every ball about \(x\) meets \(\{x_k : k \ge n\}\) for every \(n\), so we may choose indices \(n_1 \lt n_2 \lt \cdots\) with \(x_{n_k} \in \mathcal{B}[x; 1/k)\), and \(\{x_{n_k}\}\) converges to \(x\). Combined with (6) \(\Rightarrow\) (1) above, this closes the equivalence loop.

Geometric and functional characterizations: (1) \(\Longleftrightarrow\) (8), (1) \(\Longleftrightarrow\) (10), (9).
(1) \(\Rightarrow\) (8): compactness implies boundedness, since for any fixed \(x_0 \in X\) the cover \(\{\mathcal{B}[x_0; n) : n \in \mathbb{N}\}\) admits a finite subcover, placing \(X\) inside a single ball. For the nearest-point property: for any metric superspace \((Y, e)\) of \(X\) and any \(y \in Y\), the function \(e(\cdot, y): X \to \mathbb{R}\) is continuous (Lipschitz with constant 1). The Extreme Value Theorem, proved below as a consequence of (1), then shows that this function attains its minimum on the compact \(X\), and any minimizer is a nearest point.

(8) \(\Rightarrow\) (6): suppose \(X\) is bounded and has the nearest-point property, and suppose for contradiction that some sequence \(\{x_n\}\) in \(X\) has no convergent subsequence. Put \(D = \operatorname{diam}(X)\), which is finite because \(X\) lies in some ball \(\mathcal{B}[a; R)\) and then \(D \le 2R\). Adjoin a new point \(z \notin X\), and define \(e\) on \(X \cup \{z\}\) by \(e = d\) on \(X \times X\), \(e(z, z) = 0\), and \(e(x, z) = e(z, x)\) for \(x \in X\), where \[ e(z, x) = \inf_{n \in \mathbb{N}} \Bigl( d(x, x_n) + D + 1 + \frac{1}{n} \Bigr) \quad (x \in X). \] Every term of the infimum exceeds \(D + 1\), so \(e(z, x) \ge D + 1 \gt 0\). Taking the infimum over \(n\) in \(d(x, x_n) \le d(x, y) + d(y, x_n)\) gives \(e(z, x) \le d(x, y) + e(z, y)\) for \(x, y \in X\). In the other direction, \(d(x, y) \le d(x, x_n) + d(x_n, x_m) + d(x_m, y) \le d(x, x_n) + D + d(x_m, y)\) for all \(n, m\), and the right side is smaller than the sum of the \(n\)-th and \(m\)-th terms defining \(e(z, x)\) and \(e(z, y)\), so \(d(x, y) \le e(z, x) + e(z, y)\). Hence \(e\) is a metric and \(X \cup \{z\}\) is a metric superspace of \(X\). Since \(e(z, x_m) \le D + 1 + 1/m\) for every \(m\), we get \(\operatorname{dist}(z, X) = D + 1\). A nearest point \(x\) would satisfy \(e(z, x) = D + 1\), that is, \(\inf_n \bigl(d(x, x_n) + 1/n\bigr) = 0\). Each term is at least \(1/n\), so a term below \(1/m\) forces \(n \gt m\). We may therefore choose \(n_1\) with term below \(1\), and inductively \(n_{k+1}\) with term below \(\min\{1/(k+1), 1/n_k\}\), which forces \(n_{k+1} \gt n_k\) and gives \(d(x, x_{n_k}) + 1/n_k \lt 1/k\) for every \(k\). Then \(\{x_{n_k}\}\) converges to \(x\), which is excluded. So \(z\) has no nearest point in \(X\), contradicting (8). Together with (6) \(\Rightarrow\) (1) above, this closes (8) \(\Rightarrow\) (1).

(1) \(\Rightarrow\) (10) is the Extreme Value Theorem, proved below. (10) \(\Rightarrow\) (7): we exhibit a continuous function violating (10) whenever (7) fails. If \(X\) is not complete, then by the Cauchy criterion for completeness some Cauchy sequence \(\{x_n\}\) has no limit in \(X\). For each \(x \in X\), \(|d(x, x_n) - d(x, x_m)| \le d(x_n, x_m)\) makes \(\{d(x, x_n)\}\) a Cauchy sequence of real numbers, so, \(\mathbb{R}\) being complete, \(f(x) = \lim_n d(x, x_n)\) exists, and \(|f(x) - f(y)| \le d(x, y)\) makes \(f\) continuous. Each value \(f(x)\) is positive, since \(f(x) = 0\) would mean \(x_n \to x\). Yet \(f(x_m) \le \sup_{n \ge m} d(x_m, x_n) \to 0\) as \(m \to \infty\) because the sequence is Cauchy (in the \(\varepsilon\)-\(N\) form), so \(f\) does not attain its infimum \(0\). If instead \(X\) is not totally bounded, some \(r \gt 0\) admits no finite cover of \(X\) by balls of radius \(r\), and picking any \(y_1 \in X\) and then each \(y_{n+1}\) outside \(\mathcal{B}[y_1; r) \cup \cdots \cup \mathcal{B}[y_n; r)\) produces points with \(d(y_n, y_m) \ge r\) for \(n \ne m\). Put \[ f(x) = \sum_{n=1}^{\infty} n \max\Bigl\{0, 1 - \frac{4\, d(x, y_n)}{r}\Bigr\}. \] The \(n\)-th term vanishes unless \(d(x, y_n) \lt r/4\), and a ball \(\mathcal{B}[x; r/8)\) meets at most one of the sets \(\{w : d(w, y_n) \le r/4\}\), since meeting two of them would give \(d(y_n, y_m) \le r/4 + r/8 + r/8 + r/4 \lt r\). In particular at most one term is non-zero at any point, so the sum is finite, and on each such ball \(f\) coincides with a single continuous term (or with \(0\)), so \(f\) is continuous, while \(f(y_n) = n\) makes it unbounded. Either way (10) fails, and together with (7) \(\Rightarrow\) (6) \(\Rightarrow\) (1) above, this closes (10) \(\Rightarrow\) (1).

(9): the Hausdorff-Alexandroff Theorem states that every non-empty compact metric space is a continuous image of the Cantor set. This is a classical result in descriptive set theory. Its proof, which uses the separability of compact metric spaces and a binary encoding of a countable base, is beyond the scope of this page. The reverse direction (continuous images of the Cantor set are compact) follows from the Cantor set being compact (it is \(\bigcap_k C_k\), where \(C_k\) is the union of the \(2^k\) closed intervals left after \(k\) rounds of removing open middle thirds from \([0, 1]\), hence closed and bounded in \(\mathbb{R}\), and compact by the Heine-Borel theorem below) and from continuous images of compact sets being compact, established separately below.

The Deferred Lemma: Lebesgue Numbers before Compactness

One step of the proof above rests on a statement that none of the ten criteria supplies directly. In passing from (6) to (1) we invoked a Lebesgue number for an arbitrary open cover, and the Lebesgue number lemma cannot be used there: it takes compactness as its hypothesis, and compactness is exactly the conclusion under construction. We therefore prove the sequential form on its own terms.

Lemma: Lebesgue Number from Sequential Compactness

Suppose \(X\) is a metric space in which every sequence has a subsequence that converges in \(X\), and let \(\mathcal{U}\) be an open cover of \(X\). Then there exists \(\delta \gt 0\), called a Lebesgue number for \(\mathcal{U}\), such that \(\mathcal{B}[x; \delta)\) is contained in a single member of \(\mathcal{U}\) for every \(x \in X\). Consequently every non-empty subset of \(X\) whose diameter is smaller than \(\delta\) is contained in a single member of \(\mathcal{U}\).

Proof:

Suppose no such \(\delta\) exists. In particular \(\delta = 1/n\) fails for each \(n \in \mathbb{N}\), which produces a point \(x_n \in X\) for which \(\mathcal{B}[x_n; 1/n)\) is contained in no single member of \(\mathcal{U}\). By hypothesis the sequence \(\{x_n\}\) has a subsequence \(\{x_{n_k}\}\) converging to some \(x \in X\).

Since \(\mathcal{U}\) covers \(X\), there is \(U \in \mathcal{U}\) containing \(x\), and since \(U\) is open, a ball \(\mathcal{B}[x; \varepsilon) \subseteq U\). Since \(x_{n_k} \to x\) and \(n_k \ge k\), we may fix \(k\) with both \(d(x_{n_k}, x) \lt \varepsilon/2\) and \(1/n_k \lt \varepsilon/2\). For every \(y \in \mathcal{B}[x_{n_k}; 1/n_k)\) the triangle inequality then gives \[ d(y, x) \le d(y, x_{n_k}) + d(x_{n_k}, x) \lt \frac{\varepsilon}{2} + \frac{\varepsilon}{2} = \varepsilon, \] so \(\mathcal{B}[x_{n_k}; 1/n_k) \subseteq \mathcal{B}[x; \varepsilon) \subseteq U\). That contradicts the choice of \(x_{n_k}\), and the first assertion follows.

For the second assertion, let \(A\) be a non-empty subset with \(\operatorname{diam}(A) \lt \delta\) and fix \(a \in A\). Every \(y \in A\) satisfies \(d(y, a) \le \operatorname{diam}(A) \lt \delta\), so \(A \subseteq \mathcal{B}[a; \delta)\), and that ball lies in a single member of \(\mathcal{U}\).

Once the equivalence is closed, criteria (1) and (6) describe the same spaces, so this lemma and its compactness version say the same thing about the same objects. The distinction is one of availability rather than of strength: inside the proof of the equivalence, (6) may be assumed and (1) may not.

While all ten criteria are logically equivalent in non-empty metric spaces, they offer different practical advantages depending on our objective:

Among these, Sequential Compactness (6) is particularly operational. In the context of the Bolzano-Weierstrass Theorem, it ensures that even if we cannot find the exact global minimum in finite steps, any sequence of parameters generated by our algorithm within a closed and bounded feasible set in \(\mathbb{R}^n\) contains a subsequence that converges to a limit within that set.

Intuitively, if a subset of \(\mathbb{R}^n\) is not compact, it has "holes" (missing limit points) or "ends" that stretch to infinity, allowing us to define a cover that requires infinitely many sets to handle the boundary or the unbounded tail. Compactness asserts that no matter how small our open sets are, we only need a finite number of them to capture the entire set.

While "holes" often refer to a lack of completeness, compactness is a much stronger constraint. Completeness ensures that sequences do not "fall through" internal gaps (for example, the limit of \(1, 1.4, 1.41...\) exists in \(\mathbb{R}\) but not in \(\mathbb{Q}\)). However, compactness prevents sequences from "escaping" to infinity as well.

This difference is captured in Cantor's Intersection Theorem. In a complete space, nested sets must shrink in size \(\operatorname{diam}(A_n) \to 0\) to guarantee a limit point. In a compact space, the nested intersection of non-empty closed sets is always non-empty, regardless of their size. Compactness leaves no room for any sequence to drift away or vanish, providing the ultimate structural guarantee for stability.

The Heine-Borel Theorem

While the open cover definition is powerful, checking "every" open cover is difficult. Fortunately, in the familiar Euclidean space \(\mathbb{R}^n\), compactness has a simple geometric characterization.

Theorem: Heine-Borel

A subset \(K \subseteq \mathbb{R}^n\) is compact if and only if it is:

  1. Closed: It contains all its limit points.
  2. Bounded: It can be contained within a ball of finite radius.
Proof Sketch:

(\(\Rightarrow\)) Compactness implies closed and bounded in any metric space, so we let \(K\) be a compact subset of a metric space \(X\) (for the theorem, \(X = \mathbb{R}^n\)). The case \(K = \emptyset\) is trivial in both directions, so we assume \(K \neq \emptyset\) throughout. Boundedness:
the cover \(\{\mathcal{B}[x; 1) : x \in K\}\) admits a finite subcover \(\mathcal{B}[x_1; 1), \ldots, \mathcal{B}[x_m; 1)\). Since the finite set \(\{x_1, \ldots, x_m\}\) has bounded diameter \(M := \max_{i,j} d(x_i, x_j)\), any \(y \in K\) lies in some \(\mathcal{B}[x_i; 1)\) and satisfies \(d(y, x_1) \le d(y, x_i) + d(x_i, x_1) \lt 1 + M\), placing \(K\) inside \(\mathcal{B}[x_1; 1 + M)\).

Closedness:
if \(p\) is a limit point of \(K\) not in \(K\), consider the sets \(X \setminus \mathcal{B}[p; 1/k]\) for \(k \in \mathbb{N}\). Each is open, since whenever \(d(y, p) \gt 1/k\) the triangle inequality gives \(\mathcal{B}[y; d(y, p) - 1/k) \subseteq X \setminus \mathcal{B}[p; 1/k]\). They cover \(K\) (every \(x \in K\) has positive distance to \(p\), so \(1/k \lt d(x, p)\) for some \(k\)), but no finite subcover exists. A finite subfamily, indexed by \(k_1, \ldots, k_m\), would leave \(K \cap \mathcal{B}[p; 1/\max_i k_i]\) uncovered, and that set is non-empty. Since \(p \notin K\), the limit-point assumption gives \(\operatorname{dist}(p, K) = 0\), so the open ball \(\mathcal{B}[p; 1/\max_i k_i)\) meets \(K\), and it lies inside the closed ball. This contradicts compactness of \(K\), so \(p \in K\).

(\(\Leftarrow\)) This direction is the distinctively Euclidean content. By the equivalence above, it suffices to show closed-and-bounded subsets of \(\mathbb{R}^n\) are complete and totally bounded (criterion (7)). Completeness:
\(\mathbb{R}^n\) is complete, and a closed subset of a complete metric space is complete (see complete subset theorem).

Total boundedness:
a bounded subset \(K \subseteq \mathbb{R}^n\) sits inside some cube \([-R, R]^n\). Given \(\varepsilon \gt 0\), partition this cube into finitely many sub-cubes of side length at most \(\varepsilon / (2\sqrt{n})\), each having diameter at most \(\varepsilon / 2 \lt \varepsilon\). From each sub-cube that meets \(K\) choose one point of \(K\). Any other point of \(K\) in the same sub-cube is within \(\varepsilon / 2\) of that chosen point, so the finitely many balls of \(K\) of radius \(\varepsilon\) about the chosen points cover \(K\).

It is a common pitfall to assume that all "closed and bounded" sets are compact. While the Heine-Borel Theorem guarantees this in Euclidean space \(\mathbb{R}^n\), it does not hold in infinite-dimensional spaces, such as function spaces used in Kernel Methods or Deep Learning theory.

In an infinite-dimensional Hilbert space, the unit ball is closed and bounded, but not compact (compactness of the unit ball). This is why optimization in infinite-dimensional settings requires additional structure to recover the existence guarantees that Heine-Borel freely provides in Euclidean space. Two standard routes are weak compactness and reduction to finite dimension via the representer theorem in reproducing kernel Hilbert spaces (RKHS). For the first, a closed norm ball of positive radius in an infinite-dimensional Hilbert space, such as an L2 ball, is compact, and even sequentially compact, in the weak topology, though not in the norm topology, so every weakly lower semicontinuous objective attains its minimum on it.

CS Insight: Feasible Regions

In optimization, L2 regularization comes in two forms, and the Heine-Borel theorem applies directly to only one of them. The constrained form restricts the weights to \( \|w\|_2 \le C \) with \(C \geq 0\), a closed and bounded feasible set in \(\mathbb{R}^n\). By the Heine-Borel theorem this set is compact, so by the Extreme Value Theorem a continuous loss attains its minimum over it, and the constraint itself keeps the weights bounded. The penalized form, used in ridge regression, instead adds \(\lambda \|w\|_2^2\) with \(\lambda \gt 0\) to the loss and optimizes over all of \(\mathbb{R}^n\), which is not compact. The two forms are nevertheless linked. If \(w^*\) minimizes the penalized objective, then \(w^*\) also minimizes the loss over the ball \(\|w\|_2 \le \|w^*\|_2\), since a point of that ball with a smaller loss would also have a smaller penalized objective.

Continuous Functions on Compact Sets

The power of compactness becomes fully evident when we examine continuous functions. It guarantees not only that solutions exist, but also that the function behaves predictably across the entire domain.

Theorem: Preservation of Compactness

If \(f: X \to Y\) is a continuous function between metric spaces and \(K \subseteq X\) is compact, then the image \(f(K)\) is compact in \(Y\).

Proof:

Let \(\mathcal{C} = \{V_i\}_{i \in I}\) be any open cover of \(f(K)\) in \(Y\). By continuity of \(f\), each preimage \(f^{-1}(V_i)\) is open in \(X\), and since \(\mathcal{C}\) covers \(f(K)\), the family \(\{f^{-1}(V_i)\}_{i \in I}\) covers \(K\). Compactness of \(K\) yields a finite subcollection \(f^{-1}(V_{i_1}), \ldots, f^{-1}(V_{i_n})\) covering \(K\). Applying \(f\) gives \(V_{i_1}, \ldots, V_{i_n}\) covering \(f(K)\). Thus every open cover of \(f(K)\) has a finite subcover, so \(f(K)\) is compact.

The Heine-Borel characterization of compact subsets of \(\mathbb{R}\) turns this preservation property into a statement about extreme values.

Theorem: Extreme Value Theorem (EVT)

Let \(K\) be a non-empty compact metric space and \(f: K \to \mathbb{R}\) be a continuous function. Then \(f\) is bounded and attains its maximum and minimum values. That is, there exist \(x_{\min}, x_{\max} \in K\) such that for all \(x \in K\): \[ f(x_{\min}) \le f(x) \le f(x_{\max}). \]

Proof:

By the Preservation of Compactness theorem, \(f(K) \subseteq \mathbb{R}\) is compact. By the Heine-Borel theorem, \(f(K)\) is closed and bounded in \(\mathbb{R}\). Since \(K \ne \emptyset\) we have \(f(K) \ne \emptyset\), so boundedness of \(f(K)\) means \(\sup f(K)\) and \(\inf f(K)\) exist and are finite.

Closedness of \(f(K)\) means these supremum and infimum belong to \(f(K)\) itself. Indeed, if \(\sup f(K) \notin f(K)\), then by the defining property of the supremum every \(\varepsilon\)-neighborhood contains some \(y \in f(K)\) with \(y \gt \sup f(K) - \varepsilon\), giving \(\operatorname{dist}(\sup f(K), f(K)) = 0\). Combined with \(\sup f(K) \in f(K)^c\) (which gives \(\operatorname{dist}(\sup f(K), f(K)^c) = 0\) trivially), this places \(\sup f(K)\) in \(\partial f(K) \subseteq \overline{f(K)} = f(K)\), a contradiction. Hence \(\sup f(K) \in f(K)\), and analogously \(\inf f(K) \in f(K)\).

Therefore there exist \(x_{\max}, x_{\min} \in K\) with \(f(x_{\max}) = \sup f(K)\) and \(f(x_{\min}) = \inf f(K)\), and for every \(x \in K\) we have \(f(x) \in f(K)\), so \(f(x_{\min}) \le f(x) \le f(x_{\max})\).

Insight: Existence of Optima

The EVT is the bedrock of loss minimization. If our parameter space is compact (for example, via regularization constraints) and our loss function is continuous, we are mathematically guaranteed that a global minimum exists. Without compactness (for example, on an open interval or unbounded set), the function might only approach an infimum asymptotically (like \(f(x) = 1/x\) on \((1, \infty)\)), leaving us chasing a solution that is not there.

Beyond existence of optima, compactness forces a uniform modulus of continuity across the entire domain (proved in the Heine-Cantor theorem below). On a non-compact space, a continuous function can vary arbitrarily fast near a missing boundary point or at infinity. The function \(f(x) = 1/x\) near \(x = 0\) illustrates how continuity alone gives no uniform control. Compactness eliminates this pathology.

Theorem: Heine-Cantor (Uniform Continuity)

If \(f: K \to Y\) is a continuous function from a non-empty compact metric space \(K\) to a metric space \(Y\), then \(f\) is uniformly continuous on \(K\).

Proof Sketch:

Fix \(\varepsilon \gt 0\). By continuity, for each \(x \in K\) there exists \(\delta_x \gt 0\) such that \(f(\mathcal{B}[x; \delta_x)) \subseteq \mathcal{B}[f(x); \varepsilon/2)\). The family \(\{\mathcal{B}[x; \delta_x / 2) : x \in K\}\), whose radii are deliberately halved, is an open cover of \(K\), so by compactness it admits a finite subcover \(\{\mathcal{B}[x_1; \delta_{x_1}/2), \ldots, \mathcal{B}[x_n; \delta_{x_n}/2)\}\).

Set \(\delta = \min_{i} \delta_{x_i} / 2 \gt 0\). This choice ensures \(\delta + \delta_{x_i}/2 \le \delta_{x_i}\) for every \(i\), so that any point within distance \(\delta\) of a half-size ball lies inside the corresponding full-size ball where \(f\) is controlled. Concretely, for any \(x, y \in K\) with \(d(x, y) \lt \delta\), the point \(x\) lies in some \(\mathcal{B}[x_i; \delta_{x_i}/2)\), so \(d(x, x_i) \lt \delta_{x_i}/2\). Then \(d(y, x_i) \le d(y, x) + d(x, x_i) \lt \delta + \delta_{x_i}/2 \le \delta_{x_i}\), placing \(y\) in \(\mathcal{B}[x_i; \delta_{x_i})\). Both \(f(x)\) and \(f(y)\) therefore lie in \(\mathcal{B}[f(x_i); \varepsilon/2)\), so \(d(f(x), f(y)) \lt \varepsilon\). This \(\delta\) depends only on \(\varepsilon\), proving uniform continuity.

Insight: Optimization Stability via Uniform Continuity

While the EVT guarantees that a solution exists, the Heine-Cantor Theorem gives our objective a uniform behavior. On a compact domain, the modulus of continuity \(\omega_f(\delta) = \sup\{|f(x) - f(y)| : d(x, y) \lt \delta\}\) tends to zero as \(\delta \to 0\), with a rate that does not vary across the domain.

A uniform modulus does not by itself bound the gradient. Uniform continuity is strictly weaker than Lipschitz continuity (\(f(x) = \sqrt{x}\) on \([0, 1]\) is uniformly continuous but not Lipschitz near \(x = 0\)). To obtain a bounded gradient one additionally applies the Extreme Value Theorem to \(\|\nabla f\|\) under a \(C^1\) assumption. The two pieces together, a uniform modulus plus a bounded gradient, give uniform control of \(f\) across the entire compact search space. The classical fixed step-size rule for gradient descent rests on one further property, a Lipschitz gradient. The same reasoning supplies it on a compact convex domain when \(f\) is \(C^2\), by applying the Extreme Value Theorem to the norm of the Hessian.