The Crisis of Infinite Dimensions
At the end of Dual Spaces & Riesz Representation,
we introduced the concept of reflexive spaces and hinted that reflexivity would play a
crucial role in guaranteeing the existence of optimal solutions. On this page, we make that promise
precise. But first, let us confront the problem head-on.
In Intro to Functional Analysis,
we proved a result that should have been deeply alarming (recalled here for reference):
Recall: Failure of Compactness in Infinite Dimensions
By the Compactness of the Unit Ball Theorem,
the closed unit ball \(\overline{B} = \{x \in \mathcal{X} : \|x\| \leq 1\}\) in a normed space
\(\mathcal{X}\) is compact if and only if \(\mathcal{X}\) is finite-dimensional.
In \(\mathbb{R}^n\), the
Heine-Borel Theorem tells
us that every closed and bounded set is compact. Combined with the
Extreme Value Theorem,
this guarantees that any continuous function on such a set attains its minimum, provided the set is
nonempty. This is the theoretical bedrock of finite-dimensional optimization.
But this guarantee vanishes in infinite-dimensional function spaces. These are exactly
the spaces where modern machine learning lives (parameter spaces of overparameterized networks,
Reproducing Kernel Hilbert Spaces, spaces of probability distributions). The unit ball is closed and
bounded, but not compact. Sequences can wander forever inside a bounded region without ever
converging.
The Optimization Nightmare
Let us make this concrete. Suppose we are searching for the best function \(f^*\) in an
infinite-dimensional Hilbert space \(\mathcal{H}\) (for example, an RKHS) that minimizes a loss
functional \(\mathcal{L} : \mathcal{H} \to \mathbb{R}\). Our training algorithm generates a sequence of
candidate models \(\{f_n\}_{n=1}^\infty\) with non-increasing losses:
\[
\mathcal{L}(f_1) \geq \mathcal{L}(f_2) \geq \mathcal{L}(f_3) \geq \cdots \to \inf_{f \in \mathcal{H}} \mathcal{L}(f).
\]
Even if we bound the parameters via regularization (\(\|f_n\| \leq R\)), the lack of compactness means
this sequence need not have a convergent subsequence in the norm topology. The infimum of the loss might
not be achieved by any actual function in the space.
A finite-dimensional analogy captures the flavor. Consider minimizing \(f(x) = x\) on the open interval
\((0, 1)\). The infimum is \(0\), but no \(x \in (0,1)\) achieves it. The minimum "escapes" through the
open boundary. The feasible set is bounded but not closed. It lacks compactness, so the Extreme Value
Theorem does not apply. In infinite dimensions, the pathology is far worse. Even with a closed
and bounded set, the minimum can still escape, because closed-and-bounded no longer implies compact.
A Concrete Escape: The Sliding Bump
Consider the standard orthonormal basis \(\{e_n\}_{n=1}^\infty\) in \(\ell^2\) (the space of
square-summable sequences). Every \(e_n\) lies on the unit sphere: \(\|e_n\| = 1\) for all \(n\). Yet for
\(m \neq n\), we have \(\|e_m - e_n\| = \sqrt{2}\). The sequence lives in the closed unit ball, which is
bounded and closed, but no subsequence converges in the norm topology. Each vector is "far" from every
other in the \(\sqrt{2}\) sense.
In a continuous function space like \(L^2([0,1])\), this same pathology takes a vivid form
known as a "concentrating bump" (or Dirac-like) sequence:
\[
f_n(x) = \sqrt{n}\,\mathbf{1}_{[0, 1/n]}(x).
\]
Each \(f_n\) has unit norm (\(\|f_n\|_{L^2} = 1\)), but the bumps become infinitely tall
and thin, concentrating at the origin. No subsequence converges in the \(L^2\) norm.
Weak Convergence: Convergence by Consensus
The resolution is elegant. Instead of asking "does \(\|x_n - x\| \to 0\)?", we ask a softer question:
"does every continuous linear functional agree that \(x_n\) is approaching \(x\)?" This is exactly where
the dual space \(\mathcal{X}^*\) enters the
story.
Definition: Strong (Norm) Convergence
A sequence \(\{x_n\}\) in a normed space \(\mathcal{X}\) converges strongly (or
in norm) to \(x \in \mathcal{X}\), written \(x_n \to x\), if
\[
\|x_n - x\| \to 0.
\]
Definition: Weak Convergence
A sequence \(\{x_n\}\) in a normed space \(\mathcal{X}\) converges weakly to
\(x \in \mathcal{X}\), written \(x_n \rightharpoonup x\), if
\[
\varphi(x_n) \to \varphi(x) \quad \text{for every } \varphi \in \mathcal{X}^*.
\]
Let us unpack this definition carefully. Each \(\varphi \in \mathcal{X}^*\) is a continuous linear
functional, a "measurement device" that extracts a single real (or complex) number from a vector. Weak
convergence says that no matter which measurement device we use, the readings \(\varphi(x_n)\)
eventually stabilize at \(\varphi(x)\). The sequence converges "by consensus of all observers."
Strong Implies Weak (but Not Conversely)
If \(x_n \to x\) strongly, then for every \(\varphi \in \mathcal{X}^*\):
\[
|\varphi(x_n) - \varphi(x)| = |\varphi(x_n - x)| \leq \|\varphi\|_{\mathcal{X}^*} \|x_n - x\| \to 0.
\]
Here we used the
boundedness of \(\varphi\)
(equivalently, its continuity). So strong convergence implies weak convergence. In finite dimensions, the
converse also holds (all topologies coincide on finite-dimensional spaces). But in infinite dimensions,
the converse is false, and this gap is precisely what makes weak convergence useful.
The Canonical Counterexample: Orthonormal Bases
Example: Weak Convergence to Zero in a Hilbert Space
Let \(\{e_n\}_{n=1}^\infty\) be an orthonormal basis for a separable Hilbert space
\(\mathcal{H}\).
We claim that \(e_n \rightharpoonup 0\) (weakly), even though \(\|e_n\| = 1\) for all \(n\)
(so \(e_n \not\to 0\) strongly).
By the
Riesz Representation Theorem,
every \(\varphi \in \mathcal{H}^*\) has the form \(\varphi(x) = \langle x, y_\varphi \rangle\) for a
unique \(y_\varphi \in \mathcal{H}\). Therefore:
\[
\varphi(e_n) = \langle e_n, y_\varphi \rangle.
\]
But \(\{\langle e_n, y_\varphi \rangle\}_{n=1}^\infty\) are the coordinates of \(y_\varphi\) with
respect to the orthonormal sequence \(\{e_n\}\), and by
Bessel's inequality:
\[
\sum_{n=1}^\infty |\langle e_n, y_\varphi \rangle|^2 \leq \|y_\varphi\|^2 \lt \infty.
\]
Convergence of the series forces its terms to zero: \(\langle e_n, y_\varphi \rangle \to 0\) for
every \(y_\varphi \in \mathcal{H}\). Since this holds for every \(\varphi \in \mathcal{H}^*\), we
conclude \(e_n \rightharpoonup 0\).
This is remarkable. The basis vectors \(e_n\) never get closer to the zero vector in norm
(\(\|e_n - 0\| = 1\) always), yet from the perspective of every measurement device,
they are fading to zero. Each \(e_n\) "points in a new direction" that no fixed functional
can permanently detect.
Key Properties of Weak Convergence
Weak convergence inherits some, but not all, of the properties of strong convergence. Let us record the
essential facts.
Theorem: Properties of Weak Convergence
Let \(\{x_n\}\) be a sequence in a normed space \(\mathcal{X}\).
(a) Uniqueness of weak limits. If \(x_n \rightharpoonup x\) and
\(x_n \rightharpoonup y\), then \(x = y\).
(b) Weak limits are bounded. If \(x_n \rightharpoonup x\), then
\(\sup_n \|x_n\| \lt \infty\) (by the Uniform Boundedness Principle).
(c) Norm lower semicontinuity. If \(x_n \rightharpoonup x\), then
\[
\|x\| \leq \liminf_{n \to \infty} \|x_n\|.
\]
(d) Mazur's Theorem. If \(x_n \rightharpoonup x\), then \(x\) lies in the
closed convex hull
of \(\{x_n\}\). That is, suitable convex combinations of the \(x_n\) converge to \(x\) strongly.
Proof:
(a) Uniqueness.
If \(x_n \rightharpoonup x\) and \(x_n \rightharpoonup y\), then for every
\(\varphi \in \mathcal{X}^*\), \(\varphi(x) = \lim \varphi(x_n) = \varphi(y)\), so
\(\varphi(x - y) = 0\). If \(x - y \neq 0\), the norming functional corollary of Hahn-Banach produces \(\varphi \in \mathcal{X}^*\) with \(\varphi(x - y) = \|x - y\| \neq 0\), contradicting the above. Hence \(x = y\).
(b) Boundedness.
A weakly convergent sequence is norm-bounded. Viewing each
\(x_n\) through the canonical embedding as a functional on the Banach space \(\mathcal{X}^*\), we see
that weak convergence makes the family pointwise bounded, and the uniform boundedness principle
converts this into a bound on \(\sup_n \|x_n\|\). This is proved in full as the statement that
weakly convergent sequences are bounded.
(c) Norm lower semicontinuity.
By the
norming functional corollary
of Hahn-Banach, for each nonzero \(x\) there exists \(\varphi \in \mathcal{X}^*\) with
\(\|\varphi\|_{\mathcal{X}^*} = 1\) and \(\varphi(x) = \|x\|\). For this \(\varphi\),
\[
\|x\| = |\varphi(x)| = \lim_{n \to \infty} |\varphi(x_n)|
\leq \liminf_{n \to \infty} \|\varphi\|_{\mathcal{X}^*} \|x_n\|
= \liminf_{n \to \infty} \|x_n\|.
\]
The first equality uses \(\varphi(x) = \|x\|\), a real positive value. The second uses
\(\varphi(x_n) \to \varphi(x)\) (weak convergence) together with continuity of \(|\cdot|\). The
inequality uses \(|\varphi(x_n)| \leq \|\varphi\|_{\mathcal{X}^*} \|x_n\|\) and
\(\|\varphi\|_{\mathcal{X}^*} = 1\). Finally, the case \(x = 0\) is trivial.
(d) Mazur's Theorem.
This is a consequence of the geometric Hahn-Banach theorem
(convex-set separation version, a distinct result from the extension theorem used for parts (a) and
(c) above).
The argument proceeds by contradiction. Suppose \(x\) lay outside the closed convex hull
\(C := \overline{\operatorname{co}}\{x_n\}\). Since \(C\) is closed and convex and \(x \notin C\),
the
strict separation of a point from a closed convex set
produces a functional \(\varphi \in \mathcal{X}^*\) and \(\gamma \in \mathbb{R}\) with
\(\operatorname{Re}\,\varphi(y) \leq \gamma \lt \operatorname{Re}\,\varphi(x)\) for all
\(y \in C\). In particular \(\operatorname{Re}\,\varphi(x_n) \leq \gamma\) for every \(n\), which
contradicts \(\varphi(x_n) \to \varphi(x)\) (weak convergence). Hence \(x \in C\). When
\(\mathbb{F} = \mathbb{C}\), the separation result is applied to \(\mathcal{X}\) regarded as a real
locally convex space, and the resulting real-linear functional \(f\) is turned into a member of
\(\mathcal{X}^*\) by \(\varphi(z) = f(z) - i f(iz)\), which leaves the argument unchanged.
Property (c) is especially important for optimization. It tells us that the norm can only
decrease (or stay the same) under weak limits and can never jump up. We will return to this
semicontinuity principle in the section on reflexivity, where it becomes the
key ingredient for proving that minima actually exist.
The Weak Topology
Weak convergence arises from a genuine topology on \(\mathcal{X}\), called the
weak topology \(\sigma(\mathcal{X}, \mathcal{X}^*)\). It is the
coarsest (that is, fewest open sets) topology on \(\mathcal{X}\) that makes every
functional \(\varphi \in \mathcal{X}^*\) continuous.
This topology admits a concrete description through
seminorms. The weak topology
is generated by the family \(\{\, p_\varphi : \varphi \in \mathcal{X}^* \,\}\), where each
\(p_\varphi(x) = |\varphi(x)|\) is the seminorm measuring a vector through a single functional. The
definitions below record this for the weak topology and its variants.
Definition: Weak Topology \(\sigma(\mathcal{X}, \mathcal{X}^*)\)
The weak topology on a normed space \(\mathcal{X}\) is the coarsest topology
making every functional \(\varphi \in \mathcal{X}^*\) continuous (as a map
\(\mathcal{X} \to \mathbb{F}\)). Equivalently, it is the topology generated by the family of seminorms
\[
p_\varphi(x) = |\varphi(x)|, \quad \varphi \in \mathcal{X}^*.
\]
A net
\(x_\alpha \to x\) converges in the weak topology if and only if
\(\varphi(x_\alpha) \to \varphi(x)\) for all \(\varphi \in \mathcal{X}^*\).
Because the weak topology has fewer open sets than the norm topology, it has more
compact sets, which is exactly what we need.
Connection to Machine Learning: Regularization as Weak Compactness
When we impose \(L^2\) regularization (\(\|\boldsymbol{\theta}\|_2 \leq R\)) during training, we are
confining our parameter sequence to a bounded set. In finite dimensions, the
Heine-Borel Theorem
guarantees that this set is compact (in the strong topology), so the Extreme Value Theorem applies
directly. In infinite dimensions, the same bound does not yield strong compactness, but it
does yield weak compactness in reflexive spaces (as we will see in
the reflexivity section). The weak topology is precisely the machinery
that lets regularization "work" in function spaces.
The Weak* Topology: Convergence of Observers
So far, we have weakened convergence for elements of \(\mathcal{X}\) by using
functionals from \(\mathcal{X}^*\). Now we flip the perspective: what does it mean for
a sequence of functionals to converge?
The dual space \(\mathcal{X}^*\) is itself a Banach space (as established in
Bounded Linear Operators), so it carries
a norm topology (strong convergence of functionals) and a weak topology (using functionals on
\(\mathcal{X}^*\), that is, elements of \(\mathcal{X}^{**}\)). But a third topology, often more natural,
uses only the original space \(\mathcal{X}\) as the set of "test objects."
Definition: Weak* (Weak-Star) Convergence
A sequence \(\{\varphi_n\}\) in \(\mathcal{X}^*\) converges in the weak* topology to
\(\varphi \in \mathcal{X}^*\), written \(\varphi_n \xrightarrow{w^*} \varphi\), if
\[
\varphi_n(x) \to \varphi(x) \quad \text{for every } x \in \mathcal{X}.
\]
Observe the asymmetry. Weak convergence tests elements of \(\mathcal{X}\) against all of
\(\mathcal{X}^*\), while weak* convergence tests elements of \(\mathcal{X}^*\) against only the
evaluation functionals from \(\mathcal{X}\) (via the
canonical embedding
\(J: \mathcal{X} \to \mathcal{X}^{**}\)), rather than against all of \(\mathcal{X}^{**}\).
Definition: Weak* Topology \(\sigma(\mathcal{X}^*, \mathcal{X})\)
The weak* topology on \(\mathcal{X}^*\) is the coarsest topology
making the evaluation maps
\[
\operatorname{ev}_x : \mathcal{X}^* \to \mathbb{F}, \quad \operatorname{ev}_x(\varphi) = \varphi(x)
\]
continuous for every \(x \in \mathcal{X}\). It is generated by the family of seminorms
\[
p_x(\varphi) = |\varphi(x)|, \quad x \in \mathcal{X}.
\]
The Hierarchy of Topologies in Duality
We have now seen weakened topologies on two different spaces: the weak topology
\(\sigma(\mathcal{X}, \mathcal{X}^*)\) on \(\mathcal{X}\) itself, and the weak* topology
\(\sigma(\mathcal{X}^*, \mathcal{X})\) on the dual \(\mathcal{X}^*\). Because \(\mathcal{X}^*\) is itself
a Banach space, it also carries its own weak topology, generated by all functionals on \(\mathcal{X}^*\),
that is, by all of \(\mathcal{X}^{**}\).
Definition: Weak Topology on \(\mathcal{X}^*\)
The weak topology on \(\mathcal{X}^*\), denoted
\(\sigma(\mathcal{X}^*, \mathcal{X}^{**})\), is the coarsest topology making every
\(\Psi \in \mathcal{X}^{**}\) continuous as a map \(\mathcal{X}^* \to \mathbb{F}\).
It is generated by the family of seminorms
\[
p_\Psi(\varphi) = |\Psi(\varphi)|, \quad \Psi \in \mathcal{X}^{**}.
\]
Combining all of these, we obtain a hierarchy of topologies organized by the space they
live on and the set of test functionals they use. Recall the canonical embedding
\(J: \mathcal{X} \hookrightarrow \mathcal{X}^{**}\), \(J(x)(\varphi) = \varphi(x)\),
which identifies \(\mathcal{X}\) with a subspace of \(\mathcal{X}^{**}\):
| Topology |
Space |
Notation |
Convergence means... |
Test functionals |
| Norm (Strong) |
\(\mathcal{X}\) |
Norm topology from \(\|\cdot\|_{\mathcal{X}}\) |
\(\|x_n - x\|_{\mathcal{X}} \to 0\) |
None (uniform criterion) |
| Weak |
\(\mathcal{X}\) |
\(\sigma(\mathcal{X}, \mathcal{X}^*)\) |
\(\varphi(x_n) \to \varphi(x)\) for all \(\varphi \in \mathcal{X}^*\) |
All of \(\mathcal{X}^*\) |
| Norm (Strong) |
\(\mathcal{X}^*\) |
Norm topology from \(\|\cdot\|_{\mathcal{X}^*}\) |
\(\|\varphi_n - \varphi\|_{\mathcal{X}^*} \to 0\) |
None (uniform criterion) |
| Weak |
\(\mathcal{X}^*\) |
\(\sigma(\mathcal{X}^*, \mathcal{X}^{**})\) |
\(\Psi(\varphi_n) \to \Psi(\varphi)\) for all \(\Psi \in \mathcal{X}^{**}\) |
All of \(\mathcal{X}^{**}\) |
| Weak* |
\(\mathcal{X}^*\) |
\(\sigma(\mathcal{X}^*, \mathcal{X})\) |
\(\varphi_n(x) \to \varphi(x)\) for all \(x \in \mathcal{X}\) |
Only \(J(\mathcal{X}) \subseteq \mathcal{X}^{**}\) |
Within each space, the topologies are nested from finer to coarser (a finer topology has more open sets,
hence the inclusion runs in the direction shown):
\[
\begin{align*}
\text{On } \mathcal{X}: \quad & \text{Norm topology} \supseteq \text{Weak topology}, \\\\
\text{On } \mathcal{X}^*: \quad & \text{Norm topology} \supseteq \text{Weak topology} \supseteq \text{Weak* topology}.
\end{align*}
\]
Equivalently, in terms of sequential convergence, norm convergence implies weak convergence on
\(\mathcal{X}\). On \(\mathcal{X}^*\), norm convergence implies weak convergence, which in turn implies
weak* convergence. The inclusion \(\text{Norm} \supseteq \text{Weak}\) on \(\mathcal{X}\) is strict in
every infinite-dimensional normed space, and the orthonormal basis example in \(\ell^2\) above exhibits
the gap at the level of sequences as well. The same strictness holds for the norm and weak topologies
on \(\mathcal{X}^*\) whenever \(\mathcal{X}^*\) is itself infinite-dimensional.
The implication \(\text{Weak} \Rightarrow \text{Weak*}\) on \(\mathcal{X}^*\) is strict precisely when
\(\mathcal{X}\) is not reflexive. In reflexive spaces, \(J(\mathcal{X}) = \mathcal{X}^{**}\), so
the two topologies coincide (as we establish below). The weak* topology is the "gentlest" notion of
convergence, since it asks the least of a sequence of functionals. Consequently it has the
best compactness properties, a fact that powers the Banach-Alaoglu theorem.
When Do Weak and Weak* Coincide?
The weak topology on \(\mathcal{X}^*\) uses all of \(\mathcal{X}^{**}\) as test functionals, while the
weak* topology uses only the image \(J(\mathcal{X}) \subseteq \mathcal{X}^{**}\). These two topologies
coincide if and only if \(J(\mathcal{X}) = \mathcal{X}^{**}\), that is, if and only if \(\mathcal{X}\) is
reflexive.
Proposition: Weak = Weak* in Reflexive Spaces
If \(\mathcal{X}\) is a reflexive Banach space, then the weak topology and the weak*
topology on \(\mathcal{X}^*\) coincide:
\[
\sigma(\mathcal{X}^*, \mathcal{X}^{**}) = \sigma(\mathcal{X}^*, \mathcal{X}).
\]
Proof:
By the general construction of weak topologies, both the weak topology
\(\sigma(\mathcal{X}^*, \mathcal{X}^{**})\) and the weak* topology
\(\sigma(\mathcal{X}^*, \mathcal{X})\) are defined as the coarsest topologies on \(\mathcal{X}^*\)
making a specified family of evaluation maps continuous. The former uses the family
\(\{\Psi : \Psi \in \mathcal{X}^{**}\}\), while the latter uses
\(\{J(x) : x \in \mathcal{X}\} \subseteq \mathcal{X}^{**}\), where \(J\) is the
canonical embedding.
Reflexivity means precisely that \(J(\mathcal{X}) = \mathcal{X}^{**}\), so the two families of test
maps coincide, and the induced topologies coincide as well.
Intuition: Strong vs. Weak vs. Weak*
Imagine evaluating a sequence of classifiers \(\{h_n\}\) in a hypothesis space.
Strong convergence (\(h_n \to h\)) demands that \(h_n\) and \(h\) agree "uniformly"
over the entire input domain. The worst-case discrepancy shrinks to zero.
Weak convergence (\(h_n \rightharpoonup h\)) demands only that every "test" (linear
functional) applied to \(h_n\) stabilizes. This is like requiring convergence of every moment of a
distribution, without requiring the PDFs to converge pointwise.
Weak* convergence (\(\varphi_n \xrightarrow{w^*} \varphi\)) is even gentler. The
functionals need only agree when evaluated at each fixed point. It is "pointwise convergence for
linear functionals."
The Banach-Alaoglu Theorem
We now arrive at the central result of this page, the theorem that recovers compactness in infinite
dimensions by leveraging the weak* topology.
Theorem: Banach-Alaoglu
Let \(\mathcal{X}\) be a normed space. The closed unit ball of the dual space,
\[
\overline{B}_{\mathcal{X}^*} = \{\varphi \in \mathcal{X}^* : \|\varphi\|_{\mathcal{X}^*} \leq 1\},
\]
is compact in the weak* topology \(\sigma(\mathcal{X}^*, \mathcal{X})\).
Banach-Alaoglu is the Heine-Borel theorem of infinite dimensions.
We traded topological "tightness" (the norm) for topological "gentleness" (weak*),
and in return, we recovered the compactness that infinite dimensions had stolen from us.
Proof (via Tychonoff's Theorem)
The proof is a beautiful application of point-set topology, specifically
Tychonoff's Theorem.
This is one of the rare moments where abstract topology directly solves a concrete problem in analysis.
Proof:
Step 1: Embed into a product space.
For each \(x \in \mathcal{X}\), a functional
\(\varphi \in \overline{B}_{\mathcal{X}^*}\) satisfies \(|\varphi(x)| \leq \|x\|\). Let
\[
D_x = \{z \in \mathbb{F} : |z| \leq \|x\|\}.
\]
This set is a compact interval \([-\|x\|, \|x\|] \subseteq \mathbb{R}\) when
\(\mathbb{F} = \mathbb{R}\), or a compact closed disk in \(\mathbb{C}\) when
\(\mathbb{F} = \mathbb{C}\).
Identifying each \(\varphi\) with the function \(x \mapsto \varphi(x)\), we view
\(\overline{B}_{\mathcal{X}^*}\) as a subset of the product space
\[
\prod_{x \in \mathcal{X}} D_x \subseteq \mathbb{F}^{\mathcal{X}}.
\]
Step 2: Apply Tychonoff.
By Tychonoff's Theorem, the product \(\prod_{x \in \mathcal{X}} D_x\) is compact in the
product topology, since each factor \(D_x\) is compact in \(\mathbb{F}\).
Step 3: Identify the weak* topology with the subspace topology.
Under the
identification of Step 1, the evaluation map \(\operatorname{ev}_x : \mathcal{X}^* \to \mathbb{F}\),
\(\varphi \mapsto \varphi(x)\) coincides with the restriction of the projection
\(\pi_x : \mathbb{F}^{\mathcal{X}} \to \mathbb{F}\), \(f \mapsto f(x)\).
The weak* topology is defined as the coarsest topology making each \(\operatorname{ev}_x\)
continuous, and the subspace topology inherited from the product topology on
\(\mathbb{F}^{\mathcal{X}}\) is the coarsest topology making each \(\pi_x\) continuous. Both are
therefore generated by the same family of test maps, and so they coincide. Restricting both
topologies to the subset \(\overline{B}_{\mathcal{X}^*}\) preserves this coincidence, so the weak*
topology on \(\overline{B}_{\mathcal{X}^*}\) (inherited from \(\mathcal{X}^*\)) agrees with the
topology inherited from \(\prod_x D_x\).
Step 4: Show \(\overline{B}_{\mathcal{X}^*}\) is closed in \(\prod_x D_x\).
Under the embedding of Step 1, a point \(\varphi \in \prod_x D_x\) lies in (the image of)
\(\overline{B}_{\mathcal{X}^*}\) if and only if it is linear:
- (\(\Longrightarrow\)) Elements of \(\overline{B}_{\mathcal{X}^*}\) are linear functionals by definition.
- (\(\Longleftarrow\)) Conversely, any linear \(\varphi \in \prod_x D_x\) already satisfies
\(|\varphi(x)| \leq \|x\|\) for all \(x\) (by membership in \(\prod_x D_x\)),
hence \(\|\varphi\|_{\mathcal{X}^*} \leq 1\) and \(\varphi \in \overline{B}_{\mathcal{X}^*}\).
It therefore suffices to show that the set of linear functionals in \(\prod_x D_x\) is closed. For
each fixed \(x, y \in \mathcal{X}\) and \(\alpha, \beta \in \mathbb{F}\), the linearity condition
\[
\varphi(\alpha x + \beta y) - \alpha \varphi(x) - \beta \varphi(y) = 0
\]
defines a closed subset of \(\mathbb{F}^{\mathcal{X}}\). The map
\(\varphi \mapsto \varphi(\alpha x + \beta y) - \alpha \varphi(x) - \beta \varphi(y)\) is continuous
in the product topology, since it depends on only three coordinates, namely \(\alpha x + \beta y\),
\(x\), and \(y\). The set in question is the preimage of \(\{0\} \subseteq \mathbb{F}\), which is
closed.
The image of \(\overline{B}_{\mathcal{X}^*}\) is the intersection over all such
\((x, y, \alpha, \beta)\) of these closed sets, hence closed in \(\prod_x D_x\). A closed subset of a
compact space is compact, so \(\overline{B}_{\mathcal{X}^*}\) is weak*-compact.
The elegance of this proof lies in the conceptual shift. We reinterpret a functional
\(\varphi : \mathcal{X} \to \mathbb{F}\) as a single point in the product space
\(\mathbb{F}^{\mathcal{X}}\), where each "coordinate" is the value \(\varphi(x)\). The constraint
\(\|\varphi\| \leq 1\) confines this point to a product of compact factors (intervals when
\(\mathbb{F} = \mathbb{R}\), disks when \(\mathbb{F} = \mathbb{C}\)), and Tychonoff's theorem does the
rest.
Generalization: Arbitrary Radius
By a simple rescaling argument, the Banach-Alaoglu theorem extends to balls of any radius:
Corollary: Compactness of Arbitrary Dual Balls
For any \(R \gt 0\), the closed ball
\(\{\varphi \in \mathcal{X}^* : \|\varphi\|_{\mathcal{X}^*} \leq R\}\) is weak*-compact.
A Critical Caveat: Metrizability
The weak* topology on \(\overline{B}_{\mathcal{X}^*}\) is metrizable (that is, describable by a metric)
if and only if \(\mathcal{X}\) is separable (has a countable dense subset). When
\(\mathcal{X}\) is separable, compactness and sequential compactness coincide, so we can extract
weak*-convergent subsequences, which is the form most useful in applications. The spaces
\(L^p\) over a domain in \(\mathbb{R}^d\) with Lebesgue measure, and the sequence spaces \(\ell^p\),
are separable for \(1 \leq p \lt \infty\), so this condition is satisfied in most ML-relevant settings.
Connection to Probability & GANs: Convergence of Distributions
In the theory of Generative Adversarial Networks, the generator produces a sequence of probability
distributions \(\{\mu_n\}\) aiming to match a target distribution \(\mu\). Distributions can be
viewed as continuous linear functionals on spaces of continuous functions (via integration:
\(\mu(f) = \int f \, d\mu\)). The Banach-Alaoglu theorem, applied to the
dual of \(C_0(\mathbb{R}^d)\)
(continuous functions vanishing at infinity), guarantees that any sequence of finite signed measures
with bounded total variation has a weak*-convergent subsequence.
This guarantee is one of the mathematical foundations of
weak convergence of measures. Additional conditions (such as tightness,
captured by Prokhorov's theorem) are still needed to ensure that the limit remains a probability
distribution rather than losing mass at infinity. Banach-Alaoglu thus provides the existence of
subsequential limits. The analytical refinements needed for full convergence theory in machine
learning belong to measure theory and are developed in the probability section.
Reflexivity & The Existential Guarantee
The Banach-Alaoglu theorem gives us compactness in the dual space \(\mathcal{X}^*\)
under the weak* topology. But in many optimization problems, we want compactness in the
original space \(\mathcal{X}\). This is where reflexivity becomes decisive.
From Banach-Alaoglu to Weak Compactness of \(\mathcal{X}\)
Suppose \(\mathcal{X}\) is reflexive, so the
canonical embedding
\(J: \mathcal{X} \to \mathcal{X}^{**}\) is a surjective isometric isomorphism. Apply the Banach-Alaoglu
theorem to the Banach space \(\mathcal{X}^*\). The closed unit ball of
\((\mathcal{X}^*)^* = \mathcal{X}^{**}\) is compact in the weak* topology
\(\sigma(\mathcal{X}^{**}, \mathcal{X}^*)\).
Under \(J\), the closed unit ball \(\overline{B}_{\mathcal{X}}\) corresponds to
\(\overline{B}_{\mathcal{X}^{**}}\) (since \(J\) is a surjective isometry), and the weak topology on
\(\mathcal{X}\) corresponds to the weak* topology on \(\mathcal{X}^{**}\) (since the test functionals in
\(\mathcal{X}^*\) act on \(\mathcal{X}\) and on \(\mathcal{X}^{**}\) by the same formula
\(\varphi(x) = J(x)(\varphi)\)). Transferring the conclusion through this isomorphism gives weak
compactness of \(\overline{B}_{\mathcal{X}}\).
Theorem: Weak Compactness of the Unit Ball (Reflexive Case)
Let \(\mathcal{X}\) be a reflexive Banach space. Then the closed unit ball
\[
\overline{B}_{\mathcal{X}} = \{x \in \mathcal{X} : \|x\| \leq 1\}
\]
is compact in the weak topology \(\sigma(\mathcal{X}, \mathcal{X}^*)\).
Moreover, the converse holds. If \(\overline{B}_{\mathcal{X}}\) is weakly compact, then
\(\mathcal{X}\) is reflexive.
In other words, reflexivity is equivalent to weak compactness of the unit ball. The
spaces that matter most for machine learning are reflexive. They include all Hilbert spaces (by the
Riesz Representation Theorem,
\(\mathcal{H} \cong \mathcal{H}^*\)) and \(L^p\) for \(1 \lt p \lt \infty\).
From Compactness to Subsequences: Eberlein-Šmulian
Weak compactness of \(\overline{B}_{\mathcal{X}}\) gives us, in principle, a weak topological notion of
compactness. But in applications, particularly optimization, we need to extract subsequences
from bounded sequences. In a general topological space, compactness (every open cover has a finite
subcover) and sequential compactness (every sequence has a convergent subsequence) are not
equivalent. The following theorem is one of the most remarkable results in Banach space theory. For the
weak topology, they are equivalent.
Theorem: Eberlein-Šmulian
Let \(\mathcal{X}\) be a Banach space and \(K \subseteq \mathcal{X}\) a weakly closed subset.
The following are equivalent:
- \(K\) is weakly compact (compact in the weak topology
\(\sigma(\mathcal{X}, \mathcal{X}^*)\)).
- \(K\) is weakly sequentially compact (every sequence in \(K\)
has a weakly convergent subsequence with limit in \(K\)).
In particular, in a reflexive Banach space, every bounded sequence \(\{x_n\}\) has a weakly
convergent subsequence. If \(\|x_n\| \leq R\) for all \(n\), then the weak limit \(x_*\) satisfies
\(\|x_*\| \leq R\) as well.
The proof is substantial and relies on a delicate interplay between separability considerations and the
structure of the bidual. What makes Eberlein-Šmulian remarkable is that it holds in
arbitrary Banach spaces, with no separability or metrizability assumption. In the weak topology,
compactness and sequential compactness are generally distinct notions, yet they coincide here due to the
specific geometry of Banach spaces.
Weak Lower Semicontinuity: The Missing Piece
Compactness alone guarantees convergent subsequences, but to conclude that a loss functional attains its
minimum, we need one more ingredient. The loss must "behave well" under weak limits.
Definition: Weakly Lower Semicontinuous (w-lsc) Functional
A functional \(\mathcal{L}: \mathcal{X} \to \mathbb{R} \cup \{+\infty\}\) is
weakly lower semicontinuous if whenever \(x_n \rightharpoonup x\),
\[
\mathcal{L}(x) \leq \liminf_{n \to \infty} \mathcal{L}(x_n).
\]
In words, the functional value at the weak limit is no worse than the best asymptotic value along the
sequence. The loss cannot "jump up" when we pass to weak limits.
The following classical result gives us a large supply of weakly lower semicontinuous functionals:
Proposition: Convex + Strongly lsc \(\Longrightarrow\) Weakly lsc
Let \(\mathcal{X}\) be a normed space and \(\mathcal{L}: \mathcal{X} \to \mathbb{R} \cup \{+\infty\}\).
If \(\mathcal{L}\) is convex and lower semicontinuous in the
norm topology, then \(\mathcal{L}\) is weakly lower semicontinuous.
This is a consequence of
Mazur's Theorem,
or more precisely of its
standard equivalent formulation: a convex norm-closed set is also weakly closed, which follows
from the
representation of closed convex sets as intersections of half-spaces.
Since \(\mathcal{L}\) is convex and norm-lsc, each sublevel set \(\{x : \mathcal{L}(x) \leq \alpha\}\) is
convex and norm-closed. By Mazur's Theorem, each is therefore weakly closed. A functional whose sublevel
sets are all closed in a given topology is lower semicontinuous for that topology, and hence along
sequences converging in it, so \(\mathcal{L}\) is weakly lsc. In particular, the norm itself is always
weakly lower semicontinuous (which we already recorded as property (c)).
The Fundamental Theorem of Optimization in Reflexive Spaces
We can now assemble the full argument. This is the theorem that makes
regularization rigorous.
Theorem: Existence of Minimizers
Let \(\mathcal{H}\) be a Hilbert space (or more generally, a reflexive Banach space), let
\(\Omega = \{f \in \mathcal{H} : \|f\| \leq R\}\) for some \(R \gt 0\), and let
\(\mathcal{L}: \mathcal{H} \to \mathbb{R}\) be a weakly lower semicontinuous
functional. Then \(\mathcal{L}\) attains its minimum on \(\Omega\): there exists \(f^* \in \Omega\)
such that
\[
\mathcal{L}(f^*) = \inf_{f \in \Omega} \mathcal{L}(f).
\]
Proof:
Let \(m = \inf_{f \in \Omega} \mathcal{L}(f)\) and choose a minimizing sequence
\(\{f_n\} \subseteq \Omega\) with \(\mathcal{L}(f_n) \to m\).
Since \(\mathcal{H}\) is reflexive, we may invoke the
weak compactness of the unit ball
established above.
By a linear rescaling \(f \mapsto f/R\), the closed ball \(\Omega\) of any radius \(R \gt 0\) is weakly
compact as well. Being convex and norm-closed, \(\Omega\) is also weakly closed, and the
Eberlein-Šmulian theorem
then identifies weak compactness of \(\Omega\) with weak sequential compactness, so the minimizing
sequence \(\{f_n\} \subseteq \Omega\) has a weakly convergent subsequence
\(f_{n_k} \rightharpoonup f^* \in \Omega\).
By weak lower semicontinuity:
\[
\mathcal{L}(f^*) \leq \liminf_{k \to \infty} \mathcal{L}(f_{n_k}) = m.
\]
But \(f^* \in \Omega\), so \(\mathcal{L}(f^*) \geq m\) by definition of infimum. Hence
\(\mathcal{L}(f^*) = m\).
Let us record what each ingredient contributed:
- Reflexivity (from Banach-Alaoglu + \(J\) surjective):
Provides weak compactness of \(\Omega\).
- Eberlein-Šmulian theorem:
Upgrades that compactness to the sequential form, which is what a minimizing sequence needs,
and keeps the limit inside \(\Omega\).
- Weak lower semicontinuity of \(\mathcal{L}\):
Ensures the limit point actually achieves the infimal loss value.
Application: Why Weight Decay Works
Connection to Deep Learning: \(L^2\) Regularization and Weight Decay
In practice, training a model with \(L^2\) regularization amounts to minimizing
\[
\mathcal{L}_\lambda(f) = \mathcal{L}_{\text{data}}(f) + \lambda \|f\|^2
\]
over a Hilbert space \(\mathcal{H}\) (for example, an RKHS for kernel methods, or the function space
in which overparameterized networks are analyzed in the infinite-width limit).
The regularization term \(\lambda \|f\|^2\) is convex and strongly continuous, hence weakly lower
semicontinuous. If \(\mathcal{L}_{\text{data}}\) is also convex and continuous (as in linear/kernel
regression, SVMs, and logistic regression), then the total loss \(\mathcal{L}_\lambda\) is weakly
lsc. Moreover, the penalty \(\lambda \|f\|^2\) is coercive (it grows to \(\infty\)
as \(\|f\| \to \infty\)), so any minimizer must lie in a bounded sublevel set, which is contained
in a closed ball \(\Omega\). Reflexivity of \(\mathcal{H}\) then gives the
weak compactness of the unit ball,
and hence of \(\Omega\) after rescaling.
The
existence theorem
above then tells us that the optimal model exists. This is a
rigorous mathematical theorem, not merely an engineering heuristic.
The Non-Reflexive Pitfall: \(L^1\) Regularization
As noted in Dual Spaces, the space \(L^1\) is
not reflexive (nor is its sequence-space counterpart \(\ell^1\), which underlies
Lasso-type penalties on
parameters). This means the closed unit ball of \(L^1\) is not weakly compact, and the argument
above breaks down. In practice, \(L^1\)-regularized problems in infinite-dimensional settings may lack
minimizers entirely, or require additional structural assumptions (for example, finite-dimensionality of
the effective parameter space) to guarantee solutions. This is one of the deep mathematical reasons why
\(L^2\) regularization is better-behaved than \(L^1\) in function-space settings.
Looking Ahead: From Existence to Structure
We have now closed the "existence gap" in infinite-dimensional optimization. The chain of reasoning is:
- Banach & Hilbert Spaces:
Built the spaces where models and functions live.
- Bounded Operators:
Understood the well-behaved maps between these spaces.
- Dual Spaces & Riesz:
Introduced measurement tools (functionals) and the duality machinery.
- This page: Used duality to weaken the topology, recovering compactness and proving
existence of optimal solutions.
In the pages ahead, the emphasis shifts from existence to structure. We ask not merely "does an
optimal solution exist?" but "what does it look like?" The Spectral Theorem will decompose operators into
their fundamental modes, the infinite-dimensional generalization of eigendecomposition. This
decomposition provides the structural insights that power Kernel Methods, PCA, and the Graph Laplacian.