What We Have Seen So Far...
We have encountered various "spaces" throughout our study. In linear algebra, we started with
vector spaces. To discuss "length," we introduced
normed vector spaces, and to formalize "orthogonality," we
considered inner product spaces. Strictly speaking, our familiar space \(\mathbb{R}^n\) is categorized
as a Hilbert space, which is an inner product space that guarantees completeness.
Completeness is critical in calculus and analysis because, without it, we cannot guarantee
the convergence of limits "within" the space. Moreover, we frequently encounter the \(L^2\) space in
machine learning contexts. What exactly is it? The \(L^2\) space is also a Hilbert space, but unlike \(\mathbb{R}^n\), whose
elements are finite-dimensional vectors, it is a function space whose elements are functions.
Furthermore, we have explored \(L^1\) and \(L^\infty\). These are categorized as Banach spaces, that is,
complete normed vector spaces that do not necessarily arise from an inner product. Thus, a Banach space is a more
general concept than a Hilbert space, yet it is essential in both analysis and machine learning.
For instance, in deep learning architectures, a neural network can
be viewed as a point (a function) within a vast function space, and we build highly complex models via topological connections.
While it is commonly stated in engineering that training works as long as the functions are differentiable (which provides the
local gradient), mathematically, differentiability only provides the direction. For every Cauchy sequence of optimized models
to be guaranteed a valid limit without "falling through a hole," the underlying space must be complete. Whether an
optimum exists is a separate question, and that is where further properties such as
compactness or
convexity enter.
Fundamentally, these function spaces are defined by their properties regarding integration. Integrability
conditions (and, for \(L^\infty\), essential boundedness) dictate which functions qualify to be part of spaces like
\(L^1, L^2,\) and \(L^\infty\). This is precisely why we studied
Lebesgue integration. That theory ensures these function spaces possess
completeness, and it rests on measures as its foundation. A vector space and a
measure space must not be confused. A measure space possesses no algebraic operations such as vector addition
or scalar multiplication. Instead, it provides the underlying domain upon which functions are defined.
Viewed through this lens, core concepts in statistics gain clear mathematical structure. We can now rigorously understand that
random variables are measurable functions that map outcomes from a sample space to real numbers, and that a
probability space is a measure space of total measure 1
used to describe data distributions. Consequently, the expected value \(\mathbb{E}[X]\) is a Lebesgue integral over this
probability space. For \(X \in L^2\), the variance \(\mathbb{V}[X]\) is the squared \(L^2\) norm of the random variable
centered around its mean.
From abstract algebra, we know that \(\mathbb{R}\) is a
field. Simultaneously, from the perspective of
functional analysis, \(\mathbb{R}\) itself acts as a one-dimensional Hilbert space. Thus, \(\mathbb{R}\)
serves both as the foundational "rule" for arithmetic operations and as the underlying "stage" (codomain) for our functions.
A field equipped with a topology that makes its algebraic operations continuous is known as a topological field.
In particular, \(\mathbb{R}\) and \(\mathbb{C}\) are complete topological fields. We have relied on this
since elementary school without naming it (for comparison, the field of rational numbers \(\mathbb{Q}\) is a topological field, but it is not complete).
A final note on notation is in order. In applied mathematics and machine learning contexts, \(L_1, L_2,\) and \(L_\infty\) are
frequently written with subscripts instead of superscripts (\(L^1, L^2, L^\infty\)). This variation stems from differing
conventional focuses across disciplines, but both notations refer to exactly the same mathematical concepts and spaces.
Normed Spaces & Banach Spaces
To perform calculus or optimization on a vector space, we need a notion of "distance" to define limits and convergence.
A norm provides this by measuring the "length" of a vector.
Crucially, any norm naturally induces a metric (distance function)
defined by \(d(x, y) = \|x - y\|\). Once we have a metric, we can define
Cauchy sequences and ask whether the space is
complete.
Definition: Normed Space (Normed Vector Space)
A normed space is a pair \((\mathcal{X}, \|\cdot\|)\), where \(\mathcal{X}\) is a vector space
over \(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\) and \(\|\cdot\|: \mathcal{X} \to [0, \infty)\) is a
norm, meaning that for all \(x, y \in \mathcal{X}\) and \(\alpha \in \mathbb{F}\):
- (Positive definiteness) \(\|x\| = 0\) if and only if \(x = 0\).
- (Homogeneity) \(\|\alpha x\| = |\alpha|\,\|x\|\).
- (Triangle inequality) \(\|x + y\| \leq \|x\| + \|y\|\).
Definition: Banach Space
A Banach space is a normed space that is complete with respect to the
metric defined by the norm.
Equivalently, by the
Cauchy criterion for completeness,
every Cauchy sequence in the space converges to a limit that is also within the space.
The most important examples of Banach spaces in analysis and machine learning are the \(L^p\) spaces.
Thanks to the rigorous foundation of Lebesgue integration, these spaces of functions are guaranteed to be complete.
Definition: \(L^p\) Spaces
For \(1 \leq p \lt \infty\) and a measure space \((\Omega, \mathcal{F}, \mu)\), the space \(L^p(\Omega, \mu)\) consists of
(equivalence classes, modulo a.e. equality, of) measurable functions \(f\) for which \(\int_\Omega |f|^p \, d\mu \lt \infty\).
The norm is defined as:
\[
\|f\|_p = \left( \int_\Omega |f(x)|^p \, d\mu \right)^{1/p}.
\]
The triangle inequality is Minkowski's inequality, and it is what makes \(\|\cdot\|_p\) a norm. Both this inequality and the
completeness of \(L^p\) are established when these spaces are constructed in full.
Definition: \(L^\infty\) Space
The space \(L^\infty(\Omega, \mu)\) consists of (equivalence classes, modulo a.e. equality, of)
measurable functions \(f\) that are essentially bounded. The norm is the
essential supremum:
\[
\|f\|_\infty = \operatorname{ess\,sup}_{x} |f(x)| = \inf \{ C \geq 0 : |f(x)| \leq C \text{ a.e.} \}.
\]
Inner Product Spaces & Hilbert Spaces
While a general Banach space allows us to measure lengths and distances, its norm need not come from an inner product, so it
may lack the geometric concepts of "angles" and "orthogonality" that Hilbert spaces provide.
To recover the rich geometry of Euclidean space in abstract settings, we need an
inner product. Every inner product naturally induces a
candidate norm via \(\|x\| := \sqrt{\langle x, x \rangle}\). The candidate is indeed a norm, and in particular satisfies the
triangle inequality, by the Cauchy-Schwarz inequality proved below. Hence every inner product space is automatically a normed
space.
Definition: Inner Product Space
An inner product space is a vector space \(\mathcal{X}\) over \(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\)
equipped with an inner product \(\langle \cdot, \cdot \rangle: \mathcal{X} \times \mathcal{X} \to \mathbb{F}\)
satisfying, for all \(x, y, z \in \mathcal{X}\) and \(\alpha, \beta \in \mathbb{F}\):
- (Conjugate symmetry) \(\langle x, y \rangle = \overline{\langle y, x \rangle}\).
- (Linearity in the first argument) \(\langle \alpha x + \beta y, z \rangle = \alpha \langle x, z \rangle + \beta \langle y, z \rangle\).
- (Positive definiteness) \(\langle x, x \rangle \geq 0\), with equality if and only if \(x = 0\).
We follow the mathematical convention of linearity in the first argument. The physics/quantum mechanics convention places
linearity in the second argument, which conjugates the first.
Just as a normed space requires completeness to become a Banach space, an inner product space requires completeness
to become the ultimate setting for functional analysis: a Hilbert space.
Definition: Hilbert Space
A Hilbert space \(\mathcal{H}\) is an inner product space that is complete
with respect to the metric \(d(x, y) = \|x - y\|\), where \(\|x\| := \sqrt{\langle x, x \rangle}\) is
the norm induced by the inner product (see the Cauchy-Schwarz inequality below for the proof that this
is a norm).
Hilbert spaces retain more of the geometric structure of \(\mathbb{R}^n\) than general Banach spaces do, and that extra structure
makes them the most tractable infinite-dimensional spaces. What survives includes orthogonality, projection, and self-duality. The
foundational inequality that connects the inner product to the norm is the Cauchy-Schwarz inequality, which is vital for proving
bounds in optimization.
Theorem: Cauchy-Schwarz Inequality
For all \(x, y\) in an inner product space, \( |\langle x, y \rangle| \leq \|x\| \|y\| \).
Proof:
The case \(y = 0\) is trivial. For \(y \neq 0\), set \(t = \langle x, y \rangle / \langle y, y \rangle\). Expanding by the inner product axioms,
\[
\begin{align*}
0 \leq \|x - ty\|^2 &= \|x\|^2 - t\,\overline{\langle x, y\rangle} - \bar{t}\,\langle x, y\rangle + |t|^2 \|y\|^2 \\\\
&= \|x\|^2 - \frac{|\langle x, y\rangle|^2}{\|y\|^2},
\end{align*}
\]
which rearranges to \(|\langle x, y \rangle|^2 \leq \|x\|^2 \|y\|^2\).
Cauchy-Schwarz immediately justifies the claim that \(\|x\| := \sqrt{\langle x, x\rangle}\) is a norm. Expanding
\(\|x + y\|^2 = \|x\|^2 + 2\,\mathrm{Re}\,\langle x, y\rangle + \|y\|^2 \leq \|x\|^2 + 2\|x\|\|y\| + \|y\|^2 = (\|x\| + \|y\|)^2\)
gives the triangle inequality. Positive definiteness and homogeneity are direct consequences of the inner product axioms.
But how do we know if a given Banach space is actually a Hilbert space? The answer is a single identity, and the connection
between algebra and geometry that it encodes was established by Pascual Jordan and John von Neumann. Among the \(L^p\) spaces
of dimension at least two as vector spaces, only \(L^2\) satisfies that identity, which makes it the unique Hilbert space in the \(L^p\) family.
Theorem: Jordan-von Neumann Characterization
The norm of a normed space is induced by an inner product if and only if it satisfies the
Parallelogram Law:
\[
\|x + y\|^2 + \|x - y\|^2 = 2\|x\|^2 + 2\|y\|^2.
\]
In particular, a Banach space is a Hilbert space if and only if its norm satisfies this identity.
Proof Sketch:
The forward direction follows from expanding \(\|x \pm y\|^2 = \langle x \pm y, x \pm y \rangle\) via the inner product axioms
(sesquilinearity, which reduces to bilinearity in the real case). For the converse, one defines the candidate inner product
via the polarization identity:
\[
\begin{align*}
\langle x, y \rangle &= \tfrac{1}{4}\bigl(\|x+y\|^2 - \|x-y\|^2\bigr) \quad (\mathbb{F} = \mathbb{R}), \\\\
\langle x, y \rangle &= \tfrac{1}{4} \sum_{k=0}^{3} i^k \|x + i^k y\|^2 \quad (\mathbb{F} = \mathbb{C}),
\end{align*}
\]
One then verifies the inner product axioms. The delicate step is additivity in the first argument, which follows from the
parallelogram law applied to suitable vector combinations. Additivity gives homogeneity for rational scalars, continuity
extends it to real scalars, and when \(\mathbb{F} = \mathbb{C}\) homogeneity under multiplication by \(i\) is read off
directly from the polarization formula.
Orthonormal Sequences and Bessel's Inequality
In \(\mathbb{R}^n\), every vector admits a clean expansion against any orthonormal basis:
\(x = \sum_{k=1}^n \langle x, e_k \rangle e_k\). The infinite-dimensional analogue is one of the most powerful structural features
of Hilbert spaces, and it is the gateway to Fourier-style decompositions in abstract function spaces. We restrict throughout to
the separable case (Hilbert spaces possessing a countable dense subset), which covers all settings encountered in
this curriculum.
Definition: Orthonormal Sequence
A sequence \(\{e_n\}_{n=1}^\infty\) in a Hilbert space \(\mathcal{H}\) is orthonormal if
\[
\langle e_m, e_n \rangle = \delta_{mn} = \begin{cases} 1 & m = n, \\\\ 0 & m \neq n. \end{cases}
\]
That is, each term has unit norm and terms with distinct indices are mutually orthogonal.
For any vector \(x \in \mathcal{H}\), the scalars \(\langle x, e_n \rangle\) measure the
"component" of \(x\) along each \(e_n\). The fundamental constraint relating these
coordinates to the norm of \(x\) is Bessel's inequality.
Theorem: Bessel's Inequality
Let \(\{e_n\}_{n=1}^\infty\) be an orthonormal sequence in a Hilbert space \(\mathcal{H}\). For every \(x \in \mathcal{H}\),
\[
\sum_{n=1}^\infty |\langle x, e_n \rangle|^2 \leq \|x\|^2.
\]
Proof:
Fix \(N \geq 1\) and define the partial expansion \(s_N = \sum_{k=1}^N \langle x, e_k \rangle e_k\). By orthonormality,
\[
\|s_N\|^2 = \sum_{k=1}^N |\langle x, e_k \rangle|^2.
\]
Computing the inner product of \(x - s_N\) with each \(e_j\):
\[
\begin{align*}
\langle x - s_N, e_j \rangle
&= \langle x, e_j \rangle - \sum_{k=1}^N \langle x, e_k \rangle \langle e_k, e_j \rangle \\\\
&= \langle x, e_j \rangle - \langle x, e_j \rangle = 0 \quad (1 \leq j \leq N).
\end{align*}
\]
By the conjugate linearity of the inner product in the second argument,
\[
\langle x - s_N, s_N \rangle = \sum_{j=1}^N \overline{\langle x, e_j \rangle} \langle x - s_N, e_j \rangle = 0,
\]
so \(x - s_N \perp s_N\). Applying the Pythagorean identity (a direct consequence of expanding
\(\|x\|^2 = \langle (x - s_N) + s_N, (x - s_N) + s_N \rangle\)):
\[
\|x\|^2 = \|x - s_N\|^2 + \|s_N\|^2 \geq \|s_N\|^2 = \sum_{k=1}^N |\langle x, e_k \rangle|^2.
\]
The partial sums of the non-negative series \(\sum |\langle x, e_n \rangle|^2\) are uniformly bounded by \(\|x\|^2\), so the
series converges and satisfies the stated bound.
One simple but useful consequence follows. Since the series \(\sum_n |\langle x, e_n \rangle|^2\) converges, its terms must tend
to zero. Hence \(\langle x, e_n \rangle \to 0\) for every \(x \in \mathcal{H}\). The coordinates of any fixed vector against an
orthonormal sequence "fade out" at infinity. This phenomenon reappears later as the prototypical example of weak
convergence.
Bessel's inequality holds for any orthonormal sequence, but equality holds for every \(x \in \mathcal{H}\)
precisely when the sequence is "rich enough" to express every vector.
Definition: Orthonormal Basis (Separable Case)
An orthonormal sequence \(\{e_n\}_{n=1}^\infty\) in a separable Hilbert space \(\mathcal{H}\) is an orthonormal
basis (or Hilbert basis) if its closed linear span is all of \(\mathcal{H}\):
\[
\overline{\operatorname{span}\{e_n : n \geq 1\}} = \mathcal{H}.
\]
Note that this is not the same as a Hamel (algebraic) basis. Every vector is the limit of finite linear combinations
of the \(e_n\), not necessarily a finite combination itself.
Every infinite-dimensional separable Hilbert space admits an orthonormal basis. Starting from any countable dense subset, one
inductively orthonormalizes the vectors, skipping any that already lie in the closed linear span of the previously chosen ones.
The result is a countable orthonormal sequence whose closed linear span equals \(\mathcal{H}\). This is simply the
Gram-Schmidt process extended to infinite dimensions, the
same algorithm familiar from finite-dimensional linear algebra applied countably many times. In finite dimensions only
finitely many vectors are ever produced (every later candidate is skipped), so the construction yields a finite orthonormal
basis rather than an orthonormal sequence.
Conversely, the existence of a countable orthonormal basis forces the space to be separable, since the rational (or
rational-complex) finite linear combinations of basis vectors form a countable dense subset. Thus, among infinite-dimensional
spaces, separability and the existence of a countable orthonormal basis are equivalent.
For an orthonormal basis, Bessel's inequality strengthens to an equality, and every vector
admits a convergent infinite expansion in its coordinates.
Theorem: Parseval's Identity (Hilbert Space)
Let \(\{e_n\}_{n=1}^\infty\) be an orthonormal basis for a separable Hilbert space \(\mathcal{H}\). Then every
\(x \in \mathcal{H}\) admits the convergent expansion
\[
x = \sum_{n=1}^\infty \langle x, e_n \rangle e_n,
\]
and its norm satisfies Parseval's identity:
\[
\|x\|^2 = \sum_{n=1}^\infty |\langle x, e_n \rangle|^2.
\]
Proof:
Let \(s_N = \sum_{k=1}^N \langle x, e_k \rangle e_k\) as in the proof of Bessel's inequality. For \(M \lt N\), orthonormality
gives
\[
\|s_N - s_M\|^2 = \sum_{k=M+1}^N |\langle x, e_k \rangle|^2.
\]
Since \(\sum_n |\langle x, e_n \rangle|^2\) converges (Bessel), its tail sums vanish, so \(\{s_N\}\) is Cauchy in
\(\mathcal{H}\). By the
Cauchy criterion for completeness,
there exists \(s \in \mathcal{H}\) with \(s_N \to s\).
We claim \(s = x\). For each fixed \(j\), the equality \(\langle s_N, e_j \rangle = \langle x, e_j \rangle\) holds for all
\(N \geq j\) (by the same orthonormality computation used for Bessel's inequality). Letting \(N \to \infty\), continuity of
the inner product in the first argument (a direct consequence of Cauchy-Schwarz) gives
\[
\langle s, e_j \rangle = \lim_{N \to \infty} \langle s_N, e_j \rangle = \langle x, e_j \rangle,
\]
so \(\langle x - s, e_j \rangle = 0\) for every \(j \geq 1\).
Hence \(x - s\) is orthogonal to every \(e_j\), and by sesquilinearity together with continuity of the inner product, to every
element of \(\overline{\operatorname{span}\{e_n\}} = \mathcal{H}\). In particular, \(\langle x - s, x - s \rangle = 0\), so
\(x = s = \sum_{n=1}^\infty \langle x, e_n \rangle e_n\).
Finally, since \(s_N \to x\) in norm, the norm itself is continuous (by the reverse triangle inequality), so
\(\|s_N\|^2 \to \|x\|^2\). Combined with the orthonormal computation \(\|s_N\|^2 = \sum_{k=1}^N |\langle x, e_k \rangle|^2\)
established above,
\[
\|x\|^2 = \lim_{N \to \infty} \sum_{k=1}^N |\langle x, e_k \rangle|^2
= \sum_{n=1}^\infty |\langle x, e_n \rangle|^2,
\]
which is Parseval's identity.
Parseval's identity is the abstract form of energy preservation for orthonormal expansions. Parseval's identity for Fourier
series is the canonical instance, and Plancherel's theorem for the Fourier transform is its continuous counterpart. We
revisit these results when Fourier analysis is generalized to arbitrary Hilbert spaces.
Finite vs Infinite Dimensions
Because \(\mathbb{R}^n\) is a Hilbert space, it is easy to assume that all complete spaces behave like \(\mathbb{R}^n\).
However, the transition from finite-dimensional vectors to infinite-dimensional function spaces introduces profound topological
differences.
Theorem: Equivalence of Norms in Finite Dimensions
In a finite-dimensional vector space, all norms are
Lipschitz equivalent (and
therefore topologically equivalent). In infinite-dimensional spaces, this can fail. On \(L^\infty([0, 1])\) with
Lebesgue measure, both \(\|\cdot\|_1\) and \(\|\cdot\|_\infty\) are norms (the first is finite because
\(\|f\|_1 \leq \|f\|_\infty\) on an interval of length \(1\)). The indicator functions
\(f_k = \chi_{[0, 1/k]}\) converge to \(0\) in \(\|\cdot\|_1\), since \(\|f_k\|_1 = 1/k\), while
\(\|f_k - f_j\|_\infty = 1\) for \(k \neq j\), so \(\{f_k\}\) does not converge in \(\|\cdot\|_\infty\). The two norms
are therefore not even topologically equivalent.
Proof:
If the space is \(\{0\}\), every norm on it is identically zero and there is nothing to prove. Otherwise let
\(n \geq 1\) be its dimension, fix a basis \(\{e_1, \ldots, e_n\}\), and identify the space with \(\mathbb{F}^n\) via coordinates, where
\(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\). Any norm \(N\) satisfies \(N(x) \leq \sum_i |x_i|\, N(e_i) \leq C \|x\|_2\) by
the Cauchy-Schwarz inequality on \(\mathbb{F}^n\) (with \(\|x\|_2 = (\sum_i |x_i|^2)^{1/2}\) and
\(C = (\sum_i N(e_i)^2)^{1/2}\)). The reverse triangle inequality
\[
|N(x) - N(y)| \leq N(x - y) \leq C \|x - y\|_2
\]
then shows \(N\) is Lipschitz-continuous with respect to the Euclidean topology.
The unit sphere \(S = \{x : \|x\|_2 = 1\}\) is compact (Heine-Borel applied to \(\mathbb{F}^n \cong \mathbb{R}^n\) or
\(\mathbb{R}^{2n}\)), so \(N\) attains a minimum \(m\) and maximum \(M\) on \(S\). Moreover \(m \gt 0\), since \(N(x) = 0\)
would force \(x = 0\) by positive definiteness, contradicting \(x \in S\). Scaling gives
\(m \|x\|_2 \leq N(x) \leq M \|x\|_2\) for all \(x\), which makes the metric of \(N\) Lipschitz
equivalent to the Euclidean metric. For two norms \(N_1, N_2\) with constants \(m_1, M_1\) and \(m_2, M_2\), combining the
bounds gives \(N_1(x) \leq M_1 \|x\|_2 \leq (M_1 / m_2)\, N_2(x)\) and, symmetrically,
\(N_2(x) \leq (M_2 / m_1)\, N_1(x)\). Applied to \(x - y\), these bounds make the metrics of any two norms
Lipschitz equivalent. Lipschitz maps are continuous, so the identity map is a homeomorphism, which is
topological equivalence.
The most drastic difference, however, lies in compactness.
As we established in the Heine-Borel Theorem,
in \(\mathbb{R}^n\), every closed and bounded set is compact. This guarantees that any continuous real-valued objective
on a nonempty closed and bounded region attains its minimum. But as we hinted earlier, this comfort vanishes in infinite
dimensions.
Theorem: Compactness of the Unit Ball
The closed unit ball \(\{x : \|x\| \leq 1\}\) in a normed space is compact if and only if the space is
finite-dimensional.
Corollary: In any infinite-dimensional space, the closed unit ball (which is certainly closed and bounded) is
never compact.
If the space is finite-dimensional, compactness follows from Heine-Borel applied to \(\mathbb{F}^n\) via the Lipschitz
equivalence of norms on finite-dimensional spaces established above. The converse rests on a subtle geometric fact known as
Riesz's Lemma, which guarantees, for any proper closed subspace, the existence of a unit vector of the
whole space almost orthogonal to it. Iterating this construction yields an infinite sequence of unit vectors pairwise separated by \(\geq 1/2\),
precluding any convergent subsequence. Both directions are given in full where
Riesz's Lemma is proved.
Why does this matter? Because a closed unit ball is the prototypical "closed and bounded" set. The theorem tells us that in an
infinite-dimensional function space, simply bounding the parameters does not guarantee that a minimizing sequence has a
limit point inside that boundary. A sequence can wander endlessly inside a bounded ball without any subsequence converging to
a point.
Geometry of Unit Balls & ML Applications
The choice of norm in a Banach space dictates the geometry of its
open and closed balls.
In machine learning, we frequently use norms as regularization terms to constrain our model parameters
(\(\|\mathbf{w}\|_p \leq C\)). The geometric shape of these balls helps explain why different regularizers tend to produce
different types of models.
Geometry of \(L^p\) Unit Balls in \(\mathbb{R}^2\)
- \(L^1\) norm (Lasso):
The unit ball is a diamond (a square rotated by 45 degrees). It has sharp corners precisely on the axes.
- \(L^2\) norm (Ridge):
The unit ball is a perfect circle. It is strictly convex and rotationally invariant.
- \(L^\infty\) norm:
The unit ball is a square aligned with the axes.
When we optimize a loss function subject to an \(L^1\) penalty (Lasso regression), the expanding contour of the loss function
is highly likely to hit the "sharp corners" of the \(L^1\) diamond first. Because these corners lie exactly on the axes,
many parameter weights become exactly zero. This geometric property of the \(L^1\) Banach space is the mathematical mechanism
behind Sparsity and feature selection in ML.
Conversely, the \(L^2\) Hilbert space penalty (Ridge regression) has a perfectly round, strictly convex unit ball.
The loss contour will touch it at a tangent, smoothly shrinking all parameters but rarely setting them to exactly zero.
When the loss is convex, as in least squares, adding the strictly convex penalty \(\lambda \|\mathbf{w}\|_2^2\) with
\(\lambda \gt 0\) makes the objective strongly convex, so the optimization problem has a unique global solution,
making it mathematically highly stable.
Finally, when we leverage the inner product structure of Hilbert spaces, we can map data into high- (possibly infinite-)
dimensional feature spaces to find linear boundaries for non-linear data. This is the foundation of the
Kernel Trick and
Reproducing Kernel Hilbert Spaces (RKHS), which power
Support Vector Machines (SVMs) and
Gaussian processes.
From Spaces to Functionals: Why "Functional" Analysis?
We have spent this entire page defining the "containers": Banach spaces and Hilbert spaces. These are the stages upon which
mathematics takes place. But treating a function \(f\) merely as a "point" in a space is not enough. To do analysis, we need to
measure these points.
In classical calculus, we input a number \(x\) and get a number \(y = f(x)\). In Functional Analysis, the paradigm shifts:
we input a function \(f\) (a point in our function space) and output a scalar
(a number in the underlying field \(\mathbb{R}\) or \(\mathbb{C}\)).
Definition: Functional
A functional on a vector space \(V\) over \(\mathbb{F}\) is a mapping
\[
\varphi: V \to \mathbb{F} \quad (\mathbb{F} = \mathbb{R} \text{ or } \mathbb{C}).
\]
Definition: Linear Functional
A functional \(\varphi: V \to \mathbb{F}\) is called a linear functional if
\[
\varphi(\alpha x + \beta y) = \alpha \varphi(x) + \beta \varphi(y)
\]
for all \(x, y \in V\) and \(\alpha, \beta \in \mathbb{F}\).
Why is this shift so important? Because functionals are the "measurement tools" of the space. They can be broadly categorized
into nonlinear and linear tools:
- The Norm (Nonlinear):
The map \(f \mapsto \|f\|\) takes a function and returns a scalar representing its "size" or "length."
- The Loss Function (Nonlinear):
In Machine Learning, the objective function \(\mathcal{L}(\theta)\) evaluates a model (a function defined by parameters \(\theta\))
and returns a scalar "error."
- Integration (Linear):
The map \(f \mapsto \int f(x) dx\) takes a function and returns a scalar representing its "area." The linearity of the integral
is fundamental to measure theory.
- Evaluation (Linear):
The map \(\delta_x(f) = f(x)\) evaluates a function at a specific point \(x\), whenever that evaluation is well-defined. For
\(L^p\) spaces, whose elements are a.e.-equivalence classes, well-definedness is subtle, and recovering it is one motivation
for RKHS.
The Path to Dual Spaces and Manifolds
This concept bridges directly to the next phase of our curriculum. In infinite-dimensional spaces, we are specifically interested in
measurement tools that preserve both the algebraic structure (linearity) and the topological structure (continuity). If we collect
all the continuous linear functionals for a specific space \(V\), we form a new space called the
topological dual space, denoted \(V^*\).
Understanding the profound relationship between a space and its dual is the absolute prerequisite for:
- Quantum Mechanics:
The "Bra-ket" notation \(\langle \phi | \psi \rangle\) is the action of a dual vector (the Bra \(\langle \phi |\) in \(V^*\))
acting on a state vector (the Ket \(| \psi \rangle\) in \(V\)). The physics convention makes this pairing conjugate-linear in
\(\phi\) and linear in \(\psi\). Under our mathematical convention of first-slot linearity, the two are related by
\(\langle \phi | \psi \rangle_{\text{phys}} = \langle \psi, \phi \rangle_{\text{math}}\).
- Optimization & The Riesz Representation:
In calculus, the true derivative of a function is a differential \(df\) (an element of the dual space). We will see in the
Riesz Representation Theorem that the "gradient" \(\nabla f\) is exactly this dual element "converted" back
into a regular vector via the space's inner product, allowing us to perform gradient descent updates.
- Geometric Deep Learning:
In smooth manifolds, a "Tangent Vector" represents a directional derivative (movement), while a "Cotangent Vector" (a functional)
represents the measurement of that change.
We have built the spaces. The pages ahead study the Operators that transform them and the
Functionals that measure them.