What We Have Seen So Far...
We have encountered various "spaces" throughout our study. In linear algebra, we started with
vector spaces. To discuss "length," we introduced
normed vector spaces, and to formalize "orthogonality," we
considered inner product spaces. Strictly speaking, our familiar space \(\mathbb{R}^n\) is categorized
as a Hilbert space, which is an inner product space that guarantees completeness.
Completeness is critical in calculus and analysis because, without it, we cannot guarantee
the convergence of limits "within" the space. Moreover, we frequently encounter the \(L^2\) space in
machine learning contexts. What exactly is it? The \(L^2\) space is also a Hilbert space, but unlike \(\mathbb{R}^n\), whose
elements are finite-dimensional vectors, it is a function space whose elements are functions.
Furthermore, we have explored \(L^1\) and \(L^\infty\). These are categorized as Banach spaces - complete
normed vector spaces that do not necessarily arise from an inner product. Thus, a Banach space is a more general concept than a Hilbert space,
yet it is essential in both analysis and machine learning.
For instance, in deep learning architectures, a neural network can be viewed
as a point (a function) within a vast function space, and we build highly complex models via topological connections. While it is commonly
stated in engineering that training works as long as the functions are differentiable (which provides the local gradient), mathematically,
differentiability only provides the direction. For a sequence of optimized models to actually converge to a valid limit without "falling
through a hole," the underlying space must first be complete - an absolute prerequisite before we can even rely on properties
like convexity or
compactness
to guarantee the existence of an optimum.
Fundamentally, these function spaces are defined by their properties regarding integration. Specific rules of integrability
dictate which functions qualify to be part of spaces like \(L^1, L^2,\) and \(L^\infty\).
This is precisely why we studied Lebesgue integration: it is the mathematical framework that
ensures these function spaces possess completeness, using measures as their foundation. It is crucial
not to confuse a vector space with a measure space. A measure space is not a vector space; rather, it provides the underlying
domain upon which functions are defined. In other words, a measure space itself does not possess algebraic operations like vector addition or
scalar multiplication.
Viewed through this lens, core concepts in statistics gain clear mathematical structure. We can now rigorously understand that
random variables are actually measurable functions that map outcomes from a sample space to real numbers, and a
probability space is simply a specific type of measure space (with a total measure of 1)
used to describe data distributions. Consequently, the expected value \(\mathbb{E}[X]\) is precisely a Lebesgue integral over this probability
space, and the variance \(\mathbb{V}[X]\) is the squared \(L^2\) norm of the random variable centered around its mean.
Going back to our algebraic foundations, we know that \(\mathbb{R}\) is a field
in abstract algebra. Simultaneously, from the perspective of functional analysis, \(\mathbb{R}\) itself acts as a one-dimensional
Hilbert space. Thus, \(\mathbb{R}\) serves both as the foundational "rule" for arithmetic operations and as the underlying "stage"
(codomain) for our functions. A field equipped with a topology that makes its algebraic operations continuous is known as a topological field.
In particular, \(\mathbb{R}\) and \(\mathbb{C}\) are indispensable because they are complete topological fields - a highly special
property in mathematics, even though we have implicitly relied on them since elementary school (for comparison, the field of rational numbers \(\mathbb{Q}\)
is a topological field, but it is not complete).
As a final note on notation: in applied mathematics and machine learning contexts, you will frequently see \(L_1, L_2,\) and \(L_\infty\)
written with subscripts instead of superscripts (\(L^1, L^2, L^\infty\)). This variation stems from differing conventional focuses across
disciplines, but both notations refer to exactly the same mathematical concepts and spaces.
Normed Spaces & Banach Spaces
To perform calculus or optimization on a vector space, we need a notion of "distance" to define limits and convergence.
A norm provides this by measuring the "length" of a vector.
Crucially, any norm naturally induces a metric (distance function)
defined by \(d(x, y) = \|x - y\|\). Once we have a metric, we can define
Cauchy sequences and ask whether the space is
complete.
Definition: Normed Space (Normed Vector Space)
A normed space is a pair \((\mathcal{X}, \|\cdot\|)\), where \(\mathcal{X}\) is a vector space
over \(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\) and \(\|\cdot\|: \mathcal{X} \to [0, \infty)\) is a
norm, meaning that for all \(x, y \in \mathcal{X}\) and \(\alpha \in \mathbb{F}\):
- (Positive definiteness) \(\|x\| = 0\) if and only if \(x = 0\);
- (Homogeneity) \(\|\alpha x\| = |\alpha|\,\|x\|\);
- (Triangle inequality) \(\|x + y\| \leq \|x\| + \|y\|\).
Definition: Banach Space
A Banach space is a normed space that is complete with respect to the
metric defined by the norm.
Equivalently: every Cauchy sequence in the space converges to a limit that is also within the space.
The most important examples of Banach spaces in analysis and machine learning are the \(L^p\) spaces.
Thanks to the rigorous foundation of Lebesgue integration, these spaces of functions are guaranteed to be complete.
Definition: \(L^p\) Spaces
For \(1 \leq p < \infty\) and a measure space \((\Omega, \mathcal{F}, \mu)\), the space \(L^p(\Omega, \mu)\)
consists of (equivalence classes, modulo a.e. equality, of) measurable functions \(f\) for which
\(\int_\Omega |f|^p \, d\mu < \infty\). The norm is defined as:
\[
\|f\|_p = \left( \int_\Omega |f(x)|^p \, d\mu \right)^{1/p}.
\]
The triangle inequality — that \(\|\cdot\|_p\) is indeed a norm — is Minkowski's inequality,
proved in a later chapter (see Lp Spaces and Riesz-Fischer, where completeness is also established).
Definition: \(L^\infty\) Space
The space \(L^\infty(\Omega, \mu)\) consists of (equivalence classes, modulo a.e. equality, of)
measurable functions \(f\) that are essentially bounded. The norm is the
essential supremum:
\[
\|f\|_\infty = \operatorname{ess\,sup}_{x} |f(x)| = \inf \{ C \geq 0 : |f(x)| \leq C \text{ a.e.} \}.
\]
Inner Product Spaces & Hilbert Spaces
While Banach spaces allow us to measure lengths and distances, they lack the geometric concepts of "angles" and "orthogonality."
To recover the rich geometry of Euclidean space in abstract settings, we need an
inner product.
Every inner product naturally induces a candidate norm via \(\|x\| := \sqrt{\langle x, x \rangle}\); that this candidate is indeed a
norm (specifically, that it satisfies the triangle inequality) is a consequence of the Cauchy-Schwarz inequality proved below.
Hence every inner product space is automatically a normed space.
Definition: Inner Product Space
An inner product space is a vector space \(\mathcal{X}\) over \(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\)
equipped with an inner product \(\langle \cdot, \cdot \rangle: \mathcal{X} \times \mathcal{X} \to \mathbb{F}\)
satisfying, for all \(x, y, z \in \mathcal{X}\) and \(\alpha, \beta \in \mathbb{F}\):
- (Conjugate symmetry) \(\langle x, y \rangle = \overline{\langle y, x \rangle}\);
- (Linearity in the first argument) \(\langle \alpha x + \beta y, z \rangle = \alpha \langle x, z \rangle + \beta \langle y, z \rangle\);
- (Positive definiteness) \(\langle x, x \rangle \geq 0\), with equality if and only if \(x = 0\).
Note: We follow the mathematical convention of linearity in the first argument.
The physics/quantum mechanics convention places linearity in the second argument, which conjugates the first.
Just as a normed space requires completeness to become a Banach space, an inner product space requires completeness
to become the ultimate setting for functional analysis: a Hilbert space.
Definition: Hilbert Space
A Hilbert space \(\mathcal{H}\) is an inner product space that is complete
with respect to the metric \(d(x, y) = \|x - y\|\), where \(\|x\| := \sqrt{\langle x, x \rangle}\) is
the norm induced by the inner product (see the Cauchy-Schwarz inequality below for the proof that this
is a norm).
Hilbert spaces retain more of the geometric structure of \(\mathbb{R}^n\) — orthogonality, projection,
self-duality — than general Banach spaces, making them the most tractable infinite-dimensional spaces.
The foundational inequality that connects the inner product to the norm is the Cauchy-Schwarz inequality, which is vital for
proving bounds in optimization.
Theorem: Cauchy-Schwarz Inequality
For all \(x, y\) in an inner product space, \( |\langle x, y \rangle| \leq \|x\| \|y\| \).
Proof:
The case \(y = 0\) is trivial. For \(y \neq 0\), set \(t = \langle x, y \rangle / \langle y, y \rangle\). Expanding by the inner product axioms,
\[
\begin{align*}
0 \leq \|x - ty\|^2 &= \|x\|^2 - t\,\overline{\langle x, y\rangle} - \bar{t}\,\langle x, y\rangle + |t|^2 \|y\|^2 \\\\
&= \|x\|^2 - \frac{|\langle x, y\rangle|^2}{\|y\|^2},
\end{align*}
\]
which rearranges to \(|\langle x, y \rangle|^2 \leq \|x\|^2 \|y\|^2\).
Cauchy-Schwarz immediately justifies the claim that \(\|x\| := \sqrt{\langle x, x\rangle}\) is a norm: expanding
\(\|x + y\|^2 = \|x\|^2 + 2\,\mathrm{Re}\,\langle x, y\rangle + \|y\|^2 \leq \|x\|^2 + 2\|x\|\|y\| + \|y\|^2 = (\|x\| + \|y\|)^2\)
gives the triangle inequality; positive definiteness and homogeneity are direct consequences of the inner product axioms.
But how do we know if a given Banach space is actually a Hilbert space? It turns out that a norm is induced by an inner product
if and only if it satisfies the Parallelogram Law. This profound connection between algebra and geometry was
proven by Pascual Jordan and John von Neumann. Among the \(L^p\) spaces, only \(L^2\) satisfies this law, making it the unique
Hilbert space in the \(L^p\) family.
Theorem: Jordan-von Neumann Characterization
The norm of a normed space is induced by an inner product if and only if it satisfies the
Parallelogram Law:
\[
\|x + y\|^2 + \|x - y\|^2 = 2\|x\|^2 + 2\|y\|^2.
\]
In particular, a Banach space is a Hilbert space if and only if its norm satisfies this identity.
Proof Sketch:
The forward direction follows from expanding \(\|x \pm y\|^2 = \langle x \pm y, x \pm y \rangle\)
via the inner product axioms (sesquilinearity, which reduces to bilinearity in the real case).
For the converse, one defines the candidate inner product via the polarization identity:
\[
\begin{align*}
\langle x, y \rangle &= \tfrac{1}{4}\bigl(\|x+y\|^2 - \|x-y\|^2\bigr) \quad (\mathbb{F} = \mathbb{R}), \\\\
\langle x, y \rangle &= \tfrac{1}{4} \sum_{k=0}^{3} i^k \|x + i^k y\|^2 \quad (\mathbb{F} = \mathbb{C}),
\end{align*}
\]
and verifies the inner product axioms. The delicate step is additivity in the first argument, which follows
from the parallelogram law applied to suitable vector combinations, extended to all scalars by continuity.
Orthonormal Sequences and Bessel's Inequality
In \(\mathbb{R}^n\), every vector admits a clean expansion against any orthonormal basis:
\(x = \sum_{k=1}^n \langle x, e_k \rangle e_k\). The infinite-dimensional analogue is one of
the most powerful structural features of Hilbert spaces — and the gateway to Fourier-style
decompositions in abstract function spaces. We restrict throughout to the separable
case (Hilbert spaces possessing a countable dense subset), which covers all settings encountered
in this curriculum.
Definition: Orthonormal Sequence
A sequence \(\{e_n\}_{n=1}^\infty\) in a Hilbert space \(\mathcal{H}\) is
orthonormal if
\[
\langle e_m, e_n \rangle = \delta_{mn} = \begin{cases} 1 & m = n, \\ 0 & m \neq n. \end{cases}
\]
That is, each vector has unit norm and distinct vectors are mutually orthogonal.
For any vector \(x \in \mathcal{H}\), the scalars \(\langle x, e_n \rangle\) measure the
"component" of \(x\) along each \(e_n\). The fundamental constraint relating these
coordinates to the norm of \(x\) is Bessel's inequality.
Theorem: Bessel's Inequality
Let \(\{e_n\}_{n=1}^\infty\) be an orthonormal sequence in a Hilbert space \(\mathcal{H}\).
For every \(x \in \mathcal{H}\),
\[
\sum_{n=1}^\infty |\langle x, e_n \rangle|^2 \;\leq\; \|x\|^2.
\]
Proof:
Fix \(N \geq 1\) and define the partial expansion
\(s_N = \sum_{k=1}^N \langle x, e_k \rangle e_k\). By orthonormality,
\[
\|s_N\|^2 = \sum_{k=1}^N |\langle x, e_k \rangle|^2.
\]
Computing the inner product of \(x - s_N\) with each basis vector:
\[
\begin{align*}
\langle x - s_N, e_j \rangle
&= \langle x, e_j \rangle - \sum_{k=1}^N \langle x, e_k \rangle \langle e_k, e_j \rangle \\\\
&= \langle x, e_j \rangle - \langle x, e_j \rangle = 0 \quad (1 \leq j \leq N).
\end{align*}
\]
By the conjugate linearity of the inner product in the second argument,
\[
\langle x - s_N, s_N \rangle = \sum_{j=1}^N \overline{\langle x, e_j \rangle} \langle x - s_N, e_j \rangle = 0,
\]
so \(x - s_N \perp s_N\).
Applying the Pythagorean identity (a direct consequence of expanding
\(\|x\|^2 = \langle (x - s_N) + s_N, (x - s_N) + s_N \rangle\)):
\[
\|x\|^2 = \|x - s_N\|^2 + \|s_N\|^2 \;\geq\; \|s_N\|^2 = \sum_{k=1}^N |\langle x, e_k \rangle|^2.
\]
The partial sums of the non-negative series
\(\sum |\langle x, e_n \rangle|^2\) are uniformly bounded by \(\|x\|^2\),
so the series converges and satisfies the stated bound.
A simple but useful consequence: since the series \(\sum_n |\langle x, e_n \rangle|^2\) converges,
its terms must tend to zero. Hence \(\langle x, e_n \rangle \to 0\)
for every \(x \in \mathcal{H}\) — the coordinates of any fixed vector against an orthonormal sequence
"fade out" at infinity. This phenomenon will reappear in the next chapter as the prototypical example of
weak convergence.
Bessel's inequality holds for any orthonormal sequence, but the upper bound is achieved
with equality precisely when the sequence is "rich enough" to express every vector.
Definition: Orthonormal Basis (Separable Case)
An orthonormal sequence \(\{e_n\}_{n=1}^\infty\) in a separable Hilbert space \(\mathcal{H}\)
is an orthonormal basis (or Hilbert basis) if its closed
linear span is all of \(\mathcal{H}\):
\[
\overline{\operatorname{span}\{e_n : n \geq 1\}} \;=\; \mathcal{H}.
\]
Note that this is not the same as a Hamel (algebraic) basis: every vector is
the limit of finite linear combinations of the \(e_n\), not necessarily a finite combination itself.
Every separable Hilbert space admits an orthonormal basis: starting from any countable
dense subset, one inductively orthonormalizes the vectors (skipping any that already lie
in the closed linear span of the previously chosen ones), producing a countable orthonormal
sequence whose closed linear span equals \(\mathcal{H}\). This is simply the
Gram-Schmidt process
extended to infinite dimensions — the same algorithm familiar from finite-dimensional linear
algebra, applied countably many times. Conversely, the existence of a countable orthonormal
basis forces the space to be separable, since the rational (or rational-complex) finite
linear combinations of basis vectors form a countable dense subset. Thus separability and
the existence of a countable orthonormal basis are equivalent.
For an orthonormal basis, Bessel's inequality strengthens to an equality, and every vector
admits a convergent infinite expansion in its coordinates.
Theorem: Parseval's Identity (Hilbert Space)
Let \(\{e_n\}_{n=1}^\infty\) be an orthonormal basis for a separable Hilbert space \(\mathcal{H}\).
Then every \(x \in \mathcal{H}\) admits the convergent expansion
\[
x \;=\; \sum_{n=1}^\infty \langle x, e_n \rangle e_n,
\]
and its norm satisfies Parseval's identity:
\[
\|x\|^2 \;=\; \sum_{n=1}^\infty |\langle x, e_n \rangle|^2.
\]
Proof:
Let \(s_N = \sum_{k=1}^N \langle x, e_k \rangle e_k\) as in the proof of Bessel's inequality.
For \(M < N\), orthonormality gives
\[
\|s_N - s_M\|^2 = \sum_{k=M+1}^N |\langle x, e_k \rangle|^2.
\]
Since \(\sum_n |\langle x, e_n \rangle|^2\) converges (Bessel), its tail sums vanish, so
\(\{s_N\}\) is Cauchy in \(\mathcal{H}\). By completeness of \(\mathcal{H}\), there exists
\(s \in \mathcal{H}\) with \(s_N \to s\).
We claim \(s = x\). For each fixed \(j\), the equality
\(\langle s_N, e_j \rangle = \langle x, e_j \rangle\) holds for all \(N \geq j\)
(by the same orthonormality computation as in the proof of Bessel's inequality).
Letting \(N \to \infty\), continuity of the inner product in the first argument
(a direct consequence of Cauchy-Schwarz) gives
\[
\langle s, e_j \rangle = \lim_{N \to \infty} \langle s_N, e_j \rangle = \langle x, e_j \rangle,
\]
so \(\langle x - s, e_j \rangle = 0\) for every \(j \geq 1\). Hence \(x - s\) is orthogonal
to every \(e_j\), and by sesquilinearity together with continuity of the inner product,
to every element of \(\overline{\operatorname{span}\{e_n\}} = \mathcal{H}\). In particular,
\(\langle x - s, x - s \rangle = 0\), so \(x = s = \sum_{n=1}^\infty \langle x, e_n \rangle e_n\).
Finally, since \(s_N \to x\) in norm, the norm itself is continuous (by the reverse triangle
inequality), so \(\|s_N\|^2 \to \|x\|^2\). Combined with the orthonormal computation
\(\|s_N\|^2 = \sum_{k=1}^N |\langle x, e_k \rangle|^2\) established in the proof of Bessel's inequality,
\[
\|x\|^2 = \lim_{N \to \infty} \sum_{k=1}^N |\langle x, e_k \rangle|^2
= \sum_{n=1}^\infty |\langle x, e_n \rangle|^2,
\]
which is Parseval's identity.
Parseval's identity is the abstract source of every "energy preservation" theorem in analysis.
Plancherel's theorem for the Fourier transform — which we will revisit when generalizing
Fourier analysis to arbitrary Hilbert spaces — is the canonical instance.
Finite vs Infinite Dimensions
Because \(\mathbb{R}^n\) is a Hilbert space, it is easy to assume that all complete spaces behave like \(\mathbb{R}^n\).
However, the transition from finite-dimensional vectors to infinite-dimensional function spaces introduces profound topological
differences.
Theorem: Equivalence of Norms in Finite Dimensions
In a finite-dimensional vector space, all norms are
Lipschitz equivalent
(and therefore topologically equivalent). In infinite-dimensional spaces, this fails: a sequence of functions
may converge in \(L^1\) while diverging in \(L^\infty\).
Proof Sketch:
Fix a basis \(\{e_1, \ldots, e_n\}\) and identify the space with \(\mathbb{F}^n\) via coordinates,
where \(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\). Any norm \(N\) satisfies
\(N(x) \leq \sum_i |x_i|\, N(e_i) \leq C \|x\|_2\) by the Cauchy-Schwarz inequality on \(\mathbb{F}^n\)
(with \(\|x\|_2 = (\sum_i |x_i|^2)^{1/2}\) and \(C = (\sum_i N(e_i)^2)^{1/2}\)). The reverse triangle inequality
\[
|N(x) - N(y)| \leq N(x - y) \leq C \|x - y\|_2
\]
then shows \(N\) is Lipschitz-continuous with respect to the Euclidean topology.
The unit sphere \(S = \{x : \|x\|_2 = 1\}\) is compact (Heine-Borel applied to
\(\mathbb{F}^n \cong \mathbb{R}^n\) or \(\mathbb{R}^{2n}\)), so \(N\) attains
a minimum \(m\) and maximum \(M\) on \(S\); moreover \(m > 0\), since \(N(x) = 0\) would force \(x = 0\) by positive
definiteness, contradicting \(x \in S\). Scaling gives \(m \|x\|_2 \leq N(x) \leq M \|x\|_2\) for all \(x\),
which is Lipschitz equivalence.
The most drastic difference, however, lies in compactness.
As we established in the Heine-Borel Theorem,
in \(\mathbb{R}^n\), every closed and bounded set is compact. This guarantees that any continuous optimization problem
on a bounded region has a minimum. But as we hinted earlier, this comfort vanishes in infinite dimensions.
Theorem: Compactness of the Unit Ball
The closed unit ball \(\{x : \|x\| \leq 1\}\) in a normed space is compact if and only if the space is
finite-dimensional.
Corollary: In any infinite-dimensional space, the closed unit ball (which is strictly closed and bounded)
is never compact.
The forward direction follows from Heine-Borel applied to \(\mathbb{F}^n\) via the Lipschitz equivalence of norms on
finite-dimensional spaces established above. The reverse direction rests on a subtle geometric fact known as
Riesz's Lemma, which guarantees, in any proper closed subspace, the existence of a unit vector
almost orthogonal to it. Iterating this construction yields an infinite sequence of unit vectors pairwise separated
by \(\geq 1/2\), precluding any convergent subsequence. Both directions are given in full in the next chapter;
see Riesz's Lemma.
Why does this matter? Because a closed unit ball is the prototypical "closed and bounded" set. The theorem tells us that in an
infinite-dimensional function space, simply bounding your parameters does not guarantee that your optimization algorithm will
converge to a solution inside that boundary. A sequence can wander endlessly inside a bounded ball without ever converging to a point.
Geometry of Unit Balls & ML Applications
The choice of norm in a Banach space dictates the geometry of its
open and closed balls.
In machine learning, we frequently use norms as regularization terms to constrain our model parameters
(\(\|w\|_p \leq C\)). The geometric shape of these unit balls perfectly explains why different regularizers produce
different types of models.
Geometry of \(L^p\) Unit Balls in \(\mathbb{R}^2\)
- \(L^1\) norm (Lasso):
The unit ball is a diamond (a square rotated by 45 degrees). It has sharp corners precisely on the axes.
- \(L^2\) norm (Ridge):
The unit ball is a perfect circle. It is strictly convex and rotationally invariant.
- \(L^\infty\) norm:
The unit ball is a square aligned with the axes.
When we optimize a loss function subject to an \(L^1\) penalty (Lasso regression), the expanding contour of the loss function
is highly likely to hit the "sharp corners" of the \(L^1\) diamond first. Because these corners lie exactly on the axes,
many parameter weights become exactly zero. This geometric property of the \(L^1\) Banach space is the mathematical mechanism
behind Sparsity and feature selection in ML.
Conversely, the \(L^2\) Hilbert space penalty (Ridge regression) has a perfectly round, strictly convex unit ball.
The loss contour will touch it at a tangent, smoothly shrinking all parameters but rarely setting them to exactly zero.
The strict convexity of \(L^2\) guarantees that the optimization problem has a unique global solution,
making it mathematically highly stable.
Finally, when we leverage the inner product structure of Hilbert spaces, we can project data into infinite-dimensional spaces
to find linear boundaries for non-linear data. This is the foundation of the
Kernel Trick and
Reproducing Kernel Hilbert Spaces (RKHS), which power
Support Vector Machines (SVMs) and
Gaussian processes.
From Spaces to Functionals: Why "Functional" Analysis?
We have spent this entire chapter defining the "containers" — Banach spaces and Hilbert spaces. These are the stages upon which
mathematics takes place. But treating a function \(f\) merely as a "point" in a space is not enough. To do analysis, we need to
measure these points.
In classical calculus, we input a number \(x\) and get a number \(y = f(x)\). In Functional Analysis, the paradigm shifts:
we input a function \(f\) (a point in our function space) and output a scalar
(a number in the underlying field \(\mathbb{R}\) or \(\mathbb{C}\)).
Definition: Functional
A functional on a vector space \(V\) over \(\mathbb{F}\) is a mapping
\[
\varphi: V \to \mathbb{F} \quad (\mathbb{F} = \mathbb{R} \text{ or } \mathbb{C}).
\]
Definition: Linear Functional
A functional \(\varphi: V \to \mathbb{F}\) is called a linear functional if
\[
\varphi(\alpha x + \beta y) = \alpha \varphi(x) + \beta \varphi(y)
\]
for all \(x, y \in V\) and \(\alpha, \beta \in \mathbb{F}\).
Why is this shift so important? Because functionals are the "measurement tools" of the space. They can be broadly categorized
into nonlinear and linear tools:
- The Norm (Nonlinear):
The map \(f \mapsto \|f\|\) takes a function and returns a scalar representing its "size" or "length."
- The Loss Function (Nonlinear):
In Machine Learning, the objective function \(\mathcal{L}(\theta)\) evaluates a model (a function defined by parameters \(\theta\))
and returns a scalar "error."
- Integration (Linear):
The map \(f \mapsto \int f(x) dx\) takes a function and returns a scalar representing its "area." The linearity of the integral
is fundamental to measure theory.
- Evaluation (Linear):
The map \(\delta_x(f) = f(x)\) evaluates a function at a specific point \(x\) (when this evaluation is well-defined;
this becomes subtle for \(L^p\) spaces whose elements are a.e.-equivalence classes — a motivation for RKHS).
The Path to Dual Spaces and Manifolds
This concept bridges directly to the next phase of our curriculum. In infinite-dimensional spaces, we are specifically interested in
measurement tools that preserve both the algebraic structure (linearity) and the topological structure (continuity). If we collect
all the continuous linear functionals for a specific space \(V\), we form a new space called the
topological dual space, denoted \(V^*\).
Understanding the profound relationship between a space and its dual is the absolute prerequisite for:
- Quantum Mechanics:
The "Bra-ket" notation \(\langle \phi | \psi \rangle\) is the action of a dual vector (the Bra \(\langle \phi |\) in \(V^*\))
acting on a state vector (the Ket \(| \psi \rangle\) in \(V\)). The physics convention makes this pairing conjugate-linear
in \(\phi\) and linear in \(\psi\); under our mathematical convention (first-slot linearity), the two are related by
\(\langle \phi | \psi \rangle_{\text{phys}} = \langle \psi, \phi \rangle_{\text{math}}\).
- Optimization & The Riesz Representation:
In calculus, the true derivative of a function is a differential \(df\) (an element of the dual space). We will see in the
Riesz Representation Theorem that the "gradient" \(\nabla f\) is exactly this dual element "converted" back
into a regular vector via the space's inner product, allowing us to perform gradient descent updates.
- Geometric Deep Learning:
In smooth manifolds, a "Tangent Vector" represents a directional derivative (movement), while a "Cotangent Vector" (a functional)
represents the measurement of that change.
We have built the spaces. In the next chapter, we will study the Operators that transform them and the Functionals
that measure them.