What We Have Seen So Far...
We have encountered various "spaces" throughout our study. In linear algebra, we started with
vector spaces. To discuss "length," we introduced
normed vector spaces, and to formalize "orthogonality," we
considered inner product spaces. Strictly speaking, our familiar space \(\mathbb{R}^n\) is categorized
as a Hilbert space, which is an inner product space that guarantees completeness.
Completeness is critical in calculus and analysis because, without it, we cannot guarantee
the convergence of limits "within" the space. Moreover, we frequently encounter the \(L^2\) space in
machine learning contexts. What exactly is it? The \(L^2\) space is also a Hilbert space, but unlike \(\mathbb{R}^n\), whose
elements are finite-dimensional vectors, it is a function space whose elements are functions.
Furthermore, we have explored \(L^1\) and \(L^\infty\). These are categorized as Banach spaces, that is,
complete normed vector spaces that do not necessarily arise from an inner product. Thus, a Banach space is a more
general concept than a Hilbert space, yet it is essential in both analysis and machine learning.
For instance, in deep learning architectures, a neural network can
be viewed as a point (a function) within a vast function space, and we build highly complex models via topological connections.
While it is commonly stated in engineering that training works as long as the functions are differentiable (which provides the
local gradient), mathematically, differentiability only provides the direction. For a sequence of optimized models to actually
converge to a valid limit without "falling through a hole," the underlying space must first be complete. That
requirement is an absolute prerequisite, and it comes before any appeal to properties like
convexity or
compactness to guarantee the existence of
an optimum.
Fundamentally, these function spaces are defined by their properties regarding integration. Specific rules of
integrability dictate which functions qualify to be part of spaces like \(L^1, L^2,\) and \(L^\infty\). This is precisely why we
studied Lebesgue integration. That theory ensures these function spaces possess
completeness, and it rests on measures as its foundation. A vector space and a
measure space must not be confused. Rather than carrying algebraic structure, a measure space provides the
underlying domain upon which functions are defined. It possesses no algebraic operations like vector addition or scalar
multiplication.
Viewed through this lens, core concepts in statistics gain clear mathematical structure. We can now rigorously understand that
random variables are actually measurable functions that map outcomes from a sample space to real numbers, and a
probability space is simply a specific type of measure space (with a total measure of 1)
used to describe data distributions. Consequently, the expected value \(\mathbb{E}[X]\) is precisely a Lebesgue integral over this probability
space, and the variance \(\mathbb{V}[X]\) is the squared \(L^2\) norm of the random variable centered around its mean.
Going back to our algebraic foundations, we know that \(\mathbb{R}\) is a
field in abstract algebra. Simultaneously, from the
perspective of functional analysis, \(\mathbb{R}\) itself acts as a one-dimensional Hilbert space. Thus,
\(\mathbb{R}\) serves both as the foundational "rule" for arithmetic operations and as the underlying "stage" (codomain) for our
functions.
A field equipped with a topology that makes its algebraic operations continuous is known as a topological field.
In particular, \(\mathbb{R}\) and \(\mathbb{C}\) are indispensable because they are complete topological fields.
That combination is highly special in mathematics, even though we have implicitly relied on both fields since elementary school
(for comparison, the field of rational numbers \(\mathbb{Q}\) is a topological field, but it is not complete).
A final note on notation is in order. In applied mathematics and machine learning contexts, \(L_1, L_2,\) and \(L_\infty\) are
frequently written with subscripts instead of superscripts (\(L^1, L^2, L^\infty\)). This variation stems from differing
conventional focuses across disciplines, but both notations refer to exactly the same mathematical concepts and spaces.
Normed Spaces & Banach Spaces
To perform calculus or optimization on a vector space, we need a notion of "distance" to define limits and convergence.
A norm provides this by measuring the "length" of a vector.
Crucially, any norm naturally induces a metric (distance function)
defined by \(d(x, y) = \|x - y\|\). Once we have a metric, we can define
Cauchy sequences and ask whether the space is
complete.
Definition: Normed Space (Normed Vector Space)
A normed space is a pair \((\mathcal{X}, \|\cdot\|)\), where \(\mathcal{X}\) is a vector space
over \(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\) and \(\|\cdot\|: \mathcal{X} \to [0, \infty)\) is a
norm, meaning that for all \(x, y \in \mathcal{X}\) and \(\alpha \in \mathbb{F}\):
- (Positive definiteness) \(\|x\| = 0\) if and only if \(x = 0\).
- (Homogeneity) \(\|\alpha x\| = |\alpha|\,\|x\|\).
- (Triangle inequality) \(\|x + y\| \leq \|x\| + \|y\|\).
Definition: Banach Space
A Banach space is a normed space that is complete with respect to the
metric defined by the norm.
Equivalently, by the
Cauchy criterion for completeness,
every Cauchy sequence in the space converges to a limit that is also within the space.
The most important examples of Banach spaces in analysis and machine learning are the \(L^p\) spaces.
Thanks to the rigorous foundation of Lebesgue integration, these spaces of functions are guaranteed to be complete.
Definition: \(L^p\) Spaces
For \(1 \leq p \lt \infty\) and a measure space \((\Omega, \mathcal{F}, \mu)\), the space \(L^p(\Omega, \mu)\) consists of
(equivalence classes, modulo a.e. equality, of) measurable functions \(f\) for which \(\int_\Omega |f|^p \, d\mu \lt \infty\).
The norm is defined as:
\[
\|f\|_p = \left( \int_\Omega |f(x)|^p \, d\mu \right)^{1/p}.
\]
The triangle inequality is Minkowski's inequality, and it is what makes \(\|\cdot\|_p\) a norm. Both this inequality and the
completeness of \(L^p\) are established when these spaces are constructed in full.
Definition: \(L^\infty\) Space
The space \(L^\infty(\Omega, \mu)\) consists of (equivalence classes, modulo a.e. equality, of)
measurable functions \(f\) that are essentially bounded. The norm is the
essential supremum:
\[
\|f\|_\infty = \operatorname{ess\,sup}_{x} |f(x)| = \inf \{ C \geq 0 : |f(x)| \leq C \text{ a.e.} \}.
\]
Inner Product Spaces & Hilbert Spaces
While Banach spaces allow us to measure lengths and distances, they lack the geometric concepts of "angles" and "orthogonality."
To recover the rich geometry of Euclidean space in abstract settings, we need an
inner product. Every inner product naturally induces a
candidate norm via \(\|x\| := \sqrt{\langle x, x \rangle}\). The candidate is indeed a norm, and in particular satisfies the
triangle inequality, by the Cauchy-Schwarz inequality proved below. Hence every inner product space is automatically a normed
space.
Definition: Inner Product Space
An inner product space is a vector space \(\mathcal{X}\) over \(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\)
equipped with an inner product \(\langle \cdot, \cdot \rangle: \mathcal{X} \times \mathcal{X} \to \mathbb{F}\)
satisfying, for all \(x, y, z \in \mathcal{X}\) and \(\alpha, \beta \in \mathbb{F}\):
- (Conjugate symmetry) \(\langle x, y \rangle = \overline{\langle y, x \rangle}\).
- (Linearity in the first argument) \(\langle \alpha x + \beta y, z \rangle = \alpha \langle x, z \rangle + \beta \langle y, z \rangle\).
- (Positive definiteness) \(\langle x, x \rangle \geq 0\), with equality if and only if \(x = 0\).
We follow the mathematical convention of linearity in the first argument. The physics/quantum mechanics convention places
linearity in the second argument, which conjugates the first.
Just as a normed space requires completeness to become a Banach space, an inner product space requires completeness
to become the ultimate setting for functional analysis: a Hilbert space.
Definition: Hilbert Space
A Hilbert space \(\mathcal{H}\) is an inner product space that is complete
with respect to the metric \(d(x, y) = \|x - y\|\), where \(\|x\| := \sqrt{\langle x, x \rangle}\) is
the norm induced by the inner product (see the Cauchy-Schwarz inequality below for the proof that this
is a norm).
Hilbert spaces retain more of the geometric structure of \(\mathbb{R}^n\) than general Banach spaces do, and that extra structure
makes them the most tractable infinite-dimensional spaces. What survives includes orthogonality, projection, and self-duality. The
foundational inequality that connects the inner product to the norm is the Cauchy-Schwarz inequality, which is vital for proving
bounds in optimization.
Theorem: Cauchy-Schwarz Inequality
For all \(x, y\) in an inner product space, \( |\langle x, y \rangle| \leq \|x\| \|y\| \).
Proof:
The case \(y = 0\) is trivial. For \(y \neq 0\), set \(t = \langle x, y \rangle / \langle y, y \rangle\). Expanding by the inner product axioms,
\[
\begin{align*}
0 \leq \|x - ty\|^2 &= \|x\|^2 - t\,\overline{\langle x, y\rangle} - \bar{t}\,\langle x, y\rangle + |t|^2 \|y\|^2 \\\\
&= \|x\|^2 - \frac{|\langle x, y\rangle|^2}{\|y\|^2},
\end{align*}
\]
which rearranges to \(|\langle x, y \rangle|^2 \leq \|x\|^2 \|y\|^2\).
Cauchy-Schwarz immediately justifies the claim that \(\|x\| := \sqrt{\langle x, x\rangle}\) is a norm. Expanding
\(\|x + y\|^2 = \|x\|^2 + 2\,\mathrm{Re}\,\langle x, y\rangle + \|y\|^2 \leq \|x\|^2 + 2\|x\|\|y\| + \|y\|^2 = (\|x\| + \|y\|)^2\)
gives the triangle inequality. Positive definiteness and homogeneity are direct consequences of the inner product axioms.
But how do we know if a given Banach space is actually a Hilbert space? The answer is a single identity, and the connection
between algebra and geometry that it encodes was established by Pascual Jordan and John von Neumann. Among the \(L^p\) spaces,
only \(L^2\) satisfies that identity, which makes it the unique Hilbert space in the \(L^p\) family.
Theorem: Jordan-von Neumann Characterization
The norm of a normed space is induced by an inner product if and only if it satisfies the
Parallelogram Law:
\[
\|x + y\|^2 + \|x - y\|^2 = 2\|x\|^2 + 2\|y\|^2.
\]
In particular, a Banach space is a Hilbert space if and only if its norm satisfies this identity.
Proof Sketch:
The forward direction follows from expanding \(\|x \pm y\|^2 = \langle x \pm y, x \pm y \rangle\) via the inner product axioms
(sesquilinearity, which reduces to bilinearity in the real case). For the converse, one defines the candidate inner product
via the polarization identity:
\[
\begin{align*}
\langle x, y \rangle &= \tfrac{1}{4}\bigl(\|x+y\|^2 - \|x-y\|^2\bigr) \quad (\mathbb{F} = \mathbb{R}), \\\\
\langle x, y \rangle &= \tfrac{1}{4} \sum_{k=0}^{3} i^k \|x + i^k y\|^2 \quad (\mathbb{F} = \mathbb{C}),
\end{align*}
\]
One then verifies the inner product axioms. The delicate step is additivity in the first argument, which follows from the
parallelogram law applied to suitable vector combinations, extended to all scalars by continuity.
Orthonormal Sequences and Bessel's Inequality
In \(\mathbb{R}^n\), every vector admits a clean expansion against any orthonormal basis:
\(x = \sum_{k=1}^n \langle x, e_k \rangle e_k\). The infinite-dimensional analogue is one of the most powerful structural features
of Hilbert spaces, and it is the gateway to Fourier-style decompositions in abstract function spaces. We restrict throughout to
the separable case (Hilbert spaces possessing a countable dense subset), which covers all settings encountered in
this curriculum.
Definition: Orthonormal Sequence
A sequence \(\{e_n\}_{n=1}^\infty\) in a Hilbert space \(\mathcal{H}\) is orthonormal if
\[
\langle e_m, e_n \rangle = \delta_{mn} = \begin{cases} 1 & m = n, \\\\ 0 & m \neq n. \end{cases}
\]
That is, each vector has unit norm and distinct vectors are mutually orthogonal.
For any vector \(x \in \mathcal{H}\), the scalars \(\langle x, e_n \rangle\) measure the
"component" of \(x\) along each \(e_n\). The fundamental constraint relating these
coordinates to the norm of \(x\) is Bessel's inequality.
Theorem: Bessel's Inequality
Let \(\{e_n\}_{n=1}^\infty\) be an orthonormal sequence in a Hilbert space \(\mathcal{H}\). For every \(x \in \mathcal{H}\),
\[
\sum_{n=1}^\infty |\langle x, e_n \rangle|^2 \leq \|x\|^2.
\]
Proof:
Fix \(N \geq 1\) and define the partial expansion \(s_N = \sum_{k=1}^N \langle x, e_k \rangle e_k\). By orthonormality,
\[
\|s_N\|^2 = \sum_{k=1}^N |\langle x, e_k \rangle|^2.
\]
Computing the inner product of \(x - s_N\) with each \(e_j\):
\[
\begin{align*}
\langle x - s_N, e_j \rangle
&= \langle x, e_j \rangle - \sum_{k=1}^N \langle x, e_k \rangle \langle e_k, e_j \rangle \\\\
&= \langle x, e_j \rangle - \langle x, e_j \rangle = 0 \quad (1 \leq j \leq N).
\end{align*}
\]
By the conjugate linearity of the inner product in the second argument,
\[
\langle x - s_N, s_N \rangle = \sum_{j=1}^N \overline{\langle x, e_j \rangle} \langle x - s_N, e_j \rangle = 0,
\]
so \(x - s_N \perp s_N\). Applying the Pythagorean identity (a direct consequence of expanding
\(\|x\|^2 = \langle (x - s_N) + s_N, (x - s_N) + s_N \rangle\)):
\[
\|x\|^2 = \|x - s_N\|^2 + \|s_N\|^2 \geq \|s_N\|^2 = \sum_{k=1}^N |\langle x, e_k \rangle|^2.
\]
The partial sums of the non-negative series \(\sum |\langle x, e_n \rangle|^2\) are uniformly bounded by \(\|x\|^2\), so the
series converges and satisfies the stated bound.
One simple but useful consequence follows. Since the series \(\sum_n |\langle x, e_n \rangle|^2\) converges, its terms must tend
to zero. Hence \(\langle x, e_n \rangle \to 0\) for every \(x \in \mathcal{H}\). The coordinates of any fixed vector against an
orthonormal sequence "fade out" at infinity. This phenomenon reappears later as the prototypical example of weak
convergence.
Bessel's inequality holds for any orthonormal sequence, but the upper bound is achieved
with equality precisely when the sequence is "rich enough" to express every vector.
Definition: Orthonormal Basis (Separable Case)
An orthonormal sequence \(\{e_n\}_{n=1}^\infty\) in a separable Hilbert space \(\mathcal{H}\) is an orthonormal
basis (or Hilbert basis) if its closed linear span is all of \(\mathcal{H}\):
\[
\overline{\operatorname{span}\{e_n : n \geq 1\}} = \mathcal{H}.
\]
Note that this is not the same as a Hamel (algebraic) basis. Every vector is the limit of finite linear combinations
of the \(e_n\), not necessarily a finite combination itself.
Every infinite-dimensional separable Hilbert space admits an orthonormal basis. Starting from any countable dense subset, one
inductively orthonormalizes the vectors, skipping any that already lie in the closed linear span of the previously chosen ones.
The result is a countable orthonormal sequence whose closed linear span equals \(\mathcal{H}\). This is simply the
Gram-Schmidt process extended to infinite dimensions, the
same algorithm familiar from finite-dimensional linear algebra applied countably many times. In finite dimensions the construction
terminates after finitely many steps, so the sequential formulation above does not apply there.
Conversely, the existence of a countable orthonormal basis forces the space to be separable, since the rational (or
rational-complex) finite linear combinations of basis vectors form a countable dense subset. Thus, among infinite-dimensional
spaces, separability and the existence of a countable orthonormal basis are equivalent.
For an orthonormal basis, Bessel's inequality strengthens to an equality, and every vector
admits a convergent infinite expansion in its coordinates.
Theorem: Parseval's Identity (Hilbert Space)
Let \(\{e_n\}_{n=1}^\infty\) be an orthonormal basis for a separable Hilbert space \(\mathcal{H}\). Then every
\(x \in \mathcal{H}\) admits the convergent expansion
\[
x = \sum_{n=1}^\infty \langle x, e_n \rangle e_n,
\]
and its norm satisfies Parseval's identity:
\[
\|x\|^2 = \sum_{n=1}^\infty |\langle x, e_n \rangle|^2.
\]
Proof:
Let \(s_N = \sum_{k=1}^N \langle x, e_k \rangle e_k\) as in the proof of Bessel's inequality. For \(M \lt N\), orthonormality
gives
\[
\|s_N - s_M\|^2 = \sum_{k=M+1}^N |\langle x, e_k \rangle|^2.
\]
Since \(\sum_n |\langle x, e_n \rangle|^2\) converges (Bessel), its tail sums vanish, so \(\{s_N\}\) is Cauchy in
\(\mathcal{H}\). By the
Cauchy criterion for completeness,
there exists \(s \in \mathcal{H}\) with \(s_N \to s\).
We claim \(s = x\). For each fixed \(j\), the equality \(\langle s_N, e_j \rangle = \langle x, e_j \rangle\) holds for all
\(N \geq j\) (by the same orthonormality computation used for Bessel's inequality). Letting \(N \to \infty\), continuity of
the inner product in the first argument (a direct consequence of Cauchy-Schwarz) gives
\[
\langle s, e_j \rangle = \lim_{N \to \infty} \langle s_N, e_j \rangle = \langle x, e_j \rangle,
\]
so \(\langle x - s, e_j \rangle = 0\) for every \(j \geq 1\).
Hence \(x - s\) is orthogonal to every \(e_j\), and by sesquilinearity together with continuity of the inner product, to every
element of \(\overline{\operatorname{span}\{e_n\}} = \mathcal{H}\). In particular, \(\langle x - s, x - s \rangle = 0\), so
\(x = s = \sum_{n=1}^\infty \langle x, e_n \rangle e_n\).
Finally, since \(s_N \to x\) in norm, the norm itself is continuous (by the reverse triangle inequality), so
\(\|s_N\|^2 \to \|x\|^2\). Combined with the orthonormal computation \(\|s_N\|^2 = \sum_{k=1}^N |\langle x, e_k \rangle|^2\)
established above,
\[
\|x\|^2 = \lim_{N \to \infty} \sum_{k=1}^N |\langle x, e_k \rangle|^2
= \sum_{n=1}^\infty |\langle x, e_n \rangle|^2,
\]
which is Parseval's identity.
Parseval's identity is the abstract source of every "energy preservation" theorem in analysis. Plancherel's theorem for the
Fourier transform is the canonical instance. We revisit it when Fourier analysis is generalized to arbitrary Hilbert spaces.
Finite vs Infinite Dimensions
Because \(\mathbb{R}^n\) is a Hilbert space, it is easy to assume that all complete spaces behave like \(\mathbb{R}^n\).
However, the transition from finite-dimensional vectors to infinite-dimensional function spaces introduces profound topological
differences.
Theorem: Equivalence of Norms in Finite Dimensions
In a finite-dimensional vector space, all norms are
Lipschitz equivalent (and
therefore topologically equivalent). In infinite-dimensional spaces, this fails. A sequence of functions may converge in
\(L^1\) while diverging in \(L^\infty\).
Proof Sketch:
Fix a basis \(\{e_1, \ldots, e_n\}\) and identify the space with \(\mathbb{F}^n\) via coordinates, where
\(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\). Any norm \(N\) satisfies \(N(x) \leq \sum_i |x_i|\, N(e_i) \leq C \|x\|_2\) by
the Cauchy-Schwarz inequality on \(\mathbb{F}^n\) (with \(\|x\|_2 = (\sum_i |x_i|^2)^{1/2}\) and
\(C = (\sum_i N(e_i)^2)^{1/2}\)). The reverse triangle inequality
\[
|N(x) - N(y)| \leq N(x - y) \leq C \|x - y\|_2
\]
then shows \(N\) is Lipschitz-continuous with respect to the Euclidean topology.
The unit sphere \(S = \{x : \|x\|_2 = 1\}\) is compact (Heine-Borel applied to \(\mathbb{F}^n \cong \mathbb{R}^n\) or
\(\mathbb{R}^{2n}\)), so \(N\) attains a minimum \(m\) and maximum \(M\) on \(S\). Moreover \(m \gt 0\), since \(N(x) = 0\)
would force \(x = 0\) by positive definiteness, contradicting \(x \in S\). Scaling gives
\(m \|x\|_2 \leq N(x) \leq M \|x\|_2\) for all \(x\), which is Lipschitz equivalence.
The most drastic difference, however, lies in compactness.
As we established in the Heine-Borel Theorem,
in \(\mathbb{R}^n\), every closed and bounded set is compact. This guarantees that any continuous optimization problem
on a bounded region has a minimum. But as we hinted earlier, this comfort vanishes in infinite dimensions.
Theorem: Compactness of the Unit Ball
The closed unit ball \(\{x : \|x\| \leq 1\}\) in a normed space is compact if and only if the space is
finite-dimensional.
Corollary: In any infinite-dimensional space, the closed unit ball (which is certainly closed and bounded) is
never compact.
If the space is finite-dimensional, compactness follows from Heine-Borel applied to \(\mathbb{F}^n\) via the Lipschitz
equivalence of norms on finite-dimensional spaces established above. The converse rests on a subtle geometric fact known as
Riesz's Lemma, which guarantees, in any proper closed subspace, the existence of a unit vector almost
orthogonal to it. Iterating this construction yields an infinite sequence of unit vectors pairwise separated by \(\geq 1/2\),
precluding any convergent subsequence. Both directions are given in full where
Riesz's Lemma is proved.
Why does this matter? Because a closed unit ball is the prototypical "closed and bounded" set. The theorem tells us that in an
infinite-dimensional function space, simply bounding the parameters does not guarantee that an optimization algorithm will
converge to a solution inside that boundary. A sequence can wander endlessly inside a bounded ball without ever converging to a
point.
Geometry of Unit Balls & ML Applications
The choice of norm in a Banach space dictates the geometry of its
open and closed balls.
In machine learning, we frequently use norms as regularization terms to constrain our model parameters
(\(\|w\|_p \leq C\)). The geometric shape of these unit balls perfectly explains why different regularizers produce
different types of models.
Geometry of \(L^p\) Unit Balls in \(\mathbb{R}^2\)
- \(L^1\) norm (Lasso):
The unit ball is a diamond (a square rotated by 45 degrees). It has sharp corners precisely on the axes.
- \(L^2\) norm (Ridge):
The unit ball is a perfect circle. It is strictly convex and rotationally invariant.
- \(L^\infty\) norm:
The unit ball is a square aligned with the axes.
When we optimize a loss function subject to an \(L^1\) penalty (Lasso regression), the expanding contour of the loss function
is highly likely to hit the "sharp corners" of the \(L^1\) diamond first. Because these corners lie exactly on the axes,
many parameter weights become exactly zero. This geometric property of the \(L^1\) Banach space is the mathematical mechanism
behind Sparsity and feature selection in ML.
Conversely, the \(L^2\) Hilbert space penalty (Ridge regression) has a perfectly round, strictly convex unit ball.
The loss contour will touch it at a tangent, smoothly shrinking all parameters but rarely setting them to exactly zero.
The strict convexity of \(L^2\) guarantees that the optimization problem has a unique global solution,
making it mathematically highly stable.
Finally, when we leverage the inner product structure of Hilbert spaces, we can project data into infinite-dimensional spaces
to find linear boundaries for non-linear data. This is the foundation of the
Kernel Trick and
Reproducing Kernel Hilbert Spaces (RKHS), which power
Support Vector Machines (SVMs) and
Gaussian processes.
From Spaces to Functionals: Why "Functional" Analysis?
We have spent this entire page defining the "containers": Banach spaces and Hilbert spaces. These are the stages upon which
mathematics takes place. But treating a function \(f\) merely as a "point" in a space is not enough. To do analysis, we need to
measure these points.
In classical calculus, we input a number \(x\) and get a number \(y = f(x)\). In Functional Analysis, the paradigm shifts:
we input a function \(f\) (a point in our function space) and output a scalar
(a number in the underlying field \(\mathbb{R}\) or \(\mathbb{C}\)).
Definition: Functional
A functional on a vector space \(V\) over \(\mathbb{F}\) is a mapping
\[
\varphi: V \to \mathbb{F} \quad (\mathbb{F} = \mathbb{R} \text{ or } \mathbb{C}).
\]
Definition: Linear Functional
A functional \(\varphi: V \to \mathbb{F}\) is called a linear functional if
\[
\varphi(\alpha x + \beta y) = \alpha \varphi(x) + \beta \varphi(y)
\]
for all \(x, y \in V\) and \(\alpha, \beta \in \mathbb{F}\).
Why is this shift so important? Because functionals are the "measurement tools" of the space. They can be broadly categorized
into nonlinear and linear tools:
- The Norm (Nonlinear):
The map \(f \mapsto \|f\|\) takes a function and returns a scalar representing its "size" or "length."
- The Loss Function (Nonlinear):
In Machine Learning, the objective function \(\mathcal{L}(\theta)\) evaluates a model (a function defined by parameters \(\theta\))
and returns a scalar "error."
- Integration (Linear):
The map \(f \mapsto \int f(x) dx\) takes a function and returns a scalar representing its "area." The linearity of the integral
is fundamental to measure theory.
- Evaluation (Linear):
The map \(\delta_x(f) = f(x)\) evaluates a function at a specific point \(x\), whenever that evaluation is well-defined. For
\(L^p\) spaces, whose elements are a.e.-equivalence classes, well-definedness is subtle, and recovering it is one motivation
for RKHS.
The Path to Dual Spaces and Manifolds
This concept bridges directly to the next phase of our curriculum. In infinite-dimensional spaces, we are specifically interested in
measurement tools that preserve both the algebraic structure (linearity) and the topological structure (continuity). If we collect
all the continuous linear functionals for a specific space \(V\), we form a new space called the
topological dual space, denoted \(V^*\).
Understanding the profound relationship between a space and its dual is the absolute prerequisite for:
- Quantum Mechanics:
The "Bra-ket" notation \(\langle \phi | \psi \rangle\) is the action of a dual vector (the Bra \(\langle \phi |\) in \(V^*\))
acting on a state vector (the Ket \(| \psi \rangle\) in \(V\)). The physics convention makes this pairing conjugate-linear in
\(\phi\) and linear in \(\psi\). Under our mathematical convention of first-slot linearity, the two are related by
\(\langle \phi | \psi \rangle_{\text{phys}} = \langle \psi, \phi \rangle_{\text{math}}\).
- Optimization & The Riesz Representation:
In calculus, the true derivative of a function is a differential \(df\) (an element of the dual space). We will see in the
Riesz Representation Theorem that the "gradient" \(\nabla f\) is exactly this dual element "converted" back
into a regular vector via the space's inner product, allowing us to perform gradient descent updates.
- Geometric Deep Learning:
In smooth manifolds, a "Tangent Vector" represents a directional derivative (movement), while a "Cotangent Vector" (a functional)
represents the measurement of that change.
We have built the spaces. The pages ahead study the Operators that transform them and the
Functionals that measure them.