From Matrices to Operators
Recall that in linear algebra we studied
linear transformations
\(T: \mathbb{R}^n \to \mathbb{R}^m\). We learned that a transformation is linear if it preserves addition and scalar
multiplication:
\[
T(\alpha x + \beta y) = \alpha T(x) + \beta T(y).
\]
In functional analysis, we keep this exact same algebraic definition, but we change the stage from
finite-dimensional vectors (\(\mathbb{R}^n\)) to infinite-dimensional function spaces (like Banach and Hilbert
spaces). We call these maps linear operators.
The Infinite-Dimensional Trap
Here is the critical difference. In finite dimensions (matrices), every linear map is "well-behaved." It is automatically
continuous, and a small change in input always results in a small change in output.
In infinite dimensions, this is no longer true. A linear operator can be "wild" and discontinuous.
Counterexample: The Derivative Operator
Let \(\mathcal{X} = C^1([0, \pi])\) be the space of continuously differentiable functions, and \(\mathcal{Y} = C^0([0, \pi])\) be the space of
continuous functions. Equip both spaces with the supremum norm: \(\|f\|_\infty = \sup_{t \in [0, \pi]} |f(t)|\)
(which equals the maximum for continuous functions on a closed interval).
Consider the differentiation operator \(T(f) = f'\).
Let \(f_k(t) = \sin(kt)\) for integers \(k \geq 1\). The "size" of the input is always \(\|f_k\|_\infty = 1\), regardless of \(k\)
(the maximum value \(1\) is attained at \(t = \pi/(2k) \in [0, \pi]\)).
However, applying the operator gives:
\[
T(f_k) = \frac{d}{dt}\sin(kt) = k\cos(kt)
\]
The size of the output is \(\|Tf_k\|_\infty = k\). As \(k \to \infty\), the output explodes to infinity even
though the input stays bounded. Thus, under the supremum norm, the differentiation operator is unbounded.
The role of the norm. For a fixed mapping rule, boundedness depends on the chosen norms. If we change the norm on
the input space \(\mathcal{X}\) to \(\|f\|_{C^1} = \|f\|_\infty + \|f'\|_\infty\), then
\[
\|Tf\|_\infty = \|f'\|_\infty \leq \|f\|_\infty + \|f'\|_\infty = \|f\|_{C^1}.
\]
Suddenly, the exact same mapping rule becomes bounded (with \(M=1\)).
Because calculus and optimization require continuity, we must restrict our focus to the class of operators that behave well:
bounded linear operators.
Boundedness & Continuity
In Intro to Functional Analysis, we established that Banach and Hilbert spaces
are the proper "stages" for doing calculus in infinite dimensions. Now, we need to identify the "actors": the maps between these
spaces that are well-behaved enough for analysis. The key idea is deceptively simple. We want operators that do not amplify inputs
by an arbitrarily large factor.
Definition: Bounded Linear Operator
Let \(\mathcal{X}\) and \(\mathcal{Y}\) be normed spaces. A linear operator \(T: \mathcal{X} \to \mathcal{Y}\)
is called bounded if there exists a constant \(M \geq 0\) such that
\[
\|T(x)\|_{\mathcal{Y}} \leq M \|x\|_{\mathcal{X}} \quad \text{for all } x \in \mathcal{X}.
\]
In other words, \(T\) has a finite "maximum gain." It cannot stretch any input by more than a factor of \(M\).
This definition captures an analogy from engineering. We may think of a bounded operator as an amplifier with a finite gain knob.
No matter what signal we feed in, the output power is controlled. The derivative operator from
the previous section, by contrast, is an amplifier with no gain limit. Feeding in
higher-frequency signals produces arbitrarily large outputs.
The following theorem is arguably the most important result on this page. It tells us that for linear maps, "bounded" and
"continuous" are exactly the same concept. The equivalence links two different kinds of property. Boundedness is
a metric condition, while continuity is a topological one.
Theorem: Bounded \(\Longleftrightarrow\) Continuous
Let \(T: \mathcal{X} \to \mathcal{Y}\) be a linear operator between normed spaces. The following are equivalent:
(a) \(T\) is continuous at every point of \(\mathcal{X}\).
(b) \(T\) is continuous at 0.
(c) \(T\) is bounded.
Proof Sketch:
(a) \(\Rightarrow\) (b) is trivial (continuity everywhere implies continuity at \(0\)).
(b) \(\Rightarrow\) (c):
Since \(T\) is linear, \(T(0) = 0\). Continuity at \(0\) means that for every
\(\varepsilon \gt 0\) there exists \(\delta \gt 0\) such that \(\|x\| \lt \delta \implies \|Tx\| \lt \varepsilon\). Choosing
\(\varepsilon = 1\) (the specific value does not matter), we obtain \(\delta \gt 0\) such that
\(\|x\| \lt \delta \implies \|Tx\| \lt 1\). For any nonzero \(x \in \mathcal{X}\), the vector
\(z = \frac{\delta}{2} \cdot \frac{x}{\|x\|}\) satisfies \(\|z\| \lt \delta\), so \(\|Tz\| \lt 1\). By linearity:
\[
\left\| T\!\left(\frac{\delta}{2} \cdot \frac{x}{\|x\|}\right) \right\| = \frac{\delta}{2\|x\|} \|Tx\| \lt 1
\implies \|Tx\| \lt \frac{2}{\delta}\|x\|.
\]
For \(x = 0\), both sides of the boundedness inequality vanish, so it holds trivially. Thus \(T\) is bounded with
\(M = 2/\delta\) (any \(M \geq 2/\delta\) also works).
(c) \(\Rightarrow\) (a):
If \(\|Tx\| \leq M\|x\|\) for all \(x\), then by linearity, for any
\(x, y \in \mathcal{X}\),
\[
\|Tx - Ty\| = \|T(x - y)\| \leq M\|x - y\|.
\]
This is the Lipschitz condition with constant \(M\), so \(T\) is
Lipschitz continuous, hence
continuous, everywhere.
Connection to Forward Pass Stability
A typical fully connected layer of a neural network composes an affine map \(x \mapsto \mathbf{W}x + \mathbf{b}\) with a
nonlinear activation. Its linear part \(x \mapsto \mathbf{W}x\) is a bounded linear operator from \(\mathbb{R}^n\) to
\(\mathbb{R}^m\), and its "gain" is precisely its operator norm (the bias shifts outputs without changing the stretch).
When we pass inputs forward through many layers, and the activations are 1-Lipschitz, the total
stretch of the signal is bounded by the product of the individual gains. Keeping these operator norms below a prescribed
constant (often 1) is the mathematical foundation of Lipschitz-constrained networks, which are widely used to
stabilize the training of Generative Adversarial Networks (GANs) and to certify robustness against adversarial attacks.
Finite Dimensions: Automatic Boundedness
Recall from Intro to Functional Analysis that in finite-dimensional spaces,
all norms are Lipschitz equivalent. Combined with a basis expansion, this yields the following consequence:
Theorem: Every Linear Operator on Finite-Dimensional Spaces is Bounded
If \(\mathcal{X}\) is a finite-dimensional normed space, then every linear operator \(T: \mathcal{X} \to \mathcal{Y}\)
(for any normed space \(\mathcal{Y}\)) is bounded.
Proof:
Fix a basis \(\{e_1, \ldots, e_n\}\) of \(\mathcal{X}\) and identify \(\mathcal{X}\) with \(\mathbb{F}^n\) via coordinates,
where \(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\). For any \(x = \sum_i x_i e_i\), linearity of \(T\) and the triangle
inequality in \(\mathcal{Y}\) give
\[
\begin{align*}
\|T x\|_{\mathcal{Y}} = \Bigl\|\sum_i x_i T(e_i)\Bigr\|_{\mathcal{Y}}
&\leq \sum_i |x_i|\, \|T(e_i)\|_{\mathcal{Y}} \\\\
&\leq \Bigl(\sum_i \|T(e_i)\|_{\mathcal{Y}}^2\Bigr)^{1/2} \|x\|_2
\end{align*}
\]
by Cauchy-Schwarz on \(\mathbb{F}^n\), where \(\|x\|_2 = (\sum_i |x_i|^2)^{1/2}\).
Hence \(T\) is bounded under the Euclidean norm on \(\mathcal{X}\). By the
equivalence of norms on finite-dimensional spaces,
\(T\) is therefore bounded under any norm on \(\mathcal{X}\).
This is why the question of "is this matrix continuous?" never arises in basic linear algebra, where it is automatically true.
Boundedness only becomes a nontrivial, essential constraint in infinite-dimensional spaces.
Finite dimension buys a second structural property, this time about the position of a subspace inside its ambient space.
However large \(\mathcal{X}\) is, its finite-dimensional subspaces can never fail to be closed.
Theorem: Finite-Dimensional Subspaces are Complete and Closed
Let \(\mathcal{X}\) be a normed space over \(\mathbb{F} \in \{\mathbb{R}, \mathbb{C}\}\) and let \(M \subseteq \mathcal{X}\)
be a finite-dimensional linear subspace. Then \(M\) is complete, and \(M\) is closed in \(\mathcal{X}\).
Proof:
If \(M = \{0\}\) both claims are immediate, so assume \(\dim M = n \geq 1\) and fix a basis \(\{e_1, \ldots, e_n\}\)
of \(M\). Let \(\Phi: \mathbb{F}^n \to M\) be the coordinate map \(\Phi(a) = \sum_{i=1}^n a_i e_i\), which is a linear
bijection. Then \(a \mapsto \|\Phi(a)\|\) is a norm on \(\mathbb{F}^n\), its definiteness coming from the injectivity
of \(\Phi\). Since \(\mathbb{F}^n\) is finite-dimensional, the
equivalence of norms on finite-dimensional spaces
makes this norm Lipschitz equivalent to the Euclidean norm, so there are constants \(0 \lt c \leq C\) with
\[
c\|a\|_2 \leq \|\Phi(a)\| \leq C\|a\|_2 \quad \text{for all } a \in \mathbb{F}^n.
\]
Let \(\{m_k\}\) be a Cauchy sequence in \(M\) and set \(a^{(k)} = \Phi^{-1}(m_k)\). Given \(\varepsilon \gt 0\), the
\(\varepsilon\)-\(N\) form
of the Cauchy condition supplies \(N\) with \(\|m_j - m_k\| \lt c\,\varepsilon\) for all \(j, k \geq N\). Since
\(m_j - m_k = \Phi(a^{(j)} - a^{(k)})\), the lower bound gives \(\|a^{(j)} - a^{(k)}\|_2 \lt \varepsilon\) for those
indices, so \(\{a^{(k)}\}\) is Cauchy in \(\mathbb{F}^n\) and converges to some \(a \in \mathbb{F}^n\) (as elsewhere on
this page, the completeness of \(\mathbb{F}\), and with it that of \(\mathbb{F}^n\) coordinatewise, is taken as given).
The upper bound then gives
\[
\|m_k - \Phi(a)\| = \|\Phi(a^{(k)} - a)\| \leq C \|a^{(k)} - a\|_2 \to 0,
\]
so \(m_k \to \Phi(a) \in M\). Every Cauchy sequence in \(M\) converges in \(M\), so \(M\) is complete by the
Cauchy criterion for completeness.
Being
complete
means being closed in every metric superspace. Taking that superspace to be \(\mathcal{X}\) gives
\(\partial M \subseteq M\), which is precisely what it means for \(M\) to be
closed
in \(\mathcal{X}\).
Riesz's Lemma and the Geometry of Infinite-Dimensional Balls
The finite-dimensional comfort we have been relying on consists of compact unit balls, automatic boundedness, and equivalence of
norms. All of it traces back to one geometric fact, the compactness of the Euclidean unit ball of \(\mathbb{F}^n\) (Heine-Borel).
In fact, compactness of the unit ball characterizes finite-dimensional spaces. The reverse direction of this characterization is
subtle and requires a careful geometric construction. The key is the following lemma, which shows that, given any proper closed
subspace, one can always find a unit vector of the whole space "almost orthogonal" to the entire subspace.
Theorem (Riesz's Lemma)
Let \(\mathcal{X}\) be a normed space and \(M \subsetneq \mathcal{X}\) a closed proper subspace. Then for
every \(\varepsilon \in (0, 1)\), there exists a unit vector \(x_\varepsilon \in \mathcal{X}\) with \(\|x_\varepsilon\| = 1\)
such that
\[
\operatorname{dist}(x_\varepsilon, M) = \inf_{m \in M} \|x_\varepsilon - m\| \geq 1 - \varepsilon.
\]
Proof:
Since \(M\) is proper, pick any \(y \in \mathcal{X} \setminus M\). Because \(M\) is closed,
\(d := \operatorname{dist}(y, M) \gt 0\) (otherwise \(y\) would lie in the closure, hence in \(M\)).
By the definition of infimum, for any \(\varepsilon \in (0, 1)\) we can choose \(m_0 \in M\) with
\[
d \leq \|y - m_0\| \leq \frac{d}{1 - \varepsilon}.
\]
Set
\[
x_\varepsilon = \frac{y - m_0}{\|y - m_0\|}.
\]
Then \(\|x_\varepsilon\| = 1\) by construction.
For any \(m \in M\), using the fact that \(m_0 + \|y - m_0\| \cdot m \in M\) (since \(M\) is a subspace),
\[
\begin{align*}
\|x_\varepsilon - m\|
&= \frac{\|y - m_0 - \|y - m_0\| \cdot m\|}{\|y - m_0\|} \\\\
&= \frac{\|y - (m_0 + \|y - m_0\|\,m)\|}{\|y - m_0\|} \\\\
&\geq \frac{d}{\|y - m_0\|} \\\\
&\geq 1 - \varepsilon.
\end{align*}
\]
Taking the infimum over \(m \in M\) yields \(\operatorname{dist}(x_\varepsilon, M) \geq 1 - \varepsilon\).
Riesz's Lemma lets us iterate. In an infinite-dimensional space, we can build an infinite sequence of unit vectors, each separated
from the span of its predecessors by distance at least \(1/2\). No subsequence can be Cauchy, let alone convergent. This is the
content of the following characterization, which we first glimpsed in
Intro to Functional Analysis.
Theorem: Compact Unit Ball \(\Longleftrightarrow\) Finite-Dimensional
The closed unit ball \(\overline{B}_1 = \{x \in \mathcal{X} : \|x\| \leq 1\}\) of a normed space
\(\mathcal{X}\) is compact if and only if \(\mathcal{X}\) is finite-dimensional.
Proof:
(\(\Leftarrow\)) Finite-dimensional \(\Rightarrow\) compact unit ball.
If \(\mathcal{X} = \{0\}\), the ball is a single point. Otherwise let \(n = \dim \mathcal{X} \geq 1\),
fix a basis, and let \(\Phi: \mathbb{F}^n \to \mathcal{X}\) be the coordinate map. As in the proof that
finite-dimensional subspaces are closed
(applied with \(M = \mathcal{X}\)), the equivalence of norms gives constants \(0 \lt c \leq C\) with
\(c\|a\|_2 \leq \|\Phi(a)\| \leq C\|a\|_2\) for all \(a \in \mathbb{F}^n\). The set
\(K = \{a \in \mathbb{F}^n : \|\Phi(a)\| \leq 1\}\) is closed, since
\(\bigl|\|\Phi(a)\| - \|\Phi(b)\|\bigr| \leq C\|a - b\|_2\) makes \(a \mapsto \|\Phi(a)\|\) continuous, and
bounded, since it lies in the Euclidean ball of radius \(1/c\). Hence \(K\) is compact by the
Heine-Borel theorem
in \(\mathbb{F}^n \cong \mathbb{R}^n\) (real case) or \(\mathbb{R}^{2n}\) (complex case). The same bound
\(\|\Phi(a) - \Phi(b)\| \leq C\|a - b\|_2\) makes \(\Phi\) continuous, so
\(\overline{B}_1 = \Phi(K)\) is compact as a continuous image of a compact set.
(\(\Rightarrow\)) Compact unit ball \(\Rightarrow\) finite-dimensional. We prove the contrapositive. If
\(\mathcal{X}\) is infinite-dimensional, then \(\overline{B}_1\) is not compact.
Construct a sequence \(\{x_n\} \subset \overline{B}_1\) with no convergent subsequence, as follows. Pick any unit vector
\(x_1\) and let \(M_1 = \operatorname{span}\{x_1\}\), which is finite-dimensional and therefore a
closed
proper subspace of \(\mathcal{X}\). By Riesz's Lemma with \(\varepsilon = 1/2\), choose a
unit vector \(x_2\) with \(\operatorname{dist}(x_2, M_1) \geq 1/2\).
Inductively, let \(M_n = \operatorname{span}\{x_1, \dots, x_n\}\), again finite-dimensional and hence closed, and choose
\(x_{n+1}\) on the unit sphere with
\(\operatorname{dist}(x_{n+1}, M_n) \geq 1/2\). Since \(\mathcal{X}\) is infinite-dimensional, \(M_n \subsetneq \mathcal{X}\)
at every stage, so the construction never terminates.
The resulting sequence satisfies \(\|x_m - x_n\| \geq 1/2\) for all \(m \neq n\) (since for \(m \lt n\), \(x_m \in M_{n-1}\)
while \(x_n\) is chosen at distance at least \(1/2\) from \(M_{n-1}\)). No subsequence is Cauchy, so no subsequence converges.
Therefore \(\overline{B}_1\) is not
sequentially compact, and hence not
compact.
Why This Matters: The Failure of Brute-Force Optimization
In finite-dimensional optimization (standard machine learning with a fixed-size parameter vector), compactness is
available whenever the parameters are confined to a closed and bounded set. A continuous loss function on a nonempty
closed and bounded set attains its minimum.
This guarantee evaporates in infinite-dimensional function space optimization: training a neural network of unbounded width,
finding the optimal function in a reproducing kernel Hilbert space, or solving a variational problem. A bounded sequence of
candidate solutions may fail to have any convergent subsequence, meaning the "optimal" function may fail to exist in the space
at all. This is why infinite-dimensional optimization requires genuinely new tools: weak convergence, convexity, and carefully
chosen function space topologies. None of these are needed to prove existence of a minimizer
on a closed bounded set in finite dimensions.
The Operator Norm
We know that a bounded operator has a finite "maximum gain." The operator norm captures this by taking the
supremum stretch over all nonzero inputs. As we show below, this supremum is equivalently the smallest constant \(M\)
that witnesses boundedness.
Definition: The Operator Norm
Let \(T: \mathcal{X} \to \mathcal{Y}\) be a bounded linear operator between normed spaces. The operator norm
of \(T\) is:
\[
\|T\| = \sup_{\substack{x \in \mathcal{X} \\ x \neq 0}} \frac{\|Tx\|_\mathcal{Y}}{\|x\|_\mathcal{X}}
\quad (\mathcal{X} \neq \{0\}),
\]
with the convention \(\|T\| := 0\) in the degenerate case \(\mathcal{X} = \{0\}\) (for which the only linear operator is
\(T = 0\)).
Intuitively, \(\|T\|\) answers the question: "What is the maximum factor by which \(T\) can amplify any input?" This
supremum is taken over all nonzero inputs, so it finds the "worst-case stretch."
The following equivalent characterizations are used constantly in practice. Each captures the same idea
from a slightly different geometric perspective:
Proof:
All three equalities rely on the linearity of \(T\) and the homogeneity of the norm. For any \(\lambda \gt 0\) we have
\[
\|T(\lambda x)\| = \lambda \|Tx\|
\]
and \(\|\lambda x\| = \lambda \|x\|\), so the ratio \(\|Tx\|/\|x\|\) is invariant under positive scaling.
Sphere = closed ball:
For any \(x \neq 0\), write \(\hat{x} = x / \|x\|\). Then
\(\|T\hat{x}\| = \|Tx\|/\|x\|\), so the supremum of \(\|Tx\|/\|x\|\) over \(x \neq 0\) equals \(\sup_{\|y\|=1} \|Ty\|\). For
the closed ball, the sphere is a subset, so \(\sup_{\|x\| \leq 1} \|Tx\| \geq \sup_{\|y\|=1} \|Ty\|\). Conversely, for
\(0 \lt \|x\| \leq 1\) we have \(\|Tx\| = \|x\|\,\|T\hat{x}\| \leq \|T\hat{x}\| \leq \sup_{\|y\|=1} \|Ty\|\), and \(x = 0\)
contributes \(\|Tx\| = 0\), which gives the reverse inequality. The three expressions coincide.
Smallest bounding constant:
Any \(M\) with \(\|Tx\| \leq M\|x\|\) for all \(x\)
satisfies \(\|Tx\|/\|x\| \leq M\) for \(x \neq 0\), hence
\(\sup_{x \neq 0} \|Tx\|/\|x\| \leq M\). Taking infimum over admissible \(M\) gives
\(\|T\| \leq \inf\{M : \ldots\}\). Conversely, \(\|T\|\) itself is an admissible \(M\) by definition,
so \(\inf\{M : \ldots\} \leq \|T\|\).
Example: Operator Norm of a Matrix
For a matrix \(\mathbf{A} \in \mathbb{R}^{m \times n}\) acting on
\((\mathbb{R}^n, \|\cdot\|_2) \to (\mathbb{R}^m, \|\cdot\|_2)\), the operator norm induced by the \(L^2\)
norm equals the largest singular value:
\[
\|\mathbf{A}\|_{\text{op}} = \sigma_{\max}(\mathbf{A}) = \sqrt{\lambda_{\max}(\mathbf{A}^\top\mathbf{A})}.
\]
This result is specific to the \(L^2\)-induced norm. If we instead equip both spaces with the \(L^1\) norm, the operator norm
becomes the maximum absolute column sum. With the \(L^\infty\) norm on both, it becomes the maximum absolute row sum. The
choice of norm fundamentally determines the operator's "gain," exactly as in
our derivative operator example.
This connects directly to the
Singular Value Decomposition (SVD). The
SVD decomposes \(\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top\). Because orthogonal matrices are
isometries under the \(L^2\) norm (they preserve lengths, so
\(\|\mathbf{U}\|_{\text{op}} = \|\mathbf{V}^\top\|_{\text{op}} = 1\)), they contribute no "stretch" to the composition.
By submultiplicativity,
\(\|\mathbf{A}\|_{\text{op}} \leq \|\mathbf{U}\|_{\text{op}}\|\boldsymbol{\Sigma}\|_{\text{op}}\|\mathbf{V}^\top\|_{\text{op}} = \|\boldsymbol{\Sigma}\|_{\text{op}}\).
Conversely, orthogonality gives \(\boldsymbol{\Sigma} = \mathbf{U}^\top \mathbf{A} \mathbf{V}\), so the same argument yields
\(\|\boldsymbol{\Sigma}\|_{\text{op}} \leq \|\mathbf{A}\|_{\text{op}}\). Hence
\(\|\mathbf{A}\|_{\text{op}} = \|\boldsymbol{\Sigma}\|_{\text{op}}\), and the latter is simply the largest diagonal entry
(since \(\boldsymbol{\Sigma}\) scales each basis vector independently).
In this light, for operators between Hilbert spaces, the operator norm is the infinite-dimensional
generalization of the largest singular value.
Key Properties of the Operator Norm
The operator norm satisfies the standard axioms of a norm (non-negativity, homogeneity, triangle inequality),
but it also has a crucial submultiplicativity property that is essential for studying compositions.
Proposition: Operator Norm Axioms
For bounded linear operators \(S, T: \mathcal{X} \to \mathcal{Y}\) and scalar \(\alpha \in \mathbb{F}\):
- (Positive definiteness) \(\|T\| \geq 0\), with equality if and only if \(T = 0\).
- (Homogeneity) \(\|\alpha T\| = |\alpha|\,\|T\|\).
- (Triangle inequality) \(\|S + T\| \leq \|S\| + \|T\|\).
Proof:
(1) \(\|T\| \geq 0\) as a supremum of non-negative quantities. If \(\|T\| = 0\), then
\(\|Tx\|_{\mathcal{Y}} \leq 0 \cdot \|x\|_{\mathcal{X}} = 0\) for all \(x\), hence \(Tx = 0\), that is, \(T = 0\). The
converse is immediate.
(2) \(\|\alpha T\| = \sup_{x \neq 0} \|\alpha Tx\|/\|x\| = |\alpha|\,\sup_{x \neq 0} \|Tx\|/\|x\| = |\alpha|\,\|T\|\)
by homogeneity of the norm on \(\mathcal{Y}\).
(3) For any \(x\),
\(\|(S + T)x\|_{\mathcal{Y}} \leq \|Sx\|_{\mathcal{Y}} + \|Tx\|_{\mathcal{Y}} \leq (\|S\| + \|T\|)\,\|x\|_{\mathcal{X}}\), so
\(S + T\) is bounded with bounding constant \(\|S\| + \|T\|\). The smallest-bounding-constant characterization gives
\(\|S + T\| \leq \|S\| + \|T\|\).
Beyond these basic axioms, the operator norm satisfies an additional crucial property governing compositions:
Proposition: Submultiplicativity
If \(T: \mathcal{X} \to \mathcal{Y}\) and \(S: \mathcal{Y} \to \mathcal{Z}\) are bounded linear operators, then the
composition \(S \circ T: \mathcal{X} \to \mathcal{Z}\) is bounded and
\[
\|S \circ T\| \leq \|S\| \cdot \|T\|.
\]
Proof:
\(S \circ T\) is linear as a composition of linear maps. For any \(x \in \mathcal{X}\), applying the defining bound of the
operator norm twice gives
\[
\|(S \circ T)(x)\|_\mathcal{Z} = \|S(Tx)\|_\mathcal{Z} \leq \|S\| \cdot \|Tx\|_\mathcal{Y}
\leq \|S\| \cdot \|T\| \cdot \|x\|_\mathcal{X}.
\]
Hence \(S \circ T\) is bounded with bounding constant \(\|S\| \cdot \|T\|\), and by the smallest-bounding-constant
characterization of the operator norm, \(\|S \circ T\| \leq \|S\| \cdot \|T\|\).
Connection to Gradient Flow and Backpropagation
The same submultiplicativity that bounds the forward stretch also bounds the backward pass from above.
In a neural network, the Jacobian \(\mathbf{J}_\ell\) of layer \(\ell\) can be decomposed via the chain rule as
\(\mathbf{J}_\ell = \mathbf{D}_\ell \mathbf{W}_\ell\), where \(\mathbf{W}_\ell\) is the weight matrix and \(\mathbf{D}_\ell\)
is the diagonal matrix of the activation function's derivatives.
For a 1-Lipschitz activation function \(\sigma\), such as ReLU or Tanh, we have \(|\sigma'(t)| \leq 1\) wherever the
derivative exists (for ReLU, any choice of \(\sigma'(0) \in [0, 1]\) keeps the bound). The diagonal entries of
\(\mathbf{D}_\ell\) are therefore all at most 1 in absolute value, so \(\|\mathbf{D}_\ell\|_{\text{op}} \leq 1\). By
submultiplicativity, the operator norm of the Jacobian \(\mathbf{J}_L \cdots \mathbf{J}_1\) of the whole \(L\)-layer map
is bounded by:
\[
\begin{align*}
\|\mathbf{J}_L \cdots \mathbf{J}_1\|_{\text{op}}
&\leq \prod_{\ell=1}^{L} \|\mathbf{D}_\ell\|_{\text{op}} \|\mathbf{W}_\ell\|_{\text{op}} \\\\
&\leq \prod_{\ell=1}^{L} \|\mathbf{W}_\ell\|_{\text{op}}.
\end{align*}
\]
The transpose of this Jacobian, which propagates the loss gradient back to the input, has the same operator norm.
This bound has a direct architectural consequence. The operator norms of the weight matrices \(\mathbf{W}_\ell\)
control this upper bound, and with it the risk of exploding gradients. Vanishing gradients are a separate matter,
since an upper bound cannot rule out the Jacobian product \(\mathbf{J}_L \cdots \mathbf{J}_1\) becoming small (small entries
of \(\mathbf{D}_\ell\) or small singular values of \(\mathbf{W}_\ell\) can drive it toward zero). Techniques like
orthogonal initialization (\(\|\mathbf{W}_\ell\|_{\text{op}} = 1\) at the start of training) and
spectral normalization (which keeps \(\|\mathbf{W}_\ell\|_{\text{op}}\) at most about 1 during training) are
direct applications of this bound. Both put these operator norms near 1, which limits how fast the upper bound can grow with
depth, but neither prevents vanishing gradients.
The Space \(\mathcal{B}(\mathcal{X}, \mathcal{Y})\) & Completeness
In functional analysis, operators themselves form a vector space. Just as we add and scale functions pointwise,
we can add and scale operators:
\[
(S + T)(x) = S(x) + T(x), \quad (\alpha T)(x) = \alpha \cdot T(x).
\]
If we collect all bounded linear operators from \(\mathcal{X}\) to \(\mathcal{Y}\) and equip this collection with the operator
norm, we obtain a normed space of its own.
Definition: The Space \(\mathcal{B}(\mathcal{X}, \mathcal{Y})\)
Let \(\mathcal{X}\) and \(\mathcal{Y}\) be normed spaces over the same scalar field. Define
\[
\mathcal{B}(\mathcal{X}, \mathcal{Y}) = \{ T: \mathcal{X} \to \mathcal{Y} \mid T \text{ is linear and bounded} \},
\]
equipped with pointwise addition \((S + T)(x) = S(x) + T(x)\), scalar multiplication \((\alpha T)(x) = \alpha\, T(x)\),
and the operator norm \(\|T\| = \sup_{\|x\| \leq 1} \|Tx\|\).
It remains to verify that this is a well-defined normed vector space. Closure under the operations is immediate. For
\(S, T \in \mathcal{B}(\mathcal{X}, \mathcal{Y})\) and \(\alpha \in \mathbb{F}\), the triangle inequality gives
\(\|(S+T)x\| \leq \|Sx\| + \|Tx\| \leq (\|S\| + \|T\|)\|x\|\), so \(S + T\) is bounded, and similarly
\(\|(\alpha T)x\| = |\alpha|\,\|Tx\| \leq |\alpha|\,\|T\|\,\|x\|\), so \(\alpha T\) is bounded. The vector space axioms are
inherited from pointwise operations on \(\mathcal{Y}\). The norm axioms are
verified above.
This construction is a hallmark of the "functional analysis philosophy." We study maps between spaces by turning those maps
into points of a new space, and then we apply the same analytical tools (norms, convergence, completeness) to this new
space of operators.
The next theorem tells us when this space of operators is complete. The answer is that the operators inherit
completeness from their codomain.
Theorem: Completeness of \(\mathcal{B}(\mathcal{X}, \mathcal{Y})\)
If \(\mathcal{X}\) is a normed space and \(\mathcal{Y}\) is a Banach space, then \(\mathcal{B}(\mathcal{X}, \mathcal{Y})\)
is also a Banach space.
Note. The completeness of the input space \(\mathcal{X}\) is entirely irrelevant for this theorem. The operator space
\(\mathcal{B}(\mathcal{X}, \mathcal{Y})\) inherits its completeness strictly from the codomain \(\mathcal{Y}\).
Proof Sketch:
Let \(\{T_n\}\) be a Cauchy sequence in \(\mathcal{B}(\mathcal{X}, \mathcal{Y})\). For each fixed \(x \in \mathcal{X}\):
\[
\|T_n(x) - T_m(x)\|_\mathcal{Y} = \|(T_n - T_m)(x)\|_{\mathcal{Y}} \leq \|T_n - T_m\|_{\text{op}} \cdot \|x\|_{\mathcal{X}}.
\]
Since \(\{T_n\}\) is Cauchy in the operator norm, \(\{T_n(x)\}\) is a Cauchy sequence in \(\mathcal{Y}\). Because
\(\mathcal{Y}\) is complete (Banach), the
Cauchy criterion for completeness
gives a limit for \(\{T_n(x)\}\), and we define \(T(x) = \lim_{n \to \infty} T_n(x)\). Linearity of \(T\) follows from the
linearity of limits.
To see that \(T\) is bounded, recall that
Cauchy sequences are bounded,
so there exists \(M\) such that \(\|T_n\|_{\text{op}} \leq M\) for all \(n\). Therefore
\(\|T_n(x)\|_{\mathcal{Y}} \leq M\|x\|_{\mathcal{X}}\), and taking \(n \to \infty\) yields
\(\|T(x)\|_{\mathcal{Y}} \leq M\|x\|_{\mathcal{X}}\), so \(T\) is bounded.
Finally, we show \(\|T_n - T\|_{\text{op}} \to 0\). Given \(\varepsilon \gt 0\), choose \(N\) such that
\(\|T_n - T_m\|_{\text{op}} \lt \varepsilon\) for all \(n, m \geq N\). In particular,
\(\|T_n(x) - T_m(x)\|_{\mathcal{Y}} \leq \varepsilon \|x\|_{\mathcal{X}}\) for all \(x\). Letting \(m \to \infty\) and using
continuity of the norm gives \(\|T_n(x) - T(x)\|_{\mathcal{Y}} \leq \varepsilon \|x\|_{\mathcal{X}}\) for all \(x\), hence
\(\|T_n - T\|_{\text{op}} \leq \varepsilon\) for \(n \geq N\).
The Special Case: \(\mathcal{X}^* = \mathcal{B}(\mathcal{X}, \mathbb{F})\)
Definition: Bounded Linear Functional
A linear functional
\(f: \mathcal{X} \to \mathbb{F}\) is called a bounded linear functional if it is bounded as an
operator. Explicitly, this requires a constant
\(M \geq 0\) with
\[
|f(x)| \leq M \|x\|_{\mathcal{X}} \quad \text{for all } x \in \mathcal{X}.
\]
By Bounded \(\Longleftrightarrow\) Continuous, this
is equivalent to \(f\) being a continuous linear functional.
When the codomain is the scalar field \(\mathbb{F}\) (which is always complete), we get a particularly important instance of the
operator space:
\[
\mathcal{X}^* = \mathcal{B}(\mathcal{X}, \mathbb{F}).
\]
As a consequence of the Completeness Theorem above, \(\mathcal{X}^*\) is always a Banach space, regardless of whether
\(\mathcal{X}\) itself is complete. This space \(\mathcal{X}^*\) is the topological dual space of
\(\mathcal{X}\), the space of all continuous linear functionals.
We introduced functionals at the end of
Intro to Functional Analysis. The dual space will be the central
object of the next page (Dual Spaces & Riesz Representation).
Connection to Function Space Optimization
The completeness of \(\mathcal{B}(\mathcal{X}, \mathcal{Y})\) (for \(\mathcal{Y}\) a Banach space) guarantees that if a
sequence of operators is Cauchy in operator norm (the difference \(T_n - T_m\) of any two sufficiently late iterates has
arbitrarily small worst-case gain), it converges to a valid bounded operator that remains in the space.
Completeness alone does not make an optimization problem attain its optimum, but without it even a Cauchy sequence of
iterates could fail to have a limit within our working space. Completeness of \(\mathcal{B}(\mathcal{X}, \mathcal{Y})\)
plays here the same foundational role that completeness of \(L^2\) plays for function approximation.
Linear Manifold vs Manifold: A Terminological Warning
Before proceeding further, we must flag a terminological pitfall that trips up readers moving between
classical functional analysis and modern differential geometry / Geometric Deep Learning.
Classical texts in functional analysis use the term "linear manifold" to mean a subspace of a
vector space, possibly non-closed in the normed setting, and reserve the word "subspace" for a closed
linear manifold. In either case the object is flat. It passes through the origin and is closed under linear combinations.
This is a completely different meaning from the word "manifold" as used in modern differential geometry and in Geometric Deep
Learning, where a manifold is a possibly curved space that only locally looks Euclidean. The adjective "linear" in
"linear manifold" is precisely what marks the object as flat. The term has largely fallen out of use outside classical functional
analysis, so a reader trained on the modern literature will rarely encounter it. When it does appear, misreading it as a curved
manifold is an easy trap.
Definition: Manifold (Topological / Smooth)
An \(n\)-dimensional manifold is a second-countable Hausdorff topological space in which every point has a
neighborhood homeomorphic to an open subset of \(\mathbb{R}^n\) (via homeomorphisms called charts). The integer
\(n\) is the dimension. A smooth manifold is a manifold equipped with a smooth structure,
a maximal collection of charts covering it whose transition maps between overlapping charts are infinitely differentiable,
enabling calculus on curved spaces.
Key examples include spheres \(S^n\) with \(n \geq 2\) and the round metric, the
rotation group \(SO(3)\) with its
bi-invariant metric, and the space of \(n \times n\) positive definite matrices with \(n \geq 2\) and the affine-invariant
metric. Under these metrics, all of them carry intrinsic curvature.
The distinction to keep in mind:
-
"Linear manifold" (classical functional analysis): a subspace of a vector space. Flat, globally linear,
passes through the origin. Same thing as what modern texts simply call a subspace.
-
"Manifold" (modern differential geometry and GDL): a locally Euclidean topological space, potentially curved,
with no ambient vector space required. Equipped with a smooth structure and a Riemannian metric, it carries rich geometric
structure: tangent spaces, geodesics, curvature.
Why This Matters for GDL
In Geometric Deep Learning, "manifold" typically refers to non-Euclidean, often curved spaces: the sphere \(S^2\) for omnidirectional images, the
Lie group \(SO(3)\) for
3D rotations in robotics, or the statistical manifold of probability distributions for
natural gradient descent. When "linear manifold" appears in
a functional analysis text, we may mentally replace it with "subspace," the flat and algebraic concept, and
reserve "manifold" for the curved, geometric object that dominates the later portions of this curriculum.
With this clarification in place, we have now assembled the essential toolkit of bounded linear operators and their norms. On the
next page, we turn to the most important special case: when the codomain is the scalar field. These are the
continuous linear functionals, and collecting them gives us the Dual Space \(\mathcal{X}^*\).
That space is the gateway to understanding why gradients are covectors and to the Riesz Representation Theorem.