Dual Spaces & Riesz Representation

The Dual Space \(\mathcal{X}^*\)

Remember, we have already met the functional. It is a map that takes a function (or vector) and returns a scalar. Also, in Bounded Linear Operators, we saw that when the codomain is the scalar field \(\mathbb{F}\), the space \(\mathcal{X}^* = \mathcal{B}(\mathcal{X}, \mathbb{F})\) is always a Banach space, regardless of whether \(\mathcal{X}\) itself is complete.

We are now ready to study this space in full. The topological dual space \(\mathcal{X}^*\) is not merely an abstract construction. It is the space of all possible "measurements" one can perform on \(\mathcal{X}\). Every element \(\varphi \in \mathcal{X}^*\) is a continuous linear functional: a device that inputs a vector and outputs a number, while respecting both linearity and the topology of the space.

The central question of this page is: "Given a continuous linear functional \(\varphi \in \mathcal{H}^*\), can we always find a concrete vector \(y \in \mathcal{H}\) that 'represents' it via the inner product?"

The answer is the Riesz Representation Theorem. It will fundamentally reshape how we think about gradients, derivatives, and optimization on curved spaces.

Formal Construction

Since \(\mathbb{F}\) is always complete, the result from the previous page immediately gives us:

Concrete Examples of Dual Spaces

To build intuition, let us examine the dual spaces of the most important spaces in our curriculum. When we say a space \(\mathcal{Y}\) is the dual of \(\mathcal{X}\) (written \(\mathcal{X}^* \cong \mathcal{Y}\)), we mean there is a natural isometric isomorphism between them, usually given by an integral or an infinite sum.

• Finite Dimensions (\(\mathbb{R}^n\)): The dual of \(\mathbb{R}^n\) is isomorphic to \(\mathbb{R}^n\) itself. The pairing is the standard dot product: \(\varphi(x) = y^\top x\). Every linear functional is just a row vector multiplying a column vector.
• Sequence Spaces (\(\ell^p\)): For \(1 \leq p \lt \infty\), the dual of \(\ell^p\) is \(\ell^q\), where \(1/p + 1/q = 1\) (Hölder conjugates). The action of \(\varphi \in \ell^q\) on \(x \in \ell^p\) is the infinite sum \(\varphi(x) = \sum x_i y_i\). This sum converges and defines a bounded functional because \(|\sum x_i y_i| \leq \|x\|_p \cdot \|y\|_q\) (Hölder's inequality, which generalizes Cauchy-Schwarz to \(L^p\) spaces).
• Function Spaces (\(L^p\)): Similarly, \((L^p)^* \cong L^q\) for \(1 \lt p \lt \infty\), and also for \(p = 1\) when the underlying measure is \(\sigma\)-finite. The functional acts via Lebesgue integration: \(\varphi(f) = \int f(t) g(t) d\mu\), where \(g \in L^q\). The fact that every continuous functional on \(L^p\) arises in this form (the converse direction of the isomorphism) is a non-trivial result whose proof rests on the Radon-Nikodym theorem. We state the duality here as a fact, with the existence of the representing function \(g\) licensed by that theorem.
• The Asymmetry of \(L^1\) and \(L^\infty\): For a \(\sigma\)-finite measure the dual of \(L^1\) is exactly \(L^\infty\). However, the dual of \(L^\infty\) is strictly larger than \(L^1\). This asymmetry means \(L^1\) is not "reflexive" (a concept we will formalize below). The failure of reflexivity causes substantial difficulties in optimization problems involving \(L^1\) regularization.

Convention (complex scalars). The duality pairings \(\sum x_i y_i\) and \(\int f g\, d\mu\) above are bilinear and carry no complex conjugation. This is the standard way to identify \((\ell^p)^*\) with \(\ell^q\) and \((L^p)^*\) with \(L^q\) as Banach-space isomorphisms. It should not be confused with the Hilbert-space inner product on \(\ell^2\) or \(L^2\), which (under our first-slot-linear convention) carries conjugation in the second slot: \(\langle x, y\rangle = \sum x_i \overline{y_i}\). The two coincide over \(\mathbb{R}\). Over \(\mathbb{C}\) they differ by a complex conjugation on the representing element.

• Hilbert Spaces (\(\mathcal{H}\)), the Self-Dual Case: For a Hilbert space, we will prove something far stronger than Hölder duality: \(\mathcal{H}^* \cong \mathcal{H}\) via the inner product itself. Every continuous linear functional on \(\mathcal{H}\) is just "taking the inner product with some fixed vector." This is the content of the Riesz Representation Theorem, the subject of the next section.

† \(L^1\) (resp. \(\ell^1\)) embeds isometrically into \((L^\infty)^*\) (resp. \((\ell^\infty)^*\)) via integration (resp. summation) against essentially bounded functions (resp. bounded sequences), but the embedding is proper. The dual contains additional functionals corresponding to finitely additive measures, namely \(ba(\Sigma, \mu)\) for \(L^\infty(\Omega, \Sigma, \mu)\) and \(ba(2^{\mathbb{N}})\) for \(\ell^\infty\). The notation \(\supsetneq\) indicates strict containment up to this canonical embedding, not literal set inclusion. These larger duals lie beyond the scope of this page.
‡ The Hilbert pairing \(\langle x, y \rangle\) is sesquilinear (conjugate-linear in the second slot under our convention), distinct from the bilinear pairings \(\sum x_i y_i\) and \(\int fg\, d\mu\) used for Banach duality above. The two coincide over \(\mathbb{R}\). Over \(\mathbb{C}\), the Riesz map is anti-linear, so \(\mathcal{H}^* \cong \mathcal{H}\) is an isometric anti-isomorphism. A strict linear isomorphism requires the conjugate Hilbert space \(\overline{\mathcal{H}}\). Consequently the Hilbert representative differs from the Banach representative by complex conjugation.

The Riesz Representation Theorem

We now arrive at one of the central results of functional analysis. In the examples above, we saw that identifying the dual of a general Banach space can be quite intricate. The relationship between \(L^p\) and \(L^q\) depends on Hölder conjugates, and pathological cases like \((L^\infty)^*\) escape any simple description.

Hilbert spaces are the exception. Their inner product provides a canonical, isometric identification between the space and its dual. This is the content of the Riesz Representation Theorem, due to Frigyes Riesz (1907).

Let us first build the intuition. In \(\mathbb{R}^n\) with the standard dot product, every linear functional \(\varphi: \mathbb{R}^n \to \mathbb{R}\) can be written as \(\varphi(x) = y^\top x\) for a unique vector \(y \in \mathbb{R}^n\). The Riesz theorem says that this works in every Hilbert space, even infinite-dimensional ones.

Let us unpack this statement before proving it. The theorem asserts three things simultaneously:

1. Existence. No matter how abstract or exotic the functional \(\varphi\) is, there is always a concrete vector \(y_\varphi\) that reproduces it via the inner product.
2. Uniqueness. This representative vector is determined uniquely. There is no ambiguity.
3. Isometry. The "size" of \(\varphi\) as a functional equals the "size" of \(y_\varphi\) as a vector. No information is lost in the translation.

Proof

Both the decomposition below and the argument that follows it are phrased in terms of orthogonality to an entire set, so we record that notion first. It is the Hilbert-space form of the orthogonal complement met in \(\mathbb{R}^n\), and, as there, it is defined for an arbitrary subset rather than only for a subspace.

The proof relies on one key fact about Hilbert spaces that has no analogue in general Banach spaces. In Linear Algebra, we proved that every vector in \(\mathbb{R}^n\) can be uniquely decomposed as the sum of a component in a subspace \(W\) and a component in \(W^\perp\). The following theorem extends this to infinite-dimensional Hilbert spaces, where the assumption that \(M\) is closed becomes essential.

The proof of the Projection Theorem relies on two key ingredients specific to Hilbert spaces: the completeness of \(\mathcal{H}\) (to guarantee that a minimizing sequence converges) and the Parallelogram Law (to extract Cauchyness from the minimization). This is precisely where the Hilbert space structure, as opposed to mere Banach space structure, is indispensable. In a Banach space without an inner product, closest points in closed subspaces need not exist or be unique.

Two consequences of the decomposition will be needed once we turn to operators. The first says that passing from a set to its linear span costs nothing on the orthogonal side; the second says that for a closed subspace the complement operation is an involution. Closedness is essential in the second statement and absent from the first.

Closedness in part (b) is not a technical convenience but the whole content of the statement. Let \(N \subseteq \mathcal{H}\) be a dense subspace and let \(x \in N^\perp\). Density gives \(\operatorname{dist}(x, N) = 0\), so for each \(\varepsilon \gt 0\) there is \(y \in N\) with \(\|x - y\| \lt \varepsilon\). Since \(\langle x, y \rangle = 0\), \[ \|x\|^2 = \langle x, x - y \rangle + \langle x, y \rangle = \langle x, x - y \rangle, \] and Cauchy-Schwarz turns this into \(\|x\|^2 = |\langle x, x - y \rangle| \leq \|x\|\,\|x - y\| \lt \|x\|\,\varepsilon\). Were \(x\) nonzero we could divide by \(\|x\|\) and obtain \(\|x\| \lt \varepsilon\) for every \(\varepsilon \gt 0\), which is impossible; hence \(x = 0\). So \(N^\perp = \{0\}\) and \(\left(N^\perp\right)^\perp = \mathcal{H}\), which differs from \(N\) whenever the dense subspace is proper.

With the Projection Theorem in hand, we can now prove the Riesz Representation Theorem.

The Isomorphism \(\mathcal{H}^* \cong \mathcal{H}\): A Hilbert Space is Self-Dual

The Riesz theorem can be restated more compactly as the existence of a canonical map between \(\mathcal{H}\) and \(\mathcal{H}^*\).

The Riesz Representation Theorem tells us that \(\Phi\) has three structural properties:

1. \(\Phi\) is isometric: \(\|\Phi(y)\|_{\mathcal{H}^*} = \|y\|_{\mathcal{H}}\) for all \(y\).
2. \(\Phi\) is surjective: every \(\varphi \in \mathcal{H}^*\) equals \(\Phi(y_\varphi)\) for some \(y_\varphi\).
3. Over \(\mathbb{R}\), \(\Phi\) is linear. Over \(\mathbb{C}\), \(\Phi\) is conjugate-linear (anti-linear): \(\Phi(\alpha y_1 + y_2) = \overline{\alpha}\,\Phi(y_1) + \Phi(y_2)\).

Therefore, \(\Phi\) is an isometric (anti-linear) isomorphism. In the real case, it is simply an isometric isomorphism. Either way, \(\mathcal{H}\) and \(\mathcal{H}^*\) carry exactly the same norm structure.

This is the cleanest possible answer to the question we posed in the beginning of this page. While the duals of general Banach spaces can be exotic (\((L^\infty)^*\) is unwieldy), Hilbert spaces have the property that their dual is, via the Riesz isomorphism, a mirror image of themselves. The inner product provides the mirror.

Musical Isomorphisms: The \(\sharp\) and \(\flat\) Dictionary

The Riesz Representation Theorem gives us a canonical way to convert between a Hilbert space and its dual. In differential geometry, this conversion has an evocative name: the Musical Isomorphisms, denoted by the symbols \(\flat\) ("flat") and \(\sharp\) ("sharp"). Both symbols are borrowed from musical notation.

These names are not mere whimsy. They encode one of the deepest ideas in modern geometry. The inner product (or more generally, the metric tensor) serves as a "dictionary" that translates between two fundamentally different types of geometric objects.

The Riesz theorem guarantees that \(\flat\) and \(\sharp\) are inverses of each other, and both are isometries: \[ (\flat \circ \sharp) = \operatorname{id}_{\mathcal{H}^*}, \quad (\sharp \circ \flat) = \operatorname{id}_{\mathcal{H}}, \quad \|y^\flat\|_{\mathcal{H}^*} = \|y\|_{\mathcal{H}}. \]

Index Notation: Raising and Lowering Indices

We now shift perspective: from the abstract Hilbert space \(\mathcal{H}\) used so far to the local-coordinate picture on a finite-dimensional manifold \(M\). In differential geometry and physics, the Musical Isomorphisms are expressed in index notation (Einstein summation convention) using a chart-dependent basis. This notation makes the role of the metric tensor explicit and will be essential when we reach the manifold pages. (The infinite-dimensional case requires additional structure, for example a Schauder basis, and is not treated by simple index expressions like \(v^i\) or \(g_{ij}\).)

Following the physics and ML convention, we identify a vector \(v \in T_pM\) with its components \((v^i)\) in a chosen chart, and similarly for covectors \(\omega = \omega_i \, dx^i \in T_p^*M\). Strictly, \(v = v^i \partial_i\) where \(\{\partial_i\}\) is the coordinate basis. The map-level operations \(\flat : T_pM \to T_p^*M\) and \(\sharp : T_p^*M \to T_pM\) are then represented by the component formulas \(v_i = g_{ij} v^j\) and \(\omega^i = g^{ij} \omega_j\). We use the two interchangeably below, but they are formally distinct objects.

First, the two types of objects:

Second, the two maps connecting them (driven by the metric tensor \(g\)):

The mnemonic is simple. The flat map \(\flat\) lowers the index (superscript \(\to\) subscript) using \(g_{ij}\), while \(\sharp\) raises the index (subscript \(\to\) superscript) using \(g^{ij}\). In Euclidean space where \(g_{ij} = \delta_{ij}\), raising and lowering are invisible. The components do not change. On a curved manifold, they are nontrivial operations that depend on the geometry.

Concrete Example: How the Metric Changes the Dictionary

The Gradient is a Covector

We have now assembled all the machinery to state what is perhaps the single most important conceptual correction for anyone transitioning from standard machine learning to Geometric Deep Learning:

The derivative \(df\) of a function \(f\) is not a vector. It is a covector, an element of the dual space (cotangent space). What we call "the gradient" \(\nabla f\) is the vector obtained by applying the sharp map \(\sharp\) to \(df\).

Let us make this precise. Consider a differentiable function \(f: \mathcal{H} \to \mathbb{R}\) on a Hilbert space. At each point \(x\), the Fréchet derivative (or differential) \(df_x\) is a continuous linear functional, and where it exists it is computed direction by direction as \[ df_x : \mathcal{H} \to \mathbb{R}, \quad df_x(h) = \lim_{t \to 0} \frac{f(x + th) - f(x)}{t}. \] This \(df_x\) lives in \(\mathcal{H}^*\), not in \(\mathcal{H}\). It is a covector.

The gradient \(\nabla f(x)\) is defined as the Riesz representative of this covector:

The Unifying Perspective: Every "Gradient" is \((df)^\sharp\)

Three seemingly different optimization algorithms are standard gradient descent, Natural Gradient, and Newton's method. The deepest insight of this page is that all three are the same formula, applied with different choices of inner product (metric):

All three rows are the same abstract operation \((df)^\sharp\): the sharp map applied to the covector \(df\). The component formula \((\nabla f)^i = g^{ij}(df)_j\) above is its expression in a chosen chart. Equivalently, in matrix form with \([df]\) the row vector of partials and \(G = (g_{ij})\) the metric matrix, \([\nabla f] = G^{-1}\, [df]^\top\). Newton's method uses the Hessian as a metric, producing \(\nabla_{\mathbf{H}} f\). The standard Newton update is then \(\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta\, \nabla_{\mathbf{H}} f\).
§ The Hessian \(\mathbf{H}\) is a true (Riemannian) metric only where it is positive definite, typically near strict local minima of a strongly convex \(f\). At saddle points or in non-convex regions, \(\mathbf{H}\) may be indefinite or singular, and the sharp map \(\sharp_{\mathbf{H}}\) is then not even defined. The discussion below returns to this point.

Bra-Ket Notation, Double Duals & Reflexivity

The Riesz Representation Theorem tells us that Hilbert spaces enjoy a perfect symmetry between vectors and covectors. Before we close this page, let us explore two important extensions of this idea. The first is Dirac's Bra-Ket notation, the physicist's language for the Riesz isomorphism. The second is the double dual together with reflexivity, which tells us when a general Banach space shares this self-dual property.

Dirac's Bra-Ket Notation

In quantum mechanics, Paul Dirac introduced a notation that is, in hindsight, a compact encoding of the Riesz isomorphism. The notation splits the inner product \(\langle \cdot \,,\, \cdot \rangle\) into two halves:

In this notation, the \(\flat\) map is simply the operation of "turning a Ket into a Bra": \(|\phi\rangle \mapsto \langle\phi|\). The \(\sharp\) map reverses it: \(\langle\phi| \mapsto |\phi\rangle\).

The Hahn-Banach Theorem

Before discussing the double dual, we need a fundamental result that guarantees \(\mathcal{X}^*\) always contains "enough" functionals to distinguish points of \(\mathcal{X}\) and to detect their norms. This is the Hahn-Banach Theorem, one of the three pillars of functional analysis, alongside the uniform boundedness principle and the open mapping theorem.

A full proof is given on the Hahn-Banach theorem page, by way of the general sublinear-dominated form and a transfinite extension argument. The norm-preserving statement above is its specialization to the seminorm \(p(x) = \|\varphi_0\|_{\mathcal{Y}^*}\|x\|\). We use it here to derive the concrete application we need below.

The Double Dual and Reflexive Spaces

Since \(\mathcal{X}^*\) is a Banach space, it has its own dual: the double dual \(\mathcal{X}^{**} = (\mathcal{X}^*)^*\). Elements of \(\mathcal{X}^{**}\) are linear functionals that act on covectors.

There is a natural way to embed our original space \(\mathcal{X}\) into \(\mathcal{X}^{**}\). For any vector \(x \in \mathcal{X}\), we define an evaluation functional \(J(x) \in \mathcal{X}^{**}\) that simply "evaluates" each covector at \(x\):

By the Riesz Representation Theorem, every Hilbert space is self-dual (\(\mathcal{H} \cong \mathcal{H}^*\)). Reflexivity asks for more than an abstract isomorphism, namely surjectivity of the canonical map \(J\) itself, and Hilbert spaces meet that stronger requirement.

The spaces \(L^p\) and \(\ell^p\) for \(1 \lt p \lt \infty\) are also reflexive (their double duals cycle back through Hölder conjugation: \(p \to q \to p\)). By contrast \(L^1\) and \(L^\infty\) are not reflexive whenever they are infinite-dimensional, and neither is \(c_0\) (the space of sequences converging to zero), which is infinite-dimensional by construction. The qualifier is not idle: over a finite set carrying the counting measure, \(L^1\) and \(L^\infty\) are both \(\mathbb{F}^n\), where \(J\) is an isometric injection between spaces of the same finite dimension and therefore surjective. In the infinite-dimensional regime that failure creates genuine obstacles in optimization.

Looking Ahead: From Hilbert Spaces to Manifolds

Let us summarize the conceptual journey of this page with a roadmap:

1. The Dual Space \(\mathcal{X}^*\).
   It collects all continuous linear "measurement tools" on \(\mathcal{X}\), and it is always a Banach space.
2. The Riesz Representation Theorem.
   In a Hilbert space, every measurement \(\varphi \in \mathcal{H}^*\) is secretly an inner product with a unique vector \(y_\varphi\). The space is self-dual.
3. Musical Isomorphisms \(\sharp\) and \(\flat\).
   The Riesz map and its inverse provide a "dictionary" between vectors and covectors, powered by the inner product.
4. The Gradient is a Covector.
   The derivative \(df\) naturally lives in the dual. The gradient \(\nabla f = (df)^\sharp\) depends on the choice of inner product (metric).

This final point is the bridge to the geometry track of our curriculum. On smooth manifolds, the inner product varies from point to point (it becomes the Riemannian metric tensor \(g\)), the vectors live in the tangent space \(T_pM\), and the covectors live in the cotangent space \(T_p^*M\). The Musical Isomorphisms, now point-dependent, become the central tool for translating between forces and velocities, between differentials and gradients.

Immediately ahead, however, we remain in the functional analysis setting and ask: what happens when we weaken the notion of convergence? The Weak Topologies and the Banach-Alaoglu Theorem will give us the compactness results we need to guarantee that optimization problems on infinite-dimensional spaces actually have solutions.