Continuous Fourier Transform
Fourier series decompose periodic functions into discrete
frequency components. However, most signals in practice are not periodic. Examples include a spoken sentence, a
transient pulse, and a neural network's activation. The Fourier transform extends the Fourier
series from periodic functions to suitably integrable functions on \(\mathbb{R}\), replacing the discrete spectrum
with a continuous one. This generalization is the mathematical backbone of signal processing, spectral analysis,
and frequency-domain methods throughout machine learning.
To see how this arises, consider a function \(f_L\) periodic with period \(2L\). Its complex Fourier series is:
\[
f_L(x) = \sum_{n=-\infty}^{\infty} c_n e^{-i\frac{n\pi x}{L}}, \quad c_n = \frac{1}{2L}\int_{-L}^{L} f_L(x)e^{i\frac{n\pi x}{L}} \, dx.
\]
As \(L \to \infty\), the discrete frequencies \(\omega_n = \frac{n\pi}{L}\) become densely packed with spacing
\(\Delta\omega = \frac{\pi}{L} \to 0\), and the sum approaches an integral. This limiting process yields the
Fourier transform.
Note on Conventions:
As discussed in the Fourier series conventions, we use the convention with the
positive sign in the forward transform and the negative sign in the inverse. Engineering texts and most analysis
texts use the opposite sign, for example \(\hat{f}(\omega) = \int_{-\infty}^{\infty} f(t)e^{-i\omega t} \, dt\).
Both are mathematically equivalent. Just be consistent within your work.
The Fourier transform is well-defined for different function classes:
- For \(f \in L^1(\mathbb{R})\) (absolutely integrable):
The integral defining \(\hat{f}\) converges absolutely, and \(\hat{f}\) is bounded and continuous
with \(|\hat{f}(\xi)| \leq \|f\|_{L^1}\).
- For \(f \in L^2(\mathbb{R})\) (square-integrable):
On the intersection \(L^1 \cap L^2\), the integral definition above already applies, and Plancherel's theorem
(proved in the Properties section) holds. For \(f \in L^2 \setminus L^1\), the integral does not converge absolutely,
and the Fourier transform must be redefined as a bounded linear operator on the Hilbert space
\(L^2(\mathbb{R})\). See the
Fourier analysis in Hilbert spaces
page.
- For tempered distributions:
The theory extends to generalized functions, allowing transforms of \(\delta\)-functions, polynomials,
and other non-integrable functions important in applications.
Example: Gaussian Function
Consider the Gaussian function:
\[
f(x) = e^{-\frac{x^2}{2\sigma^2}}.
\]
Its Fourier transform (using the convention of this site) is:
\[
\begin{align*}
\hat{f}(\xi) &= \int_{-\infty}^{\infty} e^{-\frac{x^2}{2\sigma^2}}e^{i\xi x} \, dx \\\\
&= \int_{-\infty}^{\infty} e^{-\frac{x^2}{2\sigma^2} + i\xi x} \, dx.
\end{align*}
\]
Complete the square in the exponent:
\[
\begin{align*}
-\frac{x^2}{2\sigma^2} + i\xi x
&= -\frac{1}{2\sigma^2}(x^2 - 2i\sigma^2\xi x) \\\\
&= -\frac{1}{2\sigma^2}((x - i\sigma^2\xi)^2 + \sigma^4\xi^2).
\end{align*}
\]
Therefore:
\[
\hat{f}(\xi) = e^{-\frac{\sigma^2\xi^2}{2}} \int_{-\infty}^{\infty} e^{-\frac{(x-i\sigma^2\xi)^2}{2\sigma^2}} \, dx.
\]
The integrand is entire and decays exponentially as \(|\mathrm{Re}\, z| \to \infty\), uniformly on horizontal
strips of bounded imaginary part. Cauchy's theorem applied to a rectangular contour therefore justifies shifting
the contour from \(\mathbb{R}\) to \(\mathbb{R} + i\sigma^2\xi\) without changing the value, since the
vertical-side contributions vanish in the limit:
\[
\begin{align*}
\int_{-\infty}^{\infty} e^{-\frac{(x-i\sigma^2\xi)^2}{2\sigma^2}} \, dx
&= \int_{-\infty}^{\infty} e^{-\frac{u^2}{2\sigma^2}} \, du \\\\
&= \sigma\sqrt{2\pi}.
\end{align*}
\]
Thus:
\[
\hat{f}(\xi) = \sigma\sqrt{2\pi} \cdot e^{-\frac{\sigma^2\xi^2}{2}}.
\]
The transform of a Gaussian is again a Gaussian, with the width inverted. A function narrow in
space has a wide transform, and conversely. The whole family is therefore preserved by the
Fourier transform, and the single choice \(\sigma = 1\) gives a genuine eigenfunction, since
\(e^{-x^2/2}\) transforms to \(\sqrt{2\pi}\, e^{-\xi^2/2}\). Normalizing by \(\sqrt{2\pi}\)
therefore fixes \(e^{-x^2/2}\) outright, and the same unitary rescaling reappears in the discrete
setting, where dividing the Fourier matrix by the square root of its size puts its eigenvalues
on the unit circle. This behavior makes Gaussians fundamental in uncertainty principles, quantum mechanics,
and signal processing.
Definition: The Schwartz Class
The Schwartz class \(\mathcal{S}(\mathbb{R})\) is the set of infinitely
differentiable functions \(f : \mathbb{R} \to \mathbb{C}\) satisfying
\[
\sup_{x \in \mathbb{R}} \bigl| x^m f^{(n)}(x) \bigr| \lt \infty
\quad \text{for all integers } m, n \geq 0.
\]
Its members are the rapidly decreasing functions, so named because every
derivative decays faster than any power of \(1/|x|\).
Two consequences of the definition are used repeatedly below. Let \(f \in \mathcal{S}(\mathbb{R})\).
The case \(m = 0\) says that \(f\) and every one of its derivatives are bounded. Taking
\((m, n) = (0, 0)\) and \((m, n) = (2, 0)\) together bounds \((1 + x^2)|f(x)|\), so \(|f(x)|\) is
dominated by a constant multiple of \((1 + x^2)^{-1}\), and therefore
\(\mathcal{S}(\mathbb{R}) \subset L^1(\mathbb{R}) \cap L^2(\mathbb{R})\).
The Gaussian of the example above is the canonical member of \(\mathcal{S}(\mathbb{R})\), and every
smooth function with compact support belongs to it as well. Membership is preserved by
differentiation, by multiplication by polynomials, and by the reflection \(f(x) \mapsto f(-x)\).
The first and third of these are immediate from the defining bounds, and the second follows from
them by the Leibniz rule.
Properties of Fourier Transform
The Fourier transform is far more than a definition. Its power lies in a collection of algebraic properties that
convert difficult operations (differentiation, convolution) into simple ones (multiplication, pointwise products).
These properties, stated below using the convention of this site \(\hat{f}(\xi) = \int_{-\infty}^{\infty} f(x)e^{ix\xi} \, dx\),
form the foundation of spectral methods in both analysis and computation.
Unless indicated otherwise, the algebraic properties below are stated for \(f\) and \(g\) in the
Schwartz class, where every
integral converges absolutely and differentiation under the integral sign is legitimate. Each
property extends to larger classes under its own hypotheses. The theorems of this section, by
contrast, carry the hypotheses given with them.
Property: Linearity
\[
\mathcal{F}\{\alpha f + \beta g\} = \alpha \mathcal{F}\{f\} + \beta \mathcal{F}\{g\}
\]
for constants \(\alpha, \beta \in \mathbb{C}\).
Property: Translation (Shift)
\[
\mathcal{F}\{f(x - a)\}(\xi) = e^{ia\xi}\hat{f}(\xi)
\]
A shift in space corresponds to a phase shift in frequency.
Property: Scaling
\[
\mathcal{F}\{f(ax)\}(\xi) = \frac{1}{|a|}\hat{f}\left(\frac{\xi}{a}\right), \quad a \neq 0
\]
This embodies the uncertainty principle. Compressing a signal in time stretches its frequency
spectrum, and vice versa. One cannot localize a signal arbitrarily well in both time and frequency
simultaneously.
Property: Differentiation
For \(f\) in the Schwartz class,
\[
\mathcal{F}\{f'(x)\}(\xi) = -i\xi \hat{f}(\xi)
\]
More generally, for the \(n\)-th derivative:
\[
\mathcal{F}\{f^{(n)}(x)\}(\xi) = (-i\xi)^n \hat{f}(\xi)
\]
This transforms differentiation into algebraic multiplication, which is why Fourier methods are powerful for
solving differential equations. Note the sign difference from the engineering convention.
Proof (the four properties):
Linearity is the linearity of the integral. The translation property follows from the
substitution \(u = x - a\):
\[
\int_{-\infty}^{\infty} f(x - a)\, e^{ix\xi}\, dx
= \int_{-\infty}^{\infty} f(u)\, e^{i(u + a)\xi}\, du
= e^{ia\xi} \hat{f}(\xi).
\]
Scaling follows from the substitution \(u = ax\). When \(a \lt 0\) this reverses the orientation
of the line, and the reversal combines with the Jacobian into the single factor \(1/|a|\):
\[
\int_{-\infty}^{\infty} f(ax)\, e^{ix\xi}\, dx
= \frac{1}{|a|} \int_{-\infty}^{\infty} f(u)\, e^{iu\xi/a}\, du
= \frac{1}{|a|} \hat{f}\left(\frac{\xi}{a}\right).
\]
Differentiation follows from integration by parts. The boundary term vanishes because
\(f(x) \to 0\) as \(|x| \to \infty\), which leaves
\[
\int_{-\infty}^{\infty} f'(x)\, e^{ix\xi}\, dx
= -\int_{-\infty}^{\infty} f(x)\, i\xi\, e^{ix\xi}\, dx
= -i\xi\, \hat{f}(\xi).
\]
Every derivative of a Schwartz function is again a Schwartz function, so the same step applies
with \(f'\) in place of \(f\), and induction on \(n\) gives
\(\mathcal{F}\{f^{(n)}\}(\xi) = (-i\xi)^n \hat{f}(\xi)\).
Theorem: Invariance of the Schwartz Class
If \(f \in \mathcal{S}(\mathbb{R})\), then \(\hat{f} \in \mathcal{S}(\mathbb{R})\).
Proof:
Fix integers \(m, n \geq 0\). Because \(x \mapsto x^k f(x)\) lies in \(\mathcal{S}(\mathbb{R})\)
and hence in \(L^1(\mathbb{R})\) for every integer \(k \geq 0\), the integral defining
\(\hat{f}\) may be differentiated under the integral sign any number of times. This makes
\(\hat{f}\) smooth and gives
\[
\frac{d^n}{d\xi^n} \hat{f}(\xi) = \widehat{(ix)^n f}(\xi).
\]
Write \(h(x) = (ix)^n f(x)\), so that \(h \in \mathcal{S}(\mathbb{R})\) and
\(\frac{d^n}{d\xi^n} \hat{f} = \hat{h}\). Applying the
differentiation property
to \(h\) gives \(\widehat{h^{(m)}}(\xi) = (-i\xi)^m \hat{h}(\xi)\), and therefore
\[
\left| \xi^m \frac{d^n}{d\xi^n} \hat{f}(\xi) \right|
= \bigl| \xi^m \hat{h}(\xi) \bigr|
= \bigl| \widehat{h^{(m)}}(\xi) \bigr|
\leq \| h^{(m)} \|_{L^1}.
\]
The right-hand side is finite and independent of \(\xi\), because \(h^{(m)}\) again lies in
\(\mathcal{S}(\mathbb{R}) \subset L^1(\mathbb{R})\). Since \(m\) and \(n\) were arbitrary,
\(\hat{f}\) satisfies every bound in the definition of the Schwartz class.
Theorem: Fourier Inversion
For suitable functions (for example, \(f \in L^1(\mathbb{R})\) with
\(\hat{f} \in L^1(\mathbb{R})\)), the original function can be recovered for almost every \(x\), and for every
\(x\) when \(f\) is continuous, in particular on \(\mathcal{S}(\mathbb{R})\):
\[
f(x) = \frac{1}{2\pi}\int_{-\infty}^{\infty} \hat{f}(\xi)e^{-ix\xi} \, d\xi.
\]
A corollary, in the same sense, is \(\mathcal{F}\{\mathcal{F}\{f\}\}(x) = 2\pi f(-x)\), a deep symmetry between the spatial and
frequency domains.
Proof (Schwartz class):
The theorem statement gives the inversion formula for the class
\(\{f \in L^1(\mathbb{R}) : \hat f \in L^1(\mathbb{R})\}\). The proof we present here is restricted to the
smaller Schwartz class \(\mathcal{S}(\mathbb{R}) \subset \{f \in L^1 : \hat f \in L^1\}\). The extension from
\(\mathcal{S}(\mathbb{R})\) to the full \(L^1\) class indicated in the statement is discussed after the proof.
The Schwartz-class technique is Gaussian regularization. We insert a Gaussian damping factor
that makes Fubini applicable, then remove it in the limit.
Step 1: regularized inverse.
For \(\epsilon \gt 0\), define
\[
I_\epsilon(x) = \frac{1}{2\pi} \int_{-\infty}^{\infty} \hat{f}(\xi)\, e^{-\epsilon \xi^2/2}\, e^{-ix\xi}\, d\xi.
\]
Since \(\hat f \in \mathcal{S}(\mathbb{R})\) by the
invariance of the Schwartz class,
the integrand is absolutely integrable for every \(\epsilon \geq 0\). In particular the
integral converges and \(I_\epsilon(x)\) is well-defined.
Step 2: Fubini interchange.
We substitute \(\hat f(\xi) = \int f(y) e^{iy\xi}\, dy\). The resulting double integrand satisfies
\[
\bigl|f(y)\, e^{iy\xi}\, e^{-\epsilon\xi^2/2}\, e^{-ix\xi}\bigr| = |f(y)|\, e^{-\epsilon\xi^2/2} \in L^1(\mathbb{R} \times \mathbb{R}),
\]
the product of an \(L^1\) function in \(y\) with an \(L^1\) function in \(\xi\). Therefore
Fubini's theorem,
applied to the real and imaginary parts of the integrand separately, permits the order swap:
\[
\begin{align*}
I_\epsilon(x) &= \int_{-\infty}^{\infty} f(y)\, K_\epsilon(y - x)\, dy, \\\\
K_\epsilon(z) &= \frac{1}{2\pi} \int_{-\infty}^{\infty} e^{iz\xi}\, e^{-\epsilon\xi^2/2}\, d\xi.
\end{align*}
\]
The kernel \(K_\epsilon\) is the forward Fourier transform of the Gaussian \(\xi \mapsto e^{-\epsilon\xi^2/2}\),
evaluated at \(z\), divided by \(2\pi\). We reuse the Gaussian-transform calculation given earlier, with the roles
of the space and frequency variables exchanged. Apply the formula
\[
\widehat{e^{-x^2/(2\sigma^2)}}(\xi) = \sigma\sqrt{2\pi}\, e^{-\sigma^2 \xi^2/2}
\]
to the variable \(\xi\) in the role of \(x\), with \(\sigma^2 = 1/\epsilon\). This gives
\[
\int e^{iz\xi}\, e^{-\epsilon\xi^2/2}\, d\xi = \sqrt{2\pi/\epsilon}\, e^{-z^2/(2\epsilon)}.
\]
Thus
\[
K_\epsilon(z) = \frac{1}{\sqrt{2\pi\epsilon}}\, e^{-z^2/(2\epsilon)},
\]
a Gaussian of unit mass: \(\int K_\epsilon(z)\, dz = 1\).
Step 3: convolution limit.
By the substitution \(z = y - x\), we have \(I_\epsilon(x) = \int f(x + z)\, K_\epsilon(z)\, dz\). Using
\(\int K_\epsilon = 1\) to write \(f(x) = \int f(x)\, K_\epsilon(z)\, dz\), we obtain
\[
I_\epsilon(x) - f(x) = \int_{-\infty}^{\infty} \bigl[ f(x + z) - f(x) \bigr]\, K_\epsilon(z)\, dz.
\]
Fix \(\eta \gt 0\). Every derivative of a Schwartz function is bounded, so
\(|f(x + z) - f(x)| \leq \|f'\|_\infty\, |z|\) for all \(x\) and \(z\). Taking
\(\delta = \eta / (1 + \|f'\|_\infty)\) therefore gives \(|f(x + z) - f(x)| \lt \eta\)
whenever \(|z| \leq \delta\), uniformly in \(x\). Split the integral accordingly.
On the inner region \(|z| \leq \delta\), positivity of \(K_\epsilon\) and unit mass give
\[
\left| \int_{|z| \leq \delta} [f(x+z) - f(x)]\, K_\epsilon(z)\, dz \right| \leq \eta.
\]
On the outer region \(|z| \gt \delta\), the substitution \(u = z/\sqrt{\epsilon}\) shows
\[
\int_{|z| \gt \delta} K_\epsilon(z)\, dz
= \int_{|u| \gt \delta/\sqrt{\epsilon}} \frac{1}{\sqrt{2\pi}}\, e^{-u^2/2}\, du \longrightarrow 0 \quad \text{as } \epsilon \to 0,
\]
since the lower bound \(\delta/\sqrt{\epsilon} \to \infty\) and the integrand is the standard-normal density.
Combined with the trivial bound \(|f(x+z) - f(x)| \leq 2 \sup_{\mathbb{R}} |f|\) on the outer region,
\[
\limsup_{\epsilon \to 0}\, |I_\epsilon(x) - f(x)| \leq \eta.
\]
Since \(\eta \gt 0\) was arbitrary, \(\lim_{\epsilon \to 0} I_\epsilon(x) = f(x)\).
Step 4: spectral-side limit.
We return to the definition of \(I_\epsilon\). The integrand
\[
\hat f(\xi)\, e^{-\epsilon \xi^2/2}\, e^{-ix\xi}
\]
is dominated by \(|\hat f(\xi)| \in L^1(\mathbb{R})\) uniformly
in \(\epsilon\) and converges pointwise to \(\hat f(\xi)\, e^{-ix\xi}\) as \(\epsilon \to 0\). The
dominated convergence theorem
applies along any sequence \(\epsilon_j \to 0\), and the resulting value does not depend on the
sequence chosen, so
\[
\lim_{\epsilon \to 0} I_\epsilon(x) = \frac{1}{2\pi}\int_{-\infty}^{\infty} \hat f(\xi)\, e^{-ix\xi}\, d\xi.
\]
Equating the two limits established in Steps 3 and 4 yields the inversion formula for every \(f \in \mathcal{S}(\mathbb{R})\).
On the \(L^1\) extension.
The Schwartz proof above does not directly extend to the larger class
\[
\{f \in L^1(\mathbb{R}) : \hat f \in L^1(\mathbb{R})\}
\]
by a one-line density argument. Schwartz functions are dense in \(L^1\), by the
density lemma below, but the resulting
approximation \(\hat{f_n} \to \hat f\) is only uniform, not \(L^1\). The right-hand side of the inversion formula
therefore does not pass to the limit for free. The \(L^1\) case can nevertheless be reached by the same Gaussian
regularization. The pointwise limit of Step 3 is replaced by convergence of the Gaussian smoothing in \(L^1\), and
an almost everywhere convergent subsequence then gives the formula for almost every \(x\). We do not carry this
out here. If in addition \(f\) is continuous, equality holds for every \(x\): the right-hand side is continuous by
dominated convergence, since \(\hat f \in L^1(\mathbb{R})\), and two continuous functions that agree almost
everywhere agree everywhere.
Inversion shows that the Fourier transform is invertible on the Schwartz class. The formula
\(f = \mathcal{F}^{-1}\mathcal{F} f\) holds for every \(f \in \mathcal{S}(\mathbb{R})\), and the corollary
\(\mathcal{F}\mathcal{F} f(x) = 2\pi f(-x)\) ensures invertibility from the other side as well.
Stage 2 of the proof of Plancherel's theorem below needs one approximation fact. An arbitrary
\(f \in L^1 \cap L^2(\mathbb{R})\) must be approached by Schwartz functions in both norms at once.
We prove it now. The smoothing device is the Gaussian kernel of the inversion proof, applied this
time to \(f\) itself rather than to its transform.
In the lemma below, integrals over \(\mathbb{R}\) are taken against the
Lebesgue measure on
the Borel sets, and all functions are Borel measurable. We write \(|E|\) for the measure of a Borel
set \(E\). Two properties of this measure are used without proof. It is
translation invariant, \(|E + y| = |E|\) for every Borel set \(E\) and every
\(y \in \mathbb{R}\), and hence \(\int h(x + y)\, dx = \int h(x)\, dx\) for every non-negative or
integrable Borel function \(h\), the identity passing from indicators to simple functions to
monotone limits. It is outer regular: for every Borel set \(E\) of finite measure
and every \(\eta \gt 0\), there is an open set \(U \supseteq E\) with \(|U \setminus E| \lt \eta\).
The first reflects the fact that \(B \mapsto |B + y|\) is again a
measure giving each interval \((a, b]\) its length, and the second that the construction measures a
set by covering it with countably many intervals.
Lemma: Density of the Schwartz Class
Let \(p \in \{1, 2\}\). The Schwartz class \(\mathcal{S}(\mathbb{R})\) is dense in
\(L^p(\mathbb{R})\): for every \(f \in L^p(\mathbb{R})\) there are
\(f_n \in \mathcal{S}(\mathbb{R})\) with \(\|f_n - f\|_p \to 0\). When
\(f \in L^1 \cap L^2(\mathbb{R})\), a single sequence \(\{f_n\}\) converges to \(f\) in
\(L^1\) and in \(L^2\) simultaneously.
Proof:
For a function \(h\) on \(\mathbb{R}\) and \(y \in \mathbb{R}\), write
\((\tau_y h)(x) = h(x + y)\) for its translate. Translation invariance gives
\(\|\tau_y h\|_p = \|h\|_p\). A step function is a finite linear combination
of indicators of bounded intervals.
Step 1: step functions are dense in \(L^p\).
Let \(g \in L^p(\mathbb{R})\). Writing
\(g = (\Re g)^+ - (\Re g)^- + i\,(\Im g)^+ - i\,(\Im g)^-\), each of the four parts is
non-negative, Borel, and dominated by \(|g|\), so it suffices to treat \(0 \leq g \in L^p\).
First, \(g\, \mathbf{1}_{[-M, M]} \to g\) in \(L^p\) as \(M \to \infty\) through the integers,
by the
dominated convergence theorem
with dominator \(|g|^p\), so we may assume that \(g\) vanishes outside \([-M, M]\). Next, the
staircase functions
\[
s_n = \min\bigl( 2^{-n} \lfloor 2^n g \rfloor,\ n \bigr)
\]
are simple, satisfy \(0 \leq s_n \leq g\), and obey \(|s_n - g| \leq 2^{-n}\) wherever
\(g \lt n\). Since \(g\) is finite almost everywhere, \(s_n \to g\) almost everywhere, and the
dominated convergence theorem with dominator \(g^p\) gives \(\|s_n - g\|_p \to 0\). Each
\(s_n\) is a finite combination \(\sum_j \alpha_j \mathbf{1}_{E_j}\) of indicators of Borel sets
\(E_j \subseteq [-M, M]\).
It remains to approximate \(\mathbf{1}_E\) for a Borel set \(E \subseteq [-M, M]\). Fix
\(\eta \gt 0\). Outer regularity gives an open \(U \supseteq E\) with \(|U \setminus E| \lt \eta\),
and \(W = U \cap (-M - 1, M + 1)\) is open, contains \(E\), and has
\(|W \setminus E| \lt \eta\) and \(|W| \leq 2M + 2\). Every open subset of \(\mathbb{R}\) is the
disjoint union of its connected components, which are open intervals, and there are at most
countably many of them because each contains a rational number. Write
\(W = \bigsqcup_k I_k\) accordingly. By countable additivity \(\sum_k |I_k| = |W| \lt \infty\),
so there is \(K\) with \(\sum_{k \gt K} |I_k| \lt \eta\). The set
\(V = I_1 \sqcup \cdots \sqcup I_K\) satisfies
\(V \triangle E \subseteq (W \setminus E) \cup (W \setminus V)\), hence
\(|V \triangle E| \lt 2\eta\), and \(\mathbf{1}_V = \sum_{k \leq K} \mathbf{1}_{I_k}\) is a step
function with
\[
\|\mathbf{1}_V - \mathbf{1}_E\|_p = |V \triangle E|^{1/p} \lt (2\eta)^{1/p}.
\]
Replacing each \(\mathbf{1}_{E_j}\) in \(s_n\) this way and chaining the approximations
proves Step 1.
Step 2: translation is continuous in \(L^p\).
We claim \(\|\tau_y g - g\|_p \to 0\) as \(y \to 0\) for every \(g \in L^p(\mathbb{R})\). For
the indicator of a bounded interval \(I\) with endpoints \(a \lt b\), the function
\(\tau_y \mathbf{1}_I - \mathbf{1}_I\) takes the values \(0\) and \(\pm 1\), the latter exactly on
\((I - y) \triangle I\), a set of measure \(2 \min(|y|, b - a) \leq 2|y|\). Hence
\(\|\tau_y \mathbf{1}_I - \mathbf{1}_I\|_p \leq (2|y|)^{1/p} \to 0\), and by the triangle
inequality the claim holds for every step function. For general \(g\), fix \(\eta \gt 0\) and a
step function \(s\) with \(\|g - s\|_p \lt \eta\) by Step 1. Then
\[
\begin{align*}
\|\tau_y g - g\|_p
&\leq \|\tau_y (g - s)\|_p + \|\tau_y s - s\|_p + \|s - g\|_p \\\\
&\lt 2\eta + \|\tau_y s - s\|_p,
\end{align*}
\]
so \(\limsup_{y \to 0} \|\tau_y g - g\|_p \leq 2\eta\) for every \(\eta \gt 0\).
Step 3: Gaussian smoothing converges in \(L^p\).
For \(\epsilon \gt 0\), let
\[
K_\epsilon(z) = \frac{1}{\sqrt{2\pi\epsilon}}\, e^{-z^2/(2\epsilon)}
\]
be the kernel of the inversion proof, positive, even, and of unit mass. Let \(g \in L^p\)
vanish outside \([-M, M]\), and assume in addition \(g \in L^1\). Define
\[
\begin{align*}
g_\epsilon(x)
&= \int_{-\infty}^{\infty} g(y)\, K_\epsilon(x - y)\, dy \\\\
&= \int_{-\infty}^{\infty} g(x + z)\, K_\epsilon(z)\, dz,
\end{align*}
\]
the integral converging absolutely for every \(x\) because \(g \in L^1\) and
\(K_\epsilon \leq (2\pi\epsilon)^{-1/2}\). The second form comes from the translation
\(y = x + z\) and the evenness of \(K_\epsilon\). Using unit mass to write
\(g(x) = \int g(x)\, K_\epsilon(z)\, dz\),
\[
g_\epsilon(x) - g(x) = \int_{-\infty}^{\infty} \bigl[ g(x + z) - g(x) \bigr]\, K_\epsilon(z)\, dz.
\]
For \(p = 1\) the triangle inequality for integrals, and for \(p = 2\) the Cauchy-Schwarz
inequality applied to the factors \([g(x + z) - g(x)] \sqrt{K_\epsilon(z)}\) and
\(\sqrt{K_\epsilon(z)}\), give in both cases
\[
|g_\epsilon(x) - g(x)|^p \leq \int_{-\infty}^{\infty} |g(x + z) - g(x)|^p\, K_\epsilon(z)\, dz.
\]
The function \((x, z) \mapsto |g(x + z) - g(x)|^p K_\epsilon(z)\) is non-negative and Borel
on \(\mathbb{R}^2\), being built from \(g\) and continuous maps. The Borel sets of
\(\mathbb{R}^2\) lie in the product \(\sigma\)-algebra of the Borel sets of \(\mathbb{R}\),
since every open subset of \(\mathbb{R}^2\) is a countable union of open rectangles with
rational corners. Integrating in \(x\) and applying, Lebesgue measure being
\(\sigma\)-finite,
Tonelli's theorem,
\[
\begin{align*}
\|g_\epsilon - g\|_p^p
&\leq \int_{-\infty}^{\infty} \omega(z)\, K_\epsilon(z)\, dz, \\\\
\omega(z) &= \|\tau_z g - g\|_p^p.
\end{align*}
\]
Here \(\omega(z) \leq (\|\tau_z g\|_p + \|g\|_p)^p = 2^p \|g\|_p^p\) for every \(z\), and
\(\omega(z) \to 0\) as \(z \to 0\) by Step 2. Fix \(\eta \gt 0\) and choose \(\delta \gt 0\)
with \(\omega(z) \lt \eta\) for \(|z| \leq \delta\). Splitting the integral as in the inversion
proof,
\[
\|g_\epsilon - g\|_p^p
\leq \eta + 2^p \|g\|_p^p \int_{|z| \gt \delta} K_\epsilon(z)\, dz,
\]
and the tail integral tends to \(0\) as \(\epsilon \to 0\), as computed in Step 3 of that proof.
Hence \(\limsup_{\epsilon \to 0} \|g_\epsilon - g\|_p^p \leq \eta\) for every \(\eta \gt 0\),
that is, \(g_\epsilon \to g\) in \(L^p\).
Step 4: the smoothed function is a Schwartz function.
Keep \(g \in L^1\) vanishing outside \([-M, M]\). The kernel \(K_\epsilon\) is a constant
multiple of the Gaussian \(e^{-z^2/(2\sigma^2)}\) of the Gaussian example above with
\(\sigma^2 = \epsilon\), hence a member of \(\mathcal{S}(\mathbb{R})\). Every derivative
\(K_\epsilon^{(n)}\) is therefore bounded, and every quantity
\(C_{m,n} = \sup_{u} |u|^m |K_\epsilon^{(n)}(u)|\) is finite. We claim
\[
g_\epsilon^{(n)}(x) = \int_{-M}^{M} g(y)\, K_\epsilon^{(n)}(x - y)\, dy
\quad \text{for all } n \geq 0.
\]
Assuming it for \(n\), the difference quotients of the integrand in \(x\) are bounded, by the
mean value theorem, by \(|g(y)|\, \|K_\epsilon^{(n+1)}\|_\infty\), an integrable function of
\(y\) independent of \(x\). The dominated convergence theorem, applied along any sequence of
increments tending to zero, then differentiates under the integral sign and gives the case
\(n + 1\). So \(g_\epsilon\) is smooth.
For the decay, let \(|x| \geq 2M\). Every \(y\) in the range of integration has
\(|y| \leq M\), so \(|x - y| \geq |x| - |y| \geq |x|/2\), and
\[
\begin{align*}
|x^m g_\epsilon^{(n)}(x)|
&\leq \int_{-M}^{M} |g(y)|\, 2^m |x - y|^m\, |K_\epsilon^{(n)}(x - y)|\, dy \\\\
&\leq 2^m C_{m,n}\, \|g\|_1.
\end{align*}
\]
For \(|x| \lt 2M\) the same integral is at most \((2M)^m \|K_\epsilon^{(n)}\|_\infty \|g\|_1\).
Thus \(\sup_x |x^m g_\epsilon^{(n)}(x)| \lt \infty\) for all \(m, n \geq 0\), and
\(g_\epsilon \in \mathcal{S}(\mathbb{R})\).
Step 5: assembly.
Let \(f \in L^p(\mathbb{R})\) and set \(f_M = f\, \mathbf{1}_{[-M, M]}\). As in Step 1,
\(\|f - f_M\|_p \to 0\) as \(M \to \infty\). Each \(f_M\) lies in \(L^1\). For \(p = 2\) this
follows from the Cauchy-Schwarz inequality, which gives \(\int |f_M| \leq (2M)^{1/2} \|f\|_2\).
Steps 3 and 4 therefore
apply to \(g = f_M\), and \((f_M)_\epsilon \in \mathcal{S}(\mathbb{R})\) converges to \(f_M\) in
\(L^p\) as \(\epsilon \to 0\). Choose \(M_n\) with \(\|f - f_{M_n}\|_p \lt 1/n\), then
\(\epsilon_n\) with \(\|(f_{M_n})_{\epsilon_n} - f_{M_n}\|_p \lt 1/n\), and set
\(f_n = (f_{M_n})_{\epsilon_n}\). Then \(\|f_n - f\|_p \lt 2/n\).
The construction \(f \mapsto (f_M)_\epsilon\) does not depend on \(p\). If
\(f \in L^1 \cap L^2(\mathbb{R})\), both requirements on \(M_n\) hold for all large \(M\), and
both requirements on \(\epsilon_n\) hold for all small \(\epsilon\), so \(M_n\) and
\(\epsilon_n\) can be chosen to satisfy the \(L^1\) and \(L^2\) conditions at once. The
resulting sequence converges to \(f\) in both norms simultaneously.
The next theorem extracts from inversion the deeper structural fact: on \(L^1 \cap L^2(\mathbb{R})\), the Fourier
transform also preserves the \(L^2\) inner product, up to the normalization constant
\(\tfrac{1}{2\pi}\). This is Plancherel's identity, the continuous-frequency analogue of the
Parseval identity for Fourier series. Its proof uses
inversion (Stage 1 below), then extends from \(\mathcal{S}(\mathbb{R})\) to \(L^1 \cap L^2(\mathbb{R})\) by a
density argument resting on the lemma just proved (Stage 2).
Theorem: Plancherel's Theorem (on \(L^1 \cap L^2\))
For \(f, g \in L^1 \cap L^2(\mathbb{R})\), the classical Fourier transform defined by the absolutely convergent integral
\(\hat f(\xi) = \int f(x) e^{ix\xi}\, dx\) satisfies
\[
\int_{-\infty}^{\infty} f(x)\overline{g(x)} \, dx = \frac{1}{2\pi}\int_{-\infty}^{\infty} \hat{f}(\xi)\overline{\hat{g}(\xi)} \, d\xi.
\]
In particular, setting \(f = g\):
\[
\|f\|_2^2 = \int_{-\infty}^{\infty} |f(x)|^2 \, dx = \frac{1}{2\pi}\int_{-\infty}^{\infty} |\hat{f}(\xi)|^2 \, d\xi.
\]
Proof:
We prove the identity in two stages: first on the Schwartz class \(\mathcal{S}(\mathbb{R})\),
then by density on all of \(L^1 \cap L^2(\mathbb{R})\). It suffices to prove the polarized form. The special case \(f = g\)
follows by setting them equal.
Stage 1: identity on the Schwartz class.
Fix \(f, g \in \mathcal{S}(\mathbb{R})\). By the
invariance of the Schwartz class,
\(\hat f, \hat g \in \mathcal{S}(\mathbb{R})\) as well. Apply the Fourier inversion theorem (proved above) to \(g\):
\[
g(x) = \frac{1}{2\pi}\int \hat g(\xi)\, e^{-ix\xi}\, d\xi.
\]
Taking the complex conjugate of both sides, and using \(\overline{e^{-ix\xi}} = e^{+ix\xi}\),
\[
\overline{g(x)} = \frac{1}{2\pi}\int \overline{\hat g(\xi)}\, e^{ix\xi}\, d\xi.
\]
Multiplying by \(f(x)\) and integrating in \(x\),
\[
\int f(x)\overline{g(x)}\, dx = \frac{1}{2\pi}\iint f(x)\, e^{ix\xi}\, \overline{\hat g(\xi)}\, dx\, d\xi.
\]
To justify the order swap in the double integral, we check the integrability hypothesis of Fubini's theorem.
The absolute value of the integrand is \(|f(x)| \cdot |\hat g(\xi)|\), and
Tonelli's theorem
applied to this non-negative measurable function gives
\[
\iint |f(x)| \cdot |\hat g(\xi)|\, dx\, d\xi = \|f\|_1 \cdot \|\hat g\|_1 \lt \infty,
\]
since both \(f\) and \(\hat g\) are Schwartz, hence in \(L^1(\mathbb{R})\).
Therefore the integrand is in \(L^1(\mathbb{R} \times \mathbb{R})\), and
Fubini's theorem,
again applied to the real and imaginary parts separately, permits the order swap:
\[
\begin{align*}
\frac{1}{2\pi}\iint f(x)\, e^{ix\xi}\, \overline{\hat g(\xi)}\, dx\, d\xi
&= \frac{1}{2\pi}\int \overline{\hat g(\xi)} \left( \int f(x)\, e^{ix\xi}\, dx \right) d\xi \\\\
&= \frac{1}{2\pi}\int \hat f(\xi)\, \overline{\hat g(\xi)}\, d\xi,
\end{align*}
\]
where the inner integral is \(\hat f(\xi)\) by definition. This establishes the polarized identity on \(\mathcal{S}(\mathbb{R})\).
Stage 2: extension to \(L^1 \cap L^2(\mathbb{R})\).
Equip \(L^1 \cap L^2(\mathbb{R})\) with the \(L^2\) norm. Then \(\mathcal{S}(\mathbb{R})\) is a dense subspace,
by the density lemma proved above. The
lemma gives more, and the argument uses it. For a fixed \(f \in L^1 \cap L^2\), it produces an approximating
sequence \(\{f_n\} \subset \mathcal{S}(\mathbb{R})\) converging to \(f\) simultaneously in
both \(L^1\) and \(L^2\). We fix such a sequence for the remainder of the proof.
Stage 1 specialized to \(f = g\) gives
\[
\|f\|_2^2 = \frac{1}{2\pi}\|\hat f\|_2^2
\]
for every \(f \in \mathcal{S}(\mathbb{R})\). Equivalently, the linear map
\(T : \mathcal{S}(\mathbb{R}) \to L^2(\mathbb{R})\), \(Tf = \hat f / \sqrt{2\pi}\), is an isometry from
\((\mathcal{S}(\mathbb{R}), \|\cdot\|_2)\) into \(L^2(\mathbb{R})\).
From this, two observations about our sequence \(\{f_n\}\) follow:
-
The sequence \(\hat{f_n}\) is Cauchy in \(L^2\).
By Stage 1 applied to the differences
\(f_n - f_m \in \mathcal{S}(\mathbb{R})\), \(\|\hat{f_n} - \hat{f_m}\|_2 = \sqrt{2\pi}\, \|f_n - f_m\|_2 \to 0\). Hence by
completeness of \(L^2\),
\(\hat{f_n}\) converges to some \(F \in L^2(\mathbb{R})\).
-
The limit \(F\) coincides with the classical Fourier transform \(\hat f\).
Since \(f \in L^1\), the integral \(\hat f(\xi) = \int f(x)\, e^{ix\xi}\, dx\) is absolutely convergent and defines
\(\hat f \in L^\infty(\mathbb{R})\).
The \(L^1\)-convergence \(f_n \to f\) (which our sequence provides) gives \(\hat{f_n} \to \hat f\) uniformly
on \(\mathbb{R}\), by the elementary estimate
\[
\begin{align*}
\|\hat{f_n} - \hat f\|_\infty
&= \sup_\xi \left|\int (f_n - f)(x)\, e^{ix\xi}\, dx\right| \\\\
&\leq \|f_n - f\|_1 \to 0.
\end{align*}
\]
Combined with \(\hat{f_n} \to F\) in \(L^2\), passing to a subsequence (using
subsequential a.e. convergence
for \(L^2\) convergence) yields pointwise a.e. convergence to both \(F\) and \(\hat f\), so \(F = \hat f\) a.e.
The \(L^2\) norm is continuous, so we may take the limit \(n \to \infty\) in the Stage 1 identity
\(\|f_n\|_2^2 = \tfrac{1}{2\pi}\|\hat{f_n}\|_2^2\). This yields
\[
\|f\|_2^2 = \tfrac{1}{2\pi}\|\hat f\|_2^2.
\]
The polarized form follows by the same argument applied to a second sequence
\(\{g_n\} \subset \mathcal{S}(\mathbb{R})\) approximating \(g \in L^1 \cap L^2\) simultaneously in \(L^1\) and
\(L^2\). The bilinear identity
\[
\int f_n \overline{g_n}\, dx = \tfrac{1}{2\pi}\int \hat{f_n}\, \overline{\hat{g_n}}\, d\xi
\]
passes to the limit on both sides by continuity of the \(L^2\) inner product. Here \(f_n \to f\) and
\(g_n \to g\) in \(L^2\), and the same argument used above for \(\hat f\) gives \(\hat{f_n} \to \hat f\) and
\(\hat{g_n} \to \hat g\) in \(L^2\).
Looking ahead: both theorems on \(L^2(\mathbb{R})\).
The two main results of this section are Fourier inversion and Plancherel's identity. Both are stated
above on classes that require \(f \in L^1\): inversion on \(\{f \in L^1 : \hat f \in L^1\}\)
(proved here for the smaller Schwartz class), and Plancherel on \(L^1 \cap L^2\).
For functions \(f \in L^2 \setminus L^1\) (such as \((1+|x|)^{-1}\)), the defining integral
\(\hat f(\xi) = \int f(x)\, e^{ix\xi}\, dx\) fails to converge absolutely, so neither result applies directly.
The classes \(L^1\) and \(L^2\) are mutually non-contained on \(\mathbb{R}\)
(for instance \(|x|^{-1/2}\mathbf{1}_{[0,1]} \in L^1 \setminus L^2\)), so neither setting subsumes the other.
The remedy is to redefine the Fourier transform on \(L^2(\mathbb{R})\) as a bounded linear operator,
obtained as the unique continuous extension of its restriction to the dense subspace \(L^1 \cap L^2\).
Thanks to the completeness of \(L^2\) as a Hilbert space, this approach is structurally cleaner than the \(L^1\)
inversion theory. The normalized map \(f \mapsto \hat f / \sqrt{2\pi}\) becomes a unitary operator
on \(L^2(\mathbb{R})\), whose inverse coincides with its Hilbert-space adjoint. That single fact delivers an
\(L^2\) inversion formula and a Plancherel identity on all of \(L^2\) at once. This construction is carried out on
Fourier analysis in Hilbert spaces.
Convolution Theorem
The differentiation property above shows that the Fourier transform simplifies differentiation. The convolution
theorem reveals an even more powerful algebraic simplification. It converts the integral operation of
convolution, which costs \(O(n^2)\) to compute directly, into pointwise multiplication in the
frequency domain. This single result is the reason FFT-based convolution outperforms direct convolution for
sufficiently long filters and large kernels, and it underlies frequency-domain filtering throughout signal
processing.
The convolution of two functions \(f, g: \mathbb{R} \to \mathbb{C}\) is defined as:
\[
(f * g)(x) = \int_{-\infty}^{\infty} f(y)g(x-y) \, dy = \int_{-\infty}^{\infty} f(x-y)g(y) \, dy
\]
(The second equality shows convolution is commutative.)
Theorem: Convolution Theorem
For \(f, g \in L^1(\mathbb{R})\), the Fourier transform converts convolution into pointwise
multiplication:
\[
\mathcal{F}\{f * g\}(\xi) = \hat{f}(\xi) \cdot \hat{g}(\xi)
\]
and conversely, whenever \(f, g \in L^1(\mathbb{R})\) with \(\hat f \in L^1(\mathbb{R})\) (in
particular for \(f, g\) in the Schwartz class), the Fourier transform of a product is a
(scaled) convolution:
\[
\mathcal{F}\{f \cdot g\}(\xi) = \frac{1}{2\pi}(\hat{f} * \hat{g})(\xi).
\]
This is one of the most important results in applied mathematics, and it allows us to:
- Replace expensive convolution operations (\(O(n^2)\) for discrete signals) with multiplication after
FFT (\(O(n \log n)\))
- Understand filtering as multiplication in the frequency domain
- Analyze linear time-invariant systems through their frequency response
Insight: Convolution Theorem in CS and ML
- Signal filtering.
Low-pass, high-pass, and band-pass filters are convolutions in the time domain but simple multiplications in the
frequency domain
- Image processing. Gaussian blur, Sobel edge filters, and sharpening are convolution
operations, efficiently computed via FFT for large kernels. Canny edge detection begins with such
convolutions.
- Deep learning. While CNNs typically use small kernels \((3 \times 3, 5 \times 5)\) where direct
convolution is efficient, FFT-based convolution is useful for:
- Large kernels in certain architectures
- Global convolutions in some vision transformers
- Probability. The PDF of \(X + Y\) for independent random variables is
\(f_{X+Y} = f_X * f_Y\), computed efficiently via FFT
Proof (convolution to product):
Using the convention of this site with \(\hat{f}(\xi) = \int f(x)e^{ix\xi} \, dx\):
\[
\begin{align*}
\mathcal{F}\{f * g\}(\xi) &= \int_{-\infty}^{\infty} (f * g)(x)e^{ix\xi} \, dx \\\\
&= \int_{-\infty}^{\infty} \left(\int_{-\infty}^{\infty} f(y)g(x-y) \, dy\right) e^{ix\xi} \, dx \\\\
&= \int_{-\infty}^{\infty} f(y) \left(\int_{-\infty}^{\infty} g(x-y)e^{ix\xi} \, dx\right) dy.
\end{align*}
\]
Substituting \(u = x - y\) (so \(x = u + y\) and \(dx = du\)):
\[
\begin{align*}
&= \int_{-\infty}^{\infty} f(y) \left(\int_{-\infty}^{\infty} g(u)e^{i(u+y)\xi} \, du\right) dy \\\\
&= \int_{-\infty}^{\infty} f(y)e^{iy\xi} \left(\int_{-\infty}^{\infty} g(u)e^{iu\xi} \, du\right) dy \\\\
&= \left(\int_{-\infty}^{\infty} f(y)e^{iy\xi} \, dy\right) \cdot \left(\int_{-\infty}^{\infty} g(u)e^{iu\xi} \, du\right) \\\\
&= \hat{f}(\xi) \cdot \hat{g}(\xi).
\end{align*}
\]
The interchange of integration order is justified by
Fubini's theorem
when \(f, g \in L^1(\mathbb{R})\), applied to the real and imaginary parts of the integrand
separately.
Proof (product to convolution):
Under sufficient regularity (for instance, \(f, g \in L^1\) with \(\hat f \in L^1\)), the inversion formula gives
\[
f(x) = \frac{1}{2\pi}\int \hat f(\eta) e^{-ix\eta}\, d\eta.
\]
Substituting into the Fourier transform of \(fg\) and applying
Fubini's theorem
to the real and imaginary parts separately:
\[
\begin{align*}
\mathcal{F}\{f \cdot g\}(\xi)
&= \int_{-\infty}^{\infty} \left[\frac{1}{2\pi}\int_{-\infty}^{\infty} \hat f(\eta)\, e^{-ix\eta}\, d\eta\right] g(x)\, e^{ix\xi}\, dx \\\\
&= \frac{1}{2\pi}\int_{-\infty}^{\infty} \hat f(\eta)\left[\int_{-\infty}^{\infty} g(x)\, e^{ix(\xi - \eta)}\, dx\right] d\eta \\\\
&= \frac{1}{2\pi}\int_{-\infty}^{\infty} \hat f(\eta)\, \hat g(\xi - \eta)\, d\eta
= \frac{1}{2\pi}(\hat f * \hat g)(\xi).
\end{align*}
\]
Discrete Fourier Transform (DFT)
The continuous Fourier transform operates on functions defined on all of \(\mathbb{R}\), but computers work with finite,
sampled data. The Discrete Fourier Transform (DFT) bridges this gap. It is the exact discrete
analogue of the continuous transform, operating on sequences of \(N\) complex numbers and producing \(N\) frequency
coefficients. The DFT inherits all the key properties of its continuous counterpart in discrete form. They include
linearity, shift, and convolution.
Note on Sign Convention for DFT
In the continuous section above, we used the convention of this site (\(e^{+ix\xi}\)), chosen to match the
characteristic functions of probability theory. However, in the discrete domain (DFT/FFT), the
Engineering/Data Science convention (negative sign in the forward transform) is the convention of
the standard FFT libraries such as NumPy, PyTorch, and MATLAB.
To ensure that the formulas below match the code you will write in Python, we switch to the standard
computational convention for the remainder of this page:
\[
\text{DFT: } e^{-i \dots}, \quad \text{IDFT: } e^{+i \dots}.
\]
Definition: DFT & IDFT
For a sequence \(\{x_0, x_1, \ldots, x_{N-1}\}\) of \(N\) complex numbers, the DFT is:
\[
X_k = \sum_{n=0}^{N-1} x_n e^{-\frac{2\pi i}{N}kn}, \quad k = 0, 1, \ldots, N-1.
\]
The inverse DFT (IDFT) is:
\[
x_n = \frac{1}{N}\sum_{k=0}^{N-1} X_k e^{\frac{2\pi i}{N}kn}, \quad n = 0, 1, \ldots, N-1.
\]
Connection to Continuous Transform. The DFT can be viewed as:
- Sampling a periodic function at \(N\) equally spaced points
- Computing Fourier series coefficients for the periodized version
- Approximating the continuous Fourier transform for band-limited signals (via the Shannon-Nyquist sampling theorem)
Matrix Formulation:
Define \(\omega = e^{-\frac{2\pi i}{N}}\), a primitive \(N\)-th root of unity
(so \(\omega^N = 1\)). The DFT matrix is:
\[
W = \begin{bmatrix}
1 & 1 & 1 & \cdots & 1 \\\\
1 & \omega & \omega^2 & \cdots & \omega^{N-1} \\\\
1 & \omega^2 & \omega^4 & \cdots & \omega^{2(N-1)} \\\\
\vdots & \vdots & \vdots & \ddots & \vdots \\\\
1 & \omega^{N-1} & \omega^{2(N-1)} & \cdots & \omega^{(N-1)^2}
\end{bmatrix}
\]
where \(W_{kn} = \omega^{kn}\). Then:
\[
\mathbf{X} = W\mathbf{x}, \quad \mathbf{x} = \frac{1}{N}W^*\mathbf{X}
\]
where \(W^*\) is the conjugate transpose.
Properties of the DFT Matrix:
- Orthogonality: \(W^*W = NI\), so \(W^{-1} = \frac{1}{N}W^*\)
- Symmetry. The matrix \(W\) is symmetric: \(W^\top = W\)
- Fourth-power identity and eigenvalues. Writing out \((W^2)_{ij} = \sum_k \omega^{k(i+j)}\)
shows \(W^2 = N \cdot P\), where \(P\) is the permutation matrix implementing \(n \mapsto -n \bmod N\).
Since \(P^2 = I\), we get \(W^4 = N^2 I\). Equivalently, the normalized DFT matrix
\(F = \tfrac{1}{\sqrt{N}}\, W\) satisfies \(F^4 = I\). The eigenvalues of \(F\) therefore lie in
\(\{+1, -1, +i, -i\}\), while the eigenvalues of the unnormalized \(W\) are confined to the
scaled quartet \(\{\pm\sqrt{N}, \pm i\sqrt{N}\}\).
- Circulant diagonalization. The DFT matrix diagonalizes all circulant matrices
Normalization conventions across domains
The choice between the unnormalized \(W\) and the normalized \(F = \tfrac{1}{\sqrt{N}} W\) is
not a mere cosmetic rescaling. It reflects which property of the transform the user cares
about most. Three regimes coexist in practice:
- Signal processing / scientific computing (unnormalized \(W\)):
NumPy, SciPy, MATLAB, PyTorch (torch.fft.fft), and JAX all default to
forward DFT without normalization, placing the full \(1/N\) factor in the inverse. FFTW leaves both
directions unnormalized.
This is the long-standing numerical-computation convention, reinforced by
Cooley-Tukey 1965 and its descendants. It has two practical advantages: the convolution theorem
\(\mathcal{F}(f * g) = \hat f \cdot \hat g\) holds with no stray constants, and only the
inverse transform pays the \(1/N\) cost. The price is that the forward transform is
not an isometry. Norms are scaled by \(\sqrt{N}\).
- Harmonic analysis / functional analysis (normalized \(F\)):
The discrete Plancherel identity \(\|f\|_2 = \|Ff\|_2\) requires the normalized version. The map
\(F : \mathbb{C}^N \to \mathbb{C}^N\) is then a unitary operator
(\(F^*F = I\)), and all of \(L^2\)-based Fourier theory transfers to the discrete setting
without rescaling. The clean eigenvalue structure \(\{\pm 1, \pm i\}\) derived above is a
symptom of the fourth-power identity \(F^4 = I\), with unitarity forcing the roots onto
the unit circle.
- Quantum computation (normalized \(F\) is mandatory):
The
Quantum Fourier Transform, the key subroutine behind Shor's algorithm and phase
estimation, is, in its standard definition, the matrix \(\overline{F} = F^{-1}\), the normalized DFT with
the opposite sign in the exponent. It must be unitary, because every gate operation on qubits is a unitary
transformation by the postulates of quantum mechanics. There is no choice. The normalized version is the
only one that corresponds to a physically realizable quantum gate.
Most numerical libraries expose both options via a norm= argument. NumPy's
np.fft.fft(x, norm='ortho') returns the normalized transform, and
norm='backward' (the default) returns the unnormalized one. When porting a
formula from a textbook to code (or vice versa), the first question to ask is always:
which convention is this written in?
Computational Complexity:
- Direct computation: \(O(N^2)\) complex multiplications and additions
- Fast Fourier Transform: \(O(N \log N)\) operations (see next section)
Practical Considerations for CS/ML:
- Zero-padding. Padding sequences to powers of 2 enables efficient radix-2 FFT algorithms
- Windowing. Applying window functions (Hamming, Hann, Blackman) reduces spectral leakage from
finite-length effects
- Real-valued signals. For real inputs, \(X_{N-k} = \overline{X_k}\) (Hermitian symmetry). Only
half the spectrum needs to be computed, which is what the rfft routines exploit
Applications in Machine Learning
Fourier methods are fundamental to modern machine learning, particularly in areas requiring efficient computation,
signal processing, and function approximation. Here we highlight three significant applications.
1. Audio and Speech Processing
Speech recognition systems such as OpenAI's Whisper (2022) and its open-source successors
convert raw audio into log-Mel spectrograms via the Short-Time Fourier Transform (STFT). The
audio is windowed into overlapping segments, each transformed via FFT to obtain frequency content, then mapped
to the Mel scale (which approximates human auditory perception). This frequency-domain representation serves as
input to the neural network, typically a Transformer encoder. The result is robust transcription across
languages and noisy environments.
Reference: Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision" (2022)
2. Fourier Neural Operators (FNO) for Scientific Computing
Introduced by Li et al. (ICLR 2021), Fourier Neural Operators (FNOs) learn solution operators of parametric PDEs
directly in Fourier space. They achieve up to three orders of magnitude speedup over traditional PDE solvers on
benchmark problems such as the Burgers and Navier-Stokes equations. The key idea is to parameterize the kernel
\(\hat{k}(\xi)\) in the frequency domain, then multiply with the Fourier transform of the input. The approach has
since inspired a rich family of spectral neural operators.
FNO Architecture Sketch
An FNO performs the following steps, repeating steps 2 to 4 in each Fourier layer:
- Lift input to higher-dimensional channel space
- Apply spectral convolution: \(\mathcal{F}^{-1}(R \cdot \mathcal{F}(v))\) where \(R\) is a learnable weight matrix
- Add the pointwise linear term \(Wv\)
- Apply non-linearity (activation function), for example GELU
- Project to output
The spectral convolution step is where the convolution theorem provides the key efficiency gain.
Reference: Li et al., "Fourier Neural Operator for Parametric PDEs"
3. Fourier Features as Positional Encoding
Neural networks exhibit a spectral bias. They tend to learn low-frequency functions more easily
than high-frequency ones. Fourier feature mappings overcome this limitation by lifting low-dimensional inputs
into higher-dimensional spaces via sinusoidal functions. The Transformer architecture (Vaswani et al., 2017) introduced
sinusoidal positional encoding for sequence positions. This idea was later adapted to continuous spatial
coordinates, with Neural Radiance Fields (NeRF, 2020) mapping a scalar coordinate \(p\) to
\[
\gamma(p) = \left[\sin(2^0 \pi p), \cos(2^0 \pi p), \ldots, \sin(2^{L-1} \pi p), \cos(2^{L-1} \pi p)\right]
\]
so that neural networks can capture fine geometric details. Tancik et al. (2020) gave a unified Neural Tangent
Kernel analysis explaining why such encodings tame spectral bias.
The same Fourier-feature principle reappears in modern Large Language Models as Rotary Position Embedding
(RoPE), which encodes token positions through rotations in 2D subspaces of the query/key vectors and has become
a de facto standard across recent LLM families.
References:
Mildenhall et al., "NeRF: Representing Scenes as Neural Radiance Fields" (2020)
Tancik et al., "Fourier Features Let Networks Learn High Frequency Functions" (NeurIPS 2020)
Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding" (2021)