Linear Approximations
In the study of complex systems, from the orbits of planets to the loss landscapes of deep neural
networks, most functions are inherently nonlinear and difficult to solve directly. Linear
approximation is the fundamental strategy of calculus. It approximates a complex function \(f(x)\)
near a specific point \(x_o\) using the simplest possible tool, an affine function, a linear function
plus a constant.
Asymptotic notation. Throughout this page (and the rest of this section) we use the
Landau symbols. We write \(\varphi(h) = o(h)\) as \(h \to 0\) to mean
\(\lim_{h \to 0} \varphi(h)/|h| = 0\). That is, \(\varphi\) vanishes strictly faster than
\(h\). More generally \(\varphi(h) = o(|h|^k)\) means \(\varphi(h)/|h|^k \to 0\). The companion
symbol \(\varphi(h) = O(h)\) means \(|\varphi(h)| \leq C|h|\) for some constant \(C\) and all \(h\)
near \(0\). In the multivariate case, \(h\) is replaced by a vector \(\mathbf{h}\) and \(|h|\) by a
norm \(\|\mathbf{h}\|\). When \(\varphi\) is vector- or matrix-valued, \(|\varphi(h)|\) is likewise
replaced by a norm of the output.
These abbreviations let us separate the linear part of a change from everything that
vanishes faster. A differential, and then a linearization, capture exactly that separation.
Why norms enter. A derivative is fundamentally a comparison between the size of an
input displacement and the size of the resulting output change. On \(\mathbb{R}\) the absolute value
\(|h|\) plays this role implicitly, so the point passes unnoticed. Once inputs or outputs live in
higher-dimensional spaces of vectors, matrices, and eventually functions, we need a genuine measure
of magnitude, and that measure is a norm. The statement
"\(f(\mathbf{x}+\mathbf{h}) - f(\mathbf{x}) - f'(\mathbf{x})\mathbf{h} = o(\|\mathbf{h}\|)\)" is not just a
compact notation. It is the definition of differentiability, formulated so that both the
displacement and the residual are measured in compatible units.
In finite dimensions all norms are equivalent (a fact proved later in this section when we turn to
normed spaces), so the choice is inessential for pointwise calculus.
In infinite-dimensional settings the choice becomes substantive. Different norms induce different
notions of continuity and differentiability, which is why functional analysis begins, rather than
ends, with the choice of norm.
Definition: Linearization
Let \(f : \mathbb{R} \to \mathbb{R}\) be differentiable at \(x_o\), with derivative
\[
f'(x_o) = \lim_{x \to x_o} \frac{f(x)-f(x_o)}{x-x_o}.
\]
The linearization of \(f\) at \(x_o\) is the affine function
\[
L(x) = f(x_o) + f'(x_o)(x - x_o).
\]
It is the unique affine function satisfying \(L(x_o) = f(x_o)\) and \(L'(x_o) = f'(x_o)\).
Equivalently, it is characterized by
\[
f(x) - L(x) = o(x - x_o) \quad \text{as } x \to x_o.
\]
Geometrically, the graph of \(L\) is the tangent line at \((x_o, f(x_o))\). The characterization
\(f(x) - L(x) = o(x-x_o)\) is the precise statement of what "\(L(x) \approx f(x)\) near \(x_o\)"
means. The approximation error vanishes faster than the displacement itself.
This local linear model is what allows optimization algorithms to take principled "steps" toward a
minimum, even when the global shape of the function is unknown. We now refine this idea into the
language of differentials.
Differentials
Linearization produces an affine approximation \(L(x)\) to \(f(x)\). The
differential isolates the linear part of the change in \(f\) as an object
in its own right. This shift replaces "value near \(x_o\)" with "linear map on displacements". The
same move carries scalar calculus into the vector, matrix, and operator settings used throughout
optimization and machine learning.
Definition: Differential (scalar case)
Let \(f : \mathbb{R} \to \mathbb{R}\) be differentiable at \(x\). The
differential of \(f\) at \(x\) is the linear map
\[
\begin{align*}
df &: \mathbb{R} \to \mathbb{R}, \\\\
df(dx) &= f'(x)\, dx.
\end{align*}
\]
Here \(dx\) denotes an arbitrary real displacement (the input of the linear map), and
we write the value of that map as \(df\) or \(df(dx)\) interchangeably when no confusion arises.
The differential is related to, but not equal to, the finite increment
\(\Delta f := f(x+dx) - f(x)\). Differentiability at \(x\) is precisely the statement that these two
agree up to a term that vanishes faster than \(dx\):
\[
\begin{align*}
\Delta f &= f(x+dx) - f(x) \\\\
&= f'(x)\,dx + o(dx) \quad \text{as } dx \to 0.
\end{align*}
\]
In other words, \(df = f'(x)\,dx\) is the unique linear map whose deviation from the true increment
\(\Delta f\) is higher-order. (Uniqueness: if two linear maps \(\ell_1, \ell_2\) both satisfy
\(\Delta f = \ell_i(dx) + o(dx)\), then \(\ell_1 - \ell_2 = o(dx)\) is a linear map that is \(o(dx)\), hence
identically zero.)
When \(dx \neq 0\), dividing gives the familiar Leibniz ratio:
\[
\frac{df}{dx} = f'(x),
\]
where the left side is now genuinely a ratio of values of the linear map \(df\) to its input.
Derivative as an Operator
In foundational calculus we often picture \(f'(x)\) as a static number, the slope. The
differential reframes it as a linear operator. The single-number derivative
\(f'(x)\) is the \(1 \times 1\) matrix of this operator.
In the identity \(df = f'(x)\,dx\), the derivative acts as a "transformer" that maps a
displacement in the input space (\(dx\)) to a displacement in the output space (\(df\)). This
perspective is essential once \(x\) is no longer a scalar but a vector
\(\mathbf{x} \in \mathbb{R}^n\) or a matrix \(X \in \mathbb{R}^{m \times n}\). There, \(f'\) is
no longer a number but a linear map, represented by the matrix of partial derivatives called
the Jacobian (for matrix inputs, after listing the entries of \(X\) in a single
column).
In the scalar-output case, the transpose of the Jacobian yields the gradient.
Multivariate Extension and the Gradient
We now pass from scalar inputs \(x \in \mathbb{R}\) to vector inputs
\(\mathbf{x} \in \mathbb{R}^n\). From here on, bold symbols
\(\mathbf{x}, \mathbf{y}, d\mathbf{x}\) denote vectors. Plain italic \(x_i\) denotes the \(i\)-th
scalar component. For a scalar-valued differentiable function \(f : \mathbb{R}^n \to \mathbb{R}\),
restricting the definition of differentiability to the line through \(\mathbf{x}\) in direction
\(\mathbf{e}_i\) shows \(df(\mathbf{e}_i) = \partial f/\partial x_i\), and linearity then gives
the differential at \(\mathbf{x}\) as the linear map
\[
\begin{align*}
df &: \mathbb{R}^n \to \mathbb{R}, \\\\
df(d\mathbf{x}) &= \sum_{i=1}^n \frac{\partial f}{\partial x_i}\, dx_i.
\end{align*}
\]
This linear functional, being represented by a single row of partial derivatives, can be written as
an inner product with a uniquely determined column vector.
Convention (Gradient). For a scalar-valued function
\(f : \mathbb{R}^n \to \mathbb{R}\), the gradient \(\nabla f\) is the
column vector in \(\mathbb{R}^n\) defined so that the differential takes the inner-product
form, or equivalently the componentwise form
\[
\begin{align*}
df &= (\nabla f)^\top\, d\mathbf{x}, \\\\
(\nabla f)_i &= \frac{\partial f}{\partial x_i}.
\end{align*}
\]
In practice, when a computation yields \(df = \mathbf{v}^\top\, d\mathbf{x}\) for some column vector
\(\mathbf{v}\), we read off the gradient as \(\nabla f = \mathbf{v}\).
The same increment-and-linear-part story transfers verbatim:
\[
\begin{align*}
\Delta f &= f(\mathbf{x} + d\mathbf{x}) - f(\mathbf{x}) \\\\
&= (\nabla f)^\top\, d\mathbf{x} + o(\|d\mathbf{x}\|) \quad \text{as } d\mathbf{x} \to \mathbf{0}.
\end{align*}
\]
The three examples that follow all instantiate a single pattern. We expand \(\Delta f\), collect
terms linear in \(d\mathbf{x}\), and read off \(\nabla f\) from the resulting row vector.
Vector Derivatives: Optimization Foundations
In machine learning, we rarely
optimize single scalar variables. We optimize weight vectors. The differential
calculus just developed is the right tool. Rather than writing out \(n\) partial derivatives by
hand, we manipulate \(d(\cdot)\) symbolically and read off the gradient from the row-vector
coefficient of \(d\mathbf{x}\). The three examples below are the squared norm, the quadratic form,
and the \(L_2\) norm. Together they cover the gradients that appear most often in standard
regularization and in the quadratic models of second-order optimization. We first state the product
rule in the differential form that will be used throughout.
Theorem: Differential Product Rule
Let \(g\) and \(h\) be differentiable at \(x\), and let \(f(x) = g(x)h(x)\). Then \(f\) is
differentiable at \(x\), and its differential satisfies
\[
df = (dg)\,h + g\,(dh).
\]
In product-compatible settings (vectors, matrices) where \(g\) and \(h\) need not commute, the
order of factors in each term is preserved.
Proof.
Let \(\Delta g := g(x+dx) - g(x)\) and \(\Delta h := h(x+dx) - h(x)\). By differentiability,
\(\Delta g = dg + o(dx)\) and \(\Delta h = dh + o(dx)\). Expanding the increment of \(f\):
\[
\begin{align*}
\Delta f &= g(x+dx)\,h(x+dx) - g(x)\,h(x) \\\\
&= \bigl[g(x) + \Delta g\bigr]\bigl[h(x) + \Delta h\bigr] - g(x)\,h(x) \\\\
&= (\Delta g)\,h(x) + g(x)\,(\Delta h) + (\Delta g)(\Delta h).
\end{align*}
\]
The first two terms contribute the linear part \((dg)\,h(x) + g(x)\,(dh)\), up to \(o(dx)\).
Here we use that \(h(x)\) and \(g(x)\) are fixed (independent of \(dx\)), so multiplication by
either preserves \(o(dx)\). In the vector/matrix setting the same step holds because each entry
of a product with a fixed factor is a fixed linear combination of the entries of the other factor,
so multiplication by \(g(x)\) or \(h(x)\) preserves \(o(dx)\) entrywise. The cross term
\((\Delta g)(\Delta h)\) is \(O(dx)\cdot O(dx) = o(dx)\), since \(\Delta g = dg + o(dx) = O(dx)\)
and likewise for \(\Delta h\). Therefore
\[
\Delta f = (dg)\,h + g\,(dh) + o(dx),
\]
which identifies \(df = (dg)\,h + g\,(dh)\) as the linear part of \(\Delta f\).
Example 1: Squared \(L_2\) Norm \(f(\mathbf{x}) = \mathbf{x}^\top\mathbf{x}\), \(\mathbf{x} \in \mathbb{R}^n\)
This function represents the squared distance from the origin, a core component of
Mean Squared Error (MSE). The input is the vector \(\mathbf{x}\), and the output is
the scalar \(\mathbf{x}^\top\mathbf{x}\).
Example 1.
Applying the
differential product rule
to \(f(\mathbf{x}) = \mathbf{x}^\top\mathbf{x}\):
\[
\begin{align*}
d(\mathbf{x}^\top\mathbf{x})
&= (d\mathbf{x}^\top)\,\mathbf{x} + \mathbf{x}^\top\,(d\mathbf{x}) \\\\
&= \mathbf{x}^\top\,d\mathbf{x} + \mathbf{x}^\top\,d\mathbf{x} \\\\
&= 2\,\mathbf{x}^\top\, d\mathbf{x},
\end{align*}
\]
where the first line uses the fact that \(d(\mathbf{x}^\top) = (d\mathbf{x})^\top\), since
transposition is itself a linear operation. The second line uses the scalar identity
\(\mathbf{a}^\top\mathbf{b} = \mathbf{b}^\top\mathbf{a}\) with \(\mathbf{a} = d\mathbf{x}\),
\(\mathbf{b} = \mathbf{x}\).
Reading off the row vector \(2\,\mathbf{x}^\top\) as the transpose of the gradient:
\[
\nabla f = (2\,\mathbf{x}^\top)^\top = 2\mathbf{x}.
\]
Sanity check (entrywise). Writing \(f(\mathbf{x}) = \sum_{i=1}^n x_i^2\) and
differentiating componentwise yields \(\partial f / \partial x_i = 2 x_i\), so
\[
\nabla f = \begin{bmatrix} 2x_1 \\\\ 2x_2 \\\\ \vdots \\\\ 2x_n \end{bmatrix} = 2\mathbf{x},
\]
in agreement with the differential derivation. For the remaining examples the entrywise route would
be progressively more painful. The differential calculation generalizes effortlessly.
Quadratic Forms \(f(\mathbf{x}) = \mathbf{x}^\top A\mathbf{x}\)
Quadratic forms are essential for
modeling local curvature. Specifically, the second-order term of a Taylor expansion is one-half the
quadratic form of the Hessian matrix. Let \(\mathbf{x} \in \mathbb{R}^n\) and
\(A \in \mathbb{R}^{n \times n}\).
Example 2.
Expanding the increment:
\[
\begin{align*}
\Delta f &= (\mathbf{x} + d\mathbf{x})^\top A\,(\mathbf{x} + d\mathbf{x}) - \mathbf{x}^\top A\mathbf{x} \\\\
&= d\mathbf{x}^\top A\,\mathbf{x} + \mathbf{x}^\top A\, d\mathbf{x} + d\mathbf{x}^\top A\, d\mathbf{x}.
\end{align*}
\]
The last term is bounded by a constant times \(\|d\mathbf{x}\|^2\), hence \(o(\|d\mathbf{x}\|)\),
so the linear part is
\[
df = d\mathbf{x}^\top A\,\mathbf{x} + \mathbf{x}^\top A\, d\mathbf{x}.
\]
The first term is a scalar, hence equal to its own transpose:
\((d\mathbf{x}^\top A\,\mathbf{x})^\top = \mathbf{x}^\top A^\top\, d\mathbf{x}\). Substituting,
\[
\begin{align*}
df &= \mathbf{x}^\top A^\top\, d\mathbf{x} + \mathbf{x}^\top A\, d\mathbf{x} \\\\
&= \mathbf{x}^\top(A + A^\top)\, d\mathbf{x}.
\end{align*}
\]
Reading off the row vector \(\mathbf{x}^\top(A + A^\top)\) as the transpose of the gradient (per
the convention established in Differentials):
\[
\nabla f = (A + A^\top)\,\mathbf{x}.
\]
Symmetry and Optimization Efficiency
In many optimization contexts, for instance in the second-order Taylor expansion of a loss
function, the matrix \(A\) is one-half the Hessian, which is symmetric whenever
the loss is twice continuously differentiable: \(A = A^\top\).
In this case the gradient simplifies to
\[
\nabla f = (A + A^\top)\,\mathbf{x} = 2A\,\mathbf{x}.
\]
Symmetry is not a mathematical curiosity. Numerical solvers exploit it to reduce memory and
computation by roughly half (for example, storing only the upper triangle, and, when the matrix is
also positive definite, using Cholesky in place of general LU and the conjugate-gradient method
instead of GMRES).
Moreover, Example 1 is recovered as the special case \(A = I\), for which
\(\nabla (\mathbf{x}^\top\mathbf{x}) = 2I\mathbf{x} = 2\mathbf{x}\), matching our earlier
result.
The \(L_2\) Norm \(f(\mathbf{x}) = \|\mathbf{x}\|_2\)
The \(L_2\) norm, or Euclidean norm
is the building block of one of the most common regularization terms. Weight decay
penalizes its square, the function of Example 1. The derivative of the norm itself, computed here,
is what separates the unsquared penalty from the squared one.
Example 3.
Let \(r := \|\mathbf{x}\|_2\), so that \(r^2 = \mathbf{x}^\top\mathbf{x}\). Assume
\(\mathbf{x} \neq \mathbf{0}\), so that \(r \gt 0\) and \(r\) is differentiable at
\(\mathbf{x}\) by the
chain rule,
applied to \(\sqrt{\,\cdot\,}\), which is smooth away from \(0\), composed with the polynomial
\(\mathbf{x}^\top\mathbf{x}\). Taking the differential of both
sides of \(r^2 = \mathbf{x}^\top\mathbf{x}\), and using \(d(r^2) = 2r\,dr\) on the left (scalar
chain rule) and the result of Example 1 on the right, we obtain
\[
\begin{align*}
d(r^2) &= d(\mathbf{x}^\top\mathbf{x}) \\\\
2r\, dr &= 2\,\mathbf{x}^\top\, d\mathbf{x} \\\\
dr &= \frac{\mathbf{x}^\top}{r}\, d\mathbf{x}
= \frac{\mathbf{x}^\top}{\|\mathbf{x}\|_2}\, d\mathbf{x}.
\end{align*}
\]
By the gradient convention (Differentials), the gradient is therefore
\[
\nabla \|\mathbf{x}\|_2 = \frac{\mathbf{x}}{\|\mathbf{x}\|_2}.
\]
Note that the norm is not differentiable at \(\mathbf{x} = \mathbf{0}\). The formula above holds
only on \(\mathbb{R}^n \setminus \{\mathbf{0}\}\).
Geometric Interpretation: The Unit Vector
The gradient of the \(L_2\) norm is the unit vector pointing in the direction of
\(\mathbf{x}\).
In gradient descent, this means that a
penalty on the unsquared norm pulls the weights toward the origin with a force whose magnitude is
independent of their scale. As a result, for a convex loss the zero vector minimizes the penalized
loss whenever the gradient of the loss at the origin is no longer than the penalty weight. This is
how group-sparsity penalties set a whole group of weights exactly to zero. Weight decay, by
contrast, penalizes the squared norm of Example 1, whose gradient \(2\mathbf{x}\) shrinks the
weights in proportion to their size. Zero then minimizes the penalized loss only when the gradient
of the loss vanishes at the origin, which rarely happens. \(L_1\) regularization applies the same
constant-magnitude pull coordinate by coordinate, which is why its minimizers tend to be sparse.