Linear Approximations
In the study of complex systems, from the orbits of planets to the loss landscapes of deep neural
networks, most functions are inherently nonlinear and difficult to solve directly. Linear
approximation is the fundamental strategy of calculus. It approximates a complex function \(f(x)\)
near a specific point \(x_o\) using the simplest possible tool, a linear function.
Asymptotic notation. Throughout this page (and the rest of this section) we use the
Landau symbols. We write \(\varphi(h) = o(h)\) as \(h \to 0\) to mean
\(\lim_{h \to 0} \varphi(h)/|h| = 0\). That is, \(\varphi\) vanishes strictly faster than
\(h\). More generally \(\varphi(h) = o(|h|^k)\) means \(\varphi(h)/|h|^k \to 0\). The companion
symbol \(\varphi(h) = O(h)\) means \(|\varphi(h)| \leq C|h|\) for some constant \(C\) and all \(h\)
near \(0\). In the multivariate case, \(h\) is replaced by a vector \(\mathbf{h}\) and \(|h|\) by a
norm \(\|\mathbf{h}\|\).
These abbreviations let us separate the linear part of a change from everything that
vanishes faster. A differential, and then a linearization, capture exactly that separation.
Why norms enter. A derivative is fundamentally a comparison between the size of an
input displacement and the size of the resulting output change. On \(\mathbb{R}\) the absolute value
\(|h|\) plays this role implicitly, so the point passes unnoticed. Once inputs or outputs live in
higher-dimensional spaces of vectors, matrices, and eventually functions, we need a genuine measure
of magnitude, and that measure is a norm. The statement
"\(f(x+\mathbf{h}) - f(x) - f'(x)\mathbf{h} = o(\|\mathbf{h}\|)\)" is not just a compact notation.
It is the definition of differentiability, formulated so that both the displacement and the
residual are measured in compatible units.
In finite dimensions all norms are equivalent, so the choice is inessential for pointwise calculus.
In infinite-dimensional settings the choice becomes substantive. Different norms induce different
notions of continuity and differentiability, which is why functional analysis begins, rather than
ends, with the choice of norm.
Definition: Linearization
Let \(f : \mathbb{R} \to \mathbb{R}\) be differentiable at \(x_o\), with derivative
\[
f'(x_o) = \lim_{x \to x_o} \frac{f(x)-f(x_o)}{x-x_o}.
\]
The linearization of \(f\) at \(x_o\) is the affine function
\[
L(x) = f(x_o) + f'(x_o)(x - x_o).
\]
It is the unique affine function satisfying \(L(x_o) = f(x_o)\) and \(L'(x_o) = f'(x_o)\).
Equivalently, it is characterized by
\[
f(x) - L(x) = o(x - x_o) \quad \text{as } x \to x_o.
\]
Geometrically, the graph of \(L\) is the tangent line at \((x_o, f(x_o))\). The characterization
\(f(x) - L(x) = o(x-x_o)\) is the precise statement of what "\(L(x) \approx f(x)\) near \(x_o\)"
means. The approximation error vanishes faster than the displacement itself.
This local linear model is what allows optimization algorithms to take principled "steps" toward a
minimum, even when the global shape of the function is unknown. We now refine this idea into the
language of differentials.
Differentials
Linearization produces an affine approximation \(L(x)\) to \(f(x)\). The
differential isolates the linear part of the change in \(f\) as an object
in its own right. This shift replaces "value near \(x_o\)" with "linear map on displacements". The
same move carries scalar calculus into the vector, matrix, and operator settings used throughout
optimization and machine learning.
Definition: Differential (scalar case)
Let \(f : \mathbb{R} \to \mathbb{R}\) be differentiable at \(x\). The
differential of \(f\) at \(x\) is the linear map
\[
\begin{align*}
df &: \mathbb{R} \to \mathbb{R}, \\\\
df(dx) &= f'(x)\, dx.
\end{align*}
\]
Here \(dx\) denotes an arbitrary real displacement (the input of the linear map), and
we write the value of that map as \(df\) or \(df(dx)\) interchangeably when no confusion arises.
The differential is related to, but not equal to, the finite increment
\(\Delta f := f(x+dx) - f(x)\). Differentiability at \(x\) is precisely the statement that these two
agree up to a term that vanishes faster than \(dx\):
\[
\begin{align*}
\Delta f &= f(x+dx) - f(x) \\\\
&= f'(x)\,dx + o(dx) \quad \text{as } dx \to 0.
\end{align*}
\]
In other words, \(df = f'(x)\,dx\) is the unique linear map whose deviation from the true increment
\(\Delta f\) is higher-order. (Uniqueness: if two linear maps \(L_1, L_2\) both satisfy
\(\Delta f = L_i(dx) + o(dx)\), then \(L_1 - L_2 = o(dx)\) is a linear map that is \(o(dx)\), hence
identically zero.)
When \(dx \neq 0\), dividing gives the familiar Leibniz ratio:
\[
\frac{df}{dx} = f'(x),
\]
where the left side is now genuinely a ratio of values of the linear map \(df\) to its input.
Derivative as an Operator
In foundational calculus we often picture \(f'(x)\) as a static number, the slope. The
differential reframes it as a linear operator. The single-number derivative
\(f'(x)\) is the \(1 \times 1\) matrix of this operator.
In the identity \(df = f'(x)\,dx\), the derivative acts as a "transformer" that maps a
displacement in the input space (\(dx\)) to a displacement in the output space (\(df\)). This
perspective is essential once \(x\) is no longer a scalar but a vector
\(\mathbf{x} \in \mathbb{R}^n\) or a matrix \(X \in \mathbb{R}^{m \times n}\). There, \(f'\) is
no longer a number but the matrix of partial derivatives called the Jacobian.
In the scalar-output case, the transpose of the Jacobian yields the gradient.
Multivariate Extension and the Gradient
We now pass from scalar inputs \(x \in \mathbb{R}\) to vector inputs
\(\mathbf{x} \in \mathbb{R}^n\). From here on, bold symbols
\(\mathbf{x}, \mathbf{y}, d\mathbf{x}\) denote vectors. Plain italic \(x_i\) denotes the \(i\)-th
scalar component. For a scalar-valued differentiable function \(f : \mathbb{R}^n \to \mathbb{R}\),
the differential at \(\mathbf{x}\) is the linear map
\[
\begin{align*}
df &: \mathbb{R}^n \to \mathbb{R}, \\\\
df(d\mathbf{x}) &= \sum_{i=1}^n \frac{\partial f}{\partial x_i}\, dx_i.
\end{align*}
\]
This linear functional, being represented by a single row of partial derivatives, can be written as
an inner product with a uniquely determined column vector.
Convention (Gradient). For a scalar-valued function
\(f : \mathbb{R}^n \to \mathbb{R}\), the gradient \(\nabla f\) is the
column vector in \(\mathbb{R}^n\) defined so that the differential takes the inner-product
form, or equivalently the componentwise form
\[
\begin{align*}
df &= (\nabla f)^\top\, d\mathbf{x}, \\\\
(\nabla f)_i &= \frac{\partial f}{\partial x_i}.
\end{align*}
\]
In practice, when a computation yields \(df = \mathbf{v}^\top\, d\mathbf{x}\) for some row vector
\(\mathbf{v}\), we read off the gradient as \(\nabla f = \mathbf{v}^\top\).
The same increment-and-linear-part story transfers verbatim:
\[
\begin{align*}
\Delta f &= f(\mathbf{x} + d\mathbf{x}) - f(\mathbf{x}) \\\\
&= (\nabla f)^\top\, d\mathbf{x} + o(\|d\mathbf{x}\|) \quad \text{as } d\mathbf{x} \to \mathbf{0}.
\end{align*}
\]
The three examples that follow all instantiate a single pattern. We expand \(\Delta f\), collect
terms linear in \(d\mathbf{x}\), and read off \(\nabla f\) from the resulting row vector.
Vector Derivatives: Optimization Foundations
In machine learning, we rarely
optimize single scalar variables. We optimize weight vectors. The differential
calculus just developed is the right tool. Rather than writing out \(n\) partial derivatives by
hand, we manipulate \(d(\cdot)\) symbolically and read off the gradient from the row-vector
coefficient of \(d\mathbf{x}\). The three examples below are the squared norm, the quadratic form,
and the \(L_2\) norm. Together they cover essentially every gradient that appears in standard
regularization and second-order optimization. We first state the product rule in the differential
form that will be used throughout.
Theorem: Differential Product Rule
Let \(g\) and \(h\) be differentiable at \(x\), and let \(f(x) = g(x)h(x)\). Then \(f\) is
differentiable at \(x\), and its differential satisfies
\[
df = (dg)\,h + g\,(dh).
\]
In product-compatible settings (vectors, matrices) where \(g\) and \(h\) need not commute, the
order of factors in each term is preserved.
Proof.
Let \(\Delta g := g(x+dx) - g(x)\) and \(\Delta h := h(x+dx) - h(x)\). By differentiability,
\(\Delta g = dg + o(dx)\) and \(\Delta h = dh + o(dx)\). Expanding the increment of \(f\):
\[
\begin{align*}
\Delta f &= g(x+dx)\,h(x+dx) - g(x)\,h(x) \\\\
&= \bigl[g(x) + \Delta g\bigr]\bigl[h(x) + \Delta h\bigr] - g(x)\,h(x) \\\\
&= (\Delta g)\,h(x) + g(x)\,(\Delta h) + (\Delta g)(\Delta h).
\end{align*}
\]
The first two terms contribute the linear part \((dg)\,h(x) + g(x)\,(dh)\), up to \(o(dx)\).
Here we use that \(h(x)\) and \(g(x)\) are fixed (independent of \(dx\)), so multiplication by
either preserves \(o(dx)\). In the vector/matrix setting the same step uses submultiplicativity
of the operator norm. The cross term \((\Delta g)(\Delta h)\) is \(O(dx)\cdot O(dx) = o(dx)\).
Therefore
\[
\Delta f = (dg)\,h + g\,(dh) + o(dx),
\]
which identifies \(df = (dg)\,h + g\,(dh)\) as the linear part of \(\Delta f\).
Example 1: Squared \(L_2\) Norm \(f(\mathbf{x}) = \mathbf{x}^\top\mathbf{x}\), \(\mathbf{x} \in \mathbb{R}^n\)
This function represents the squared distance from the origin, a core component of
Mean Squared Error (MSE). The input is the vector \(\mathbf{x}\), and the output is
the scalar \(\mathbf{x}^\top\mathbf{x}\).
Example 1.
Applying the
differential product rule
to \(f(\mathbf{x}) = \mathbf{x}^\top\mathbf{x}\):
\[
\begin{align*}
d(\mathbf{x}^\top\mathbf{x})
&= (d\mathbf{x}^\top)\,\mathbf{x} + \mathbf{x}^\top\,(d\mathbf{x}) \\\\
&= \mathbf{x}^\top\,d\mathbf{x} + \mathbf{x}^\top\,d\mathbf{x} \\\\
&= 2\,\mathbf{x}^\top\, d\mathbf{x},
\end{align*}
\]
where the first line uses the fact that \(d(\mathbf{x}^\top) = (d\mathbf{x})^\top\), since
transposition is itself a linear operation. The second line uses the scalar identity
\(\mathbf{a}^\top\mathbf{b} = \mathbf{b}^\top\mathbf{a}\) with \(\mathbf{a} = d\mathbf{x}\),
\(\mathbf{b} = \mathbf{x}\).
Reading off the row vector \(2\,\mathbf{x}^\top\) as the transpose of the gradient:
\[
\nabla f = (2\,\mathbf{x}^\top)^\top = 2\mathbf{x}.
\]
Sanity check (entrywise). Writing \(f(\mathbf{x}) = \sum_{i=1}^n x_i^2\) and
differentiating componentwise yields \(\partial f / \partial x_i = 2 x_i\), so
\[
\nabla f = \begin{bmatrix} 2x_1 \\\\ 2x_2 \\\\ \vdots \\\\ 2x_n \end{bmatrix} = 2\mathbf{x},
\]
in agreement with the differential derivation. For the remaining examples the entrywise route would
be progressively more painful. The differential calculation generalizes effortlessly.
Quadratic Forms \(f(\mathbf{x}) = \mathbf{x}^\top A\mathbf{x}\)
Quadratic forms are essential for
modeling local curvature. Specifically, they form the basis of the Hessian matrix
in optimization. Let \(\mathbf{x} \in \mathbb{R}^n\) and \(A \in \mathbb{R}^{n \times n}\).
Example 2.
Expanding the increment and dropping the quadratic term (which is \(o(\|d\mathbf{x}\|)\)):
\[
\begin{align*}
\Delta f &= (\mathbf{x} + d\mathbf{x})^\top A\,(\mathbf{x} + d\mathbf{x}) - \mathbf{x}^\top A\mathbf{x} \\\\
&= d\mathbf{x}^\top A\,\mathbf{x} + \mathbf{x}^\top A\, d\mathbf{x} + d\mathbf{x}^\top A\, d\mathbf{x},
\end{align*}
\]
so the linear part is
\[
df = d\mathbf{x}^\top A\,\mathbf{x} + \mathbf{x}^\top A\, d\mathbf{x}.
\]
The first term is a scalar, hence equal to its own transpose:
\((d\mathbf{x}^\top A\,\mathbf{x})^\top = \mathbf{x}^\top A^\top\, d\mathbf{x}\). Substituting,
\[
\begin{align*}
df &= \mathbf{x}^\top A^\top\, d\mathbf{x} + \mathbf{x}^\top A\, d\mathbf{x} \\\\
&= \mathbf{x}^\top(A + A^\top)\, d\mathbf{x}.
\end{align*}
\]
Reading off the row vector \(\mathbf{x}^\top(A + A^\top)\) as the transpose of the gradient (per
the convention established in Differentials):
\[
\nabla f = (A + A^\top)\,\mathbf{x}.
\]
Symmetry and Optimization Efficiency
In most machine learning contexts, for instance in the second-order Taylor expansion of a loss
function, the matrix \(A\) is the Hessian, which is symmetric: \(A = A^\top\).
In this case the gradient simplifies to
\[
\nabla f = (A + A^\top)\,\mathbf{x} = 2A\,\mathbf{x}.
\]
Symmetry is not a mathematical curiosity. Numerical solvers exploit it to reduce memory and
computation by roughly half (for example, storing only the upper triangle, using Cholesky in
place of general LU, and using the conjugate-gradient method instead of GMRES).
Moreover, Example 1 is recovered as the special case \(A = I\), for which
\(\nabla (\mathbf{x}^\top\mathbf{x}) = 2I\mathbf{x} = 2\mathbf{x}\), matching our earlier
result.
The \(L_2\) Norm \(f(\mathbf{x}) = \|\mathbf{x}\|_2\)
The \(L_2\) norm, or Euclidean distance
is the most common regularization term used to prevent overfitting in machine learning models.
Understanding its derivative is key to understanding how weight decay works.
Example 3.
Let \(r := \|\mathbf{x}\|_2\), so that \(r^2 = \mathbf{x}^\top\mathbf{x}\). Assume
\(\mathbf{x} \neq \mathbf{0}\), so that \(r \gt 0\) and \(r\) is differentiable at
\(\mathbf{x}\) (this follows from the smoothness of \(\sqrt{\,\cdot\,}\) away from \(0\)
composed with the polynomial \(\mathbf{x}^\top\mathbf{x}\)). The chain rule is formalized on the
Jacobian page. Taking the differential of both
sides of \(r^2 = \mathbf{x}^\top\mathbf{x}\), and using \(d(r^2) = 2r\,dr\) on the left (scalar
chain rule) and the result of Example 1 on the right, we obtain
\[
\begin{align*}
d(r^2) &= d(\mathbf{x}^\top\mathbf{x}) \\\\
2r\, dr &= 2\,\mathbf{x}^\top\, d\mathbf{x} \\\\
dr &= \frac{\mathbf{x}^\top}{r}\, d\mathbf{x}
= \frac{\mathbf{x}^\top}{\|\mathbf{x}\|_2}\, d\mathbf{x}.
\end{align*}
\]
By the gradient convention (Differentials), the gradient is therefore
\[
\nabla \|\mathbf{x}\|_2 = \frac{\mathbf{x}}{\|\mathbf{x}\|_2}.
\]
Note that the norm is not differentiable at \(\mathbf{x} = \mathbf{0}\). The formula above holds
only on \(\mathbb{R}^n \setminus \{\mathbf{0}\}\).
Geometric Interpretation: The Unit Vector
The gradient of the \(L_2\) norm is the unit vector pointing in the direction of
\(\mathbf{x}\).
In gradient descent, this means the
regularization force has magnitude independent of the weights' scale. The force always pulls the
weights directly toward the origin with a constant pressure (proportional to the learning rate).
This is the mathematical reason \(L_2\) regularization effectively shrinks weights but rarely
drives them to exactly zero. The contrast is with \(L_1\) regularization, whose subgradient can
point toward zero with full magnitude across a whole axis.