Scalar Functions of a Matrix

Derivation of \(\nabla \|X\|_F\).

Let \(f(X) = \|X\|_F = \sqrt{\operatorname{tr}(X^\top X)}\). Assume \(X \neq 0\), so that \(\operatorname{tr}(X^\top X) \gt 0\) and \(f\) is differentiable at \(X\) (as the composition of the smooth map \(\sqrt{\,\cdot\,}\) on \((0,\infty)\) with the polynomial map \(X \mapsto \operatorname{tr}(X^\top X)\)). Applying the chain rule with the outer scalar function \(u \mapsto \sqrt{u}\) (whose scalar derivative is \(1/(2\sqrt{u})\)), we obtain \[ df = \frac{1}{2\sqrt{\operatorname{tr}(X^\top X)}} d\bigl[\operatorname{tr}(X^\top X)\bigr]. \]

To handle the inner expression, we use linearity of the trace. The increment \(\operatorname{tr}(A + \Delta A) - \operatorname{tr}(A) = \operatorname{tr}(\Delta A)\) is exactly linear in \(\Delta A\), so \[ d\bigl(\operatorname{tr}(A)\bigr) = \operatorname{tr}(dA), \] so \(d\) and \(\operatorname{tr}\) commute. Transposition is likewise linear, so \(d(X^\top) = (dX)^\top\). The product rule applied to \(X^\top X\) then gives \[ d(X^\top X) = (dX^\top) X + X^\top (dX) = (dX)^\top X + X^\top (dX). \]

Taking the trace and using \(\operatorname{tr}(B) = \operatorname{tr}(B^\top)\), we combine the two terms: \[ \operatorname{tr}\bigl((dX)^\top X\bigr) = \operatorname{tr}\bigl(((dX)^\top X)^\top\bigr) = \operatorname{tr}(X^\top dX), \] so \(\operatorname{tr}\bigl(d(X^\top X)\bigr) = 2\,\operatorname{tr}(X^\top dX)\). Substituting back into the chain-rule expression, we find \[ \begin{align*} df &= \frac{1}{\sqrt{\operatorname{tr}(X^\top X)}}\, \operatorname{tr}(X^\top dX) \\\\ &= \operatorname{tr}\!\left(\frac{X^\top}{\|X\|_F}\, dX\right). \end{align*} \]

Comparing with \(df = \operatorname{tr}((\nabla f)^\top dX)\) from the gradient convention, we read off \[ \nabla f = \frac{X}{\|X\|_F}. \] As for the vector norm, \(f\) is not differentiable at \(X = 0\), so the formula holds on \(\mathbb{R}^{m \times n} \setminus \{0\}\).

Example.

Let \(f(A) = \mathbf{x}^\top A \mathbf{y}\) where \(A \in \mathbb{R}^{m \times n}\), \(\mathbf{x} \in \mathbb{R}^m\), \(\mathbf{y} \in \mathbb{R}^n\). The differential is \[ df = \mathbf{x}^\top (dA)\, \mathbf{y}, \] a scalar, and a scalar equals its own trace. Applying the cyclic property yields \[ \begin{align*} df &= \operatorname{tr}\bigl(\mathbf{x}^\top (dA)\, \mathbf{y}\bigr) \\\\ &= \operatorname{tr}\bigl(\mathbf{y}\mathbf{x}^\top\, dA\bigr) \\\\ &= \operatorname{tr}\bigl((\mathbf{x}\mathbf{y}^\top)^\top\, dA\bigr). \end{align*} \] Comparing with \(df = \operatorname{tr}((\nabla f)^\top dA)\), we read off \[ \nabla f = \mathbf{x}\mathbf{y}^\top. \]

Proof.

Since \(\det A\) is a polynomial in the entries of \(A\), it is \(C^\infty\) and in particular differentiable everywhere on \(\mathbb{R}^{n \times n}\). Fix indices \(i, j\) and recall the cofactor expansion along the \(i\)-th row: \[ \det A = \sum_{k=1}^n A_{ik}\, C_{ik}, \] where \(C_{ik}\) is the \((i,k)\) cofactor. The decisive structural fact is that \(C_{ik}\) is defined via the \((n-1) \times (n-1)\) minor obtained by deleting row \(i\) and column \(k\), so \(C_{ik}\) does not depend on any entry of the \(i\)-th row, and in particular does not depend on \(A_{ij}\). Differentiating the expansion with respect to \(A_{ij}\) therefore picks out the single term \(A_{ij}\, C_{ij}\): \[ \frac{\partial \det A}{\partial A_{ij}} = C_{ij}. \]

By the gradient convention, the matrix whose \((i,j)\) entry is \(\partial \det A / \partial A_{ij}\) is \(\nabla(\det A)\): \[ \nabla (\det A) = \operatorname{cofactor}(A) = \operatorname{adj}(A)^\top. \] The corresponding differential is \[ d(\det A) = \operatorname{tr}\bigl((\nabla \det A)^\top\, dA\bigr) = \operatorname{tr}\bigl(\operatorname{adj}(A)\, dA\bigr). \]

When \(A\) is invertible, \(\operatorname{adj}(A) = (\det A)\, A^{-1}\), and since \(\det A\) is a scalar it factors out of the trace: \[ d(\det A) = (\det A)\, \operatorname{tr}(A^{-1}\, dA). \]

Corollary.

For a fixed \(A \in \mathbb{R}^{n \times n}\), set \(M(x) := xI - A\). Then \(dM = (dx)\, I\), and for \(x\) such that \(M(x)\) is invertible (that is, \(x\) is not an eigenvalue of \(A\)), Jacobi's formula gives \[ \begin{align*} d\bigl(\det M(x)\bigr) &= (\det M(x))\, \operatorname{tr}\bigl(M(x)^{-1}\, (dx)\, I\bigr) \\\\ &= (\det M(x))\, \operatorname{tr}\bigl(M(x)^{-1}\bigr)\, dx. \end{align*} \] Dividing by \(dx\), we obtain \[ p'(x) = p(x)\, \operatorname{tr}\bigl((xI - A)^{-1}\bigr). \]

                            import numpy as np

                            np.random.seed(0)
                            n = 4

                            # Random matrix A
                            A = np.random.randn(n, n)

                            # Evaluation point x chosen well away from any eigenvalue of A
                            # (for real-random 4x4 matrices, |eigenvalues| are typically below ~3)
                            x = 5.0
                            dx = 1e-8

                            # Finite-difference approximation of p'(x) where p(x) = det(xI - A)
                            I = np.eye(n)
                            p_at_x = np.linalg.det(x * I - A)
                            p_approx = (np.linalg.det((x + dx) * I - A) - p_at_x) / dx

                            # Analytical formula from Jacobi:  p'(x) = p(x) * tr((xI - A)^{-1})
                            p_exact = p_at_x * np.trace(np.linalg.inv(x * I - A))

                            # Relative error
                            rel_err = abs(p_approx - p_exact) / abs(p_exact)

                            print(f"Matrix A:\n{A}\n")
                            print(f"Evaluation point x = {x}")
                            print(f"p(x) = det(xI - A) = {p_at_x:.6e}\n")
                            print(f"p'(x) finite-difference  : {p_approx:.6e}")
                            print(f"p'(x) analytical (Jacobi): {p_exact:.6e}")
                            print(f"\nRelative error: {rel_err:.2e}")