Natural Gradient Descent

The Geometry of Information: Interactive GMM Manifold

The Model: A Two-Component Gaussian Mixture

The entire canvas represents a two-dimensional parameter space \[ \Theta = \{(\mu_1, \mu_2) \in \mathbb{R}^2\}. \] Each point on this plane specifies a one-dimensional Gaussian Mixture Model \[ p(x \mid \theta) = \tfrac{1}{2}\mathcal{N}(x \mid \mu_1, \sigma^2) + \tfrac{1}{2}\mathcal{N}(x \mid \mu_2, \sigma^2), \] where the mixing weights are fixed at \(\tfrac{1}{2}\) and the variance \(\sigma^2\) is shared.

The glowing handle marks the target model \(\theta^* = (\mu_1^*, \mu_2^*)\), whose distribution generates the data and defines the loss below. The hollow dashed marker shows its swap \((\mu_2^*, \mu_1^*)\), which specifies the same distribution with the component labels exchanged. Dragging the handle moves the target through this parameter space. Clicking anywhere else places a probe point \(\theta\), a candidate model, and the panel reports the exact Fisher matrix and gradients computed there.

The Fisher Information Matrix as a Riemannian Metric

Information geometry treats a parametric family of distributions as a statistical manifold, that is, a smooth manifold whose points are probability distributions, and it takes the FIM as the natural Riemannian metric on it, called the Fisher-Rao metric. For the family \(\{p(x \mid \theta) : \theta \in \Theta\}\) this picture needs two qualifications. First, \(\Theta\) covers the family twice, since \(\theta\) and its swap specify the same distribution. On either open half-plane \(\{\mu_1 \lt \mu_2\}\) or \(\{\mu_1 \gt \mu_2\}\) each distribution of the family with distinct component means appears exactly once, by the identifiability of finite Gaussian mixtures up to relabeling, which we take for granted. Second, the FIM is positive definite at every \(\theta\) off the diagonal \(\mu_1 = \mu_2\) and singular on it, as we show below. On each open half-plane the FIM is therefore a Riemannian metric, given that its entries depend smoothly on \(\theta\) (a differentiation-under-the-integral fact we take for granted), while the diagonal is a genuine singularity of the model. (The formal definitions of a smooth manifold and of a Riemannian metric are given later in the manifold series.)

Off the diagonal this \(2 \times 2\) matrix is positive definite, and it encodes how sensitively the distribution \(p(x \mid \theta)\) changes as we perturb \(\theta\) in each direction. The FIM defines an infinitesimal distance on the manifold. For a small displacement \(d\theta\), the squared statistical distance is \[ ds^2 = d\theta^\top F(\theta)\, d\theta. \]

Up to the factor \(\tfrac{1}{2}\), this quadratic form is the second-order Taylor expansion of the KL divergence, taken here in its density form, where the sum over outcomes becomes an integral against \(p_\theta\). The expansion is derived in the natural gradient section of the Fisher information page: \[ D_{\mathbb{KL}}(p_\theta \,\|\, p_{\theta + d\theta}) \approx \tfrac{1}{2}\,d\theta^\top F(\theta)\,d\theta. \]

What the Ellipses Show: Local Quadratic Forms of the Metric

Each ellipse in the tessellation is a confidence ellipse of the Fisher-Rao metric, that is, the level set \[ \{\delta\theta : \delta\theta^\top F(\theta_0)\,\delta\theta = \varepsilon\} \] of the local quadratic form at that grid point. The ellipse axes are determined by the eigendecomposition \[ F = Q \Lambda Q^\top. \] The eigenvectors of \(F\) give the principal directions, and the semi-axis lengths are proportional to \(1/\sqrt{\lambda_i}\), where \(\lambda_1, \lambda_2\) are the eigenvalues. A long axis in some direction means the FIM eigenvalue is small along that direction. The distribution is insensitive to parameter changes there, so a large step in parameter space produces only a small change in KL divergence.

All ellipses are drawn at one global scale (the ellipse-scale slider), so their relative sizes are comparable across the canvas. Where the divergent axis near the singular locus exceeds its grid cell, the ellipse is clipped to the cell and drawn dashed. The ellipse field depends only on the grid point and \(\sigma\), not on the target handle.

The Information Singularity: When \(\mu_1 \to \mu_2\)

The singular locus appears directly on the canvas as the dashed diagonal \(\mu_1 = \mu_2\). Probe near it and observe the ellipses elongating dramatically along the exchange direction \((1, -1)\). This visualizes the information singularity of mixture models. When \(\mu_1 \approx \mu_2\), the model reduces to \[ p(x) \approx \mathcal{N}(x \mid \mu_1, \sigma^2), \] and the exchange direction stops being visible to first order. Along the line \(\theta(t) = (\mu + t, \mu - t)\) through a diagonal point \((\mu, \mu)\), the derivative of \(p(x \mid \theta(t))\) at \(t = 0\) is \(\tfrac{1}{2}\partial_\mu \mathcal{N}(x \mid \mu, \sigma^2) - \tfrac{1}{2}\partial_\mu \mathcal{N}(x \mid \mu, \sigma^2) = 0\), so a small step along \((1, -1)\) from the diagonal changes the distribution only at second order. The label swap itself leaves the distribution unchanged everywhere, and on the diagonal it fixes \(\theta\), so the two copies of the model meet there.

Formally, the scores are \(s_i(x) = \partial \log p(x \mid \theta)/\partial \mu_i = (x - \mu_i)\,\mathcal{N}(x \mid \mu_i, \sigma^2)/\bigl(2\sigma^2 p(x \mid \theta)\bigr)\). On the diagonal both equal \((x - \mu)/(2\sigma^2)\), so \(F = \tfrac{1}{4\sigma^2}\mathbf{1}\mathbf{1}^\top\) with \(\mathbf{1} = (1, 1)^\top\). This matrix is rank-deficient (positive semi-definite but not definite). Its eigenvalue along the exchange direction \((1, -1)\) is \(0\), and along the mean-shift direction \((1, 1)\) it is \(1/(2\sigma^2)\). Off the diagonal, \(F\) is positive definite. Indeed, \(F\) is the covariance of the score vector, so it is singular exactly when \(a s_1 + b s_2 = 0\) for some \((a, b) \neq (0, 0)\) and all \(x\) (both sides are continuous in \(x\)). Multiplying by \(2\sigma^2 p(x \mid \theta)\) and dividing by \(\mathcal{N}(x \mid \mu_2, \sigma^2)\) turns this into \(a(x - \mu_1)e^{cx + d} + b(x - \mu_2) = 0\) with \(c = (\mu_1 - \mu_2)/\sigma^2\). When \(c \neq 0\), that is, off the diagonal, letting \(x \to \pm\infty\) with the sign of \(c\) forces \(a = 0\), and then \(b = 0\). As a probe approaches the diagonal, the smaller eigenvalue tends to \(0\) (numerically, like the square of the gap \(\mu_1 - \mu_2\)) while the larger tends to \(1/(2\sigma^2)\), so the condition number \[ \kappa(F) = \lambda_{\max}/\lambda_{\min} \] diverges, as the \(\kappa(F)\) readout in the panel reports.

This singularity is not a numerical artifact but a genuine geometric property of the model. Gradient-based learning is known to slow down sharply near such singular regions of mixture and multilayer models, where it can spend long stretches on plateaus. The Euclidean gradient does not account for the degenerate geometry there.

Well-Separated Regime: When \(|\mu_1 - \mu_2| \gg \sigma\)

Away from the diagonal, where \(|\mu_1 - \mu_2| \gg \sigma\), the ellipses become nearly circular, because the FIM approaches a scalar multiple of the identity \[ F(\theta) \approx \frac{1}{2\sigma^2} I. \] In this regime, the two Gaussian components have negligible overlap, and each parameter \(\mu_i\) is informed almost exclusively by data generated from its own component. That component carries weight \(\tfrac{1}{2}\), so \(\mu_i\) receives half the Fisher information \(1/\sigma^2\) of a single Gaussian mean. The metric is nearly constant in this regime, and the natural gradient is approximately the ordinary gradient scaled by \(2\sigma^2\), so the two arrows point in almost the same direction.

Euclidean vs. Natural Gradient: The Red and Cyan Arrows

The loss whose descent directions are shown is itself statistical: \(L(\theta) = D_{\mathbb{KL}}(p_{\theta^*} \,\|\, p_\theta)\), the divergence from the target model to the candidate \(\theta\), computed by quadrature. Click anywhere on the canvas to place a probe point. Two arrows appear:

• The red arrow:
  The Euclidean steepest-descent direction \(-\nabla_\theta L\), computed in the standard \(\ell^2\) geometry of parameter space. Among displacements of unit Euclidean length \(\|d\theta\|_2\), it is the direction along which \(L\) decreases fastest to first order.
• The cyan arrow:
  The natural-gradient descent direction \(-\widetilde{\nabla}_\theta L\), where \(\widetilde{\nabla}_\theta L = F(\theta)^{-1}\nabla_\theta L\) is the natural gradient at a probe \(\theta\) off the diagonal, where \(F(\theta)\) is invertible. Among displacements of unit Fisher-Rao length \(\sqrt{d\theta^\top F(\theta)\,d\theta}\), it is the direction along which \(L\) decreases fastest to first order. To see this, write \(\langle \mathbf{u}, \mathbf{v} \rangle_F = \mathbf{u}^\top F(\theta) \mathbf{v}\). Then \(\nabla_\theta L^\top d\theta = \langle F(\theta)^{-1}\nabla_\theta L, d\theta \rangle_F\), and the Cauchy-Schwarz inequality for \(\langle \cdot, \cdot \rangle_F\) bounds this first-order change below by \(-\|F(\theta)^{-1}\nabla_\theta L\|_F\) on the unit sphere of \(\|\cdot\|_F\). Equality holds exactly when \(d\theta\) is a positive multiple of \(-\widetilde{\nabla}_\theta L\) (the equality case of the inequality). Since \(\tfrac{1}{2}\,d\theta^\top F(\theta)\,d\theta\) approximates \(D_{\mathbb{KL}}(p_\theta \| p_{\theta+d\theta})\) to second order, the same direction solves the linearized problem of minimizing \(L(\theta + d\theta)\) under a small KL budget.

Both arrows are normalized to the same display length, so the display compares only their directions. The angle between them, reported in the panel, measures how much the manifold geometry distorts the gradient direction at that point. It always stays below \(90^\circ\), because \(\nabla_\theta L^\top F(\theta)^{-1}\nabla_\theta L \gt 0\) whenever \(\nabla_\theta L \neq \mathbf{0}\), so the natural direction is always a descent direction. In the well-separated regime this angle is near \(0^\circ\). Near the singularity it can exceed \(45^\circ\), which shows that Euclidean descent and natural-gradient descent select substantially different directions there. Whether the natural-gradient direction is operationally preferable depends on the loss landscape and the optimization objective. The information-geometric metric measures distance between distributions, which is the relevant scale for likelihood-based objectives but not the "correct" notion of distance for every learning problem.

On the singular locus the demo refuses to display a natural gradient at all. It treats \(F\) as numerically singular when \(\kappa(F) \gt 10^8\), which, given the quadratic decay of the smaller eigenvalue, happens only within about \(10^{-4}\sigma\) of the diagonal. Inverting \(F\) there would require an arbitrary regularization. The refusal is the honest answer, and it is precisely the geometric content of the singularity.

Flow Mode: Particle Trajectories Along the Natural Gradient Field

Activating Flow releases particles that move along the negative natural gradient field \[ -F(\theta)^{-1}\nabla L(\theta). \] Each particle takes steps of a fixed length along this field, so it follows the steepest descent direction in the Fisher-Rao metric at each point, that is, the direction of fastest first-order decrease in loss per unit Fisher-Rao length. To second order that length is \(\sqrt{2 D_{\mathbb{KL}}}\) between neighboring distributions. The resulting flow is distinct from both Euclidean steepest descent and the manifold's geodesics, but it shares with the latter the property of respecting the underlying geometry.

Observe that the particles do not converge to a single point. The loss has two global minima, the target \(\theta^*\) and its swap \((\mu_2^*, \mu_1^*)\), because exchanging the component labels leaves the mixture distribution unchanged. These two points are the only zeros of \(L\). A KL divergence between densities vanishes only when the densities agree (the density form of Gibbs' inequality, assumed here without proof), and identifiability up to relabeling then leaves only \(\theta^*\) and its swap. The flow makes this label symmetry directly visible. Particles that enter the refusal band around the singular locus stall there because the metric is degenerate, which is also why the probe refuses. The preconditioning by \(F(\theta)^{-1}\) that shapes this flow is the geometric correction that Amari's natural gradient descent, K-FAC, and the closely related Fisher-preconditioned methods are designed to exploit. These methods have been reported to converge faster than plain gradient descent on neural networks and mixture models, although the gain depends on the problem.

The Controls and What They Change Geometrically

The \(\sigma\) slider controls the shared standard deviation. Decreasing \(\sigma\) sharpens both components, increasing the base Fisher information \(\sim 1/\sigma^2\) and making the singularity zone narrower (the components must be closer before they overlap). Increasing \(\sigma\) broadens the overlap region and makes the singularity dominate a larger portion of parameter space.

The grid density slider controls the tessellation resolution. More ellipses give a finer picture of how the metric tensor varies across parameter space, at the cost of more computation per frame. The ellipse scale slider uniformly scales the display size of the ellipses without altering the underlying eigenvalues. This is a visualization aid, not a mathematical parameter.