Neural Networks Basics

Knowledge Map Atlas
Multilayer Perceptron (MLP) Activation Functions Learning in Neural Networks Neural Networks Demo Development of Deep Learning

Multilayer Perceptron (MLP)

On the classification page, we saw that logistic regression defines a linear decision boundary and that the kernel trick can extend this to nonlinear settings. That extension, however, requires choosing a fixed feature map. Neural networks take a fundamentally different approach. They learn the feature representation jointly with the classifier by composing many simple, differentiable transformations. The resulting gradients are computed via the chain rule, which we formalize on this page as the backpropagation algorithm.

The key idea of deep neural networks (DNNs) is to compose a vast number of simple functions into a single complex function. We focus on a specific type of DNN known as the multilayer perceptron (MLP), also referred to as a feedforward neural network (FFNN).

Each layer consists of two operations: an affine transformation followed by an activation function. In a hidden layer, the activation is a nonlinear function \( g_\ell : \mathbb{R} \to \mathbb{R}\) applied elementwise. Such a function is differentiable except possibly at finitely many points (the ReLU below has a single kink at \(0\)). An MLP consists of an input layer, one or more hidden layers, and an output layer.

Definition: Multilayer Perceptron

A multilayer perceptron (MLP) with \(L\) layers defines a parameterized function \[ f(\mathbf{x}; \boldsymbol{\theta}) = f_L \circ f_{L-1} \circ \cdots \circ f_1(\mathbf{x}), \] where each layer computes \[ \mathbf{z}^{(\ell)} = g_\ell\!\left(W^{(\ell)} \mathbf{z}^{(\ell-1)} + \mathbf{b}^{(\ell)}\right), \] with \(\mathbf{z}^{(0)} = \mathbf{x}\), weight matrices \(W^{(\ell)} \in \mathbb{R}^{d_\ell \times d_{\ell-1}}\), bias vectors \(\mathbf{b}^{(\ell)} \in \mathbb{R}^{d_\ell}\), and activation functions \(g_\ell\), nonlinear for \(\ell \lt L\). The output layer uses an activation appropriate to the task (sigmoid for binary classification, softmax for multiclass, identity for regression).

We call \(\mathbf{a}^{(\ell)} = W^{(\ell)} \mathbf{z}^{(\ell-1)} + \mathbf{b}^{(\ell)}\) the pre-activations of layer \(\ell\) and \(\mathbf{z}^{(\ell)} = g_\ell(\mathbf{a}^{(\ell)})\) its activations, which for \(\ell \lt L\) are called hidden units. Every \(g_\ell\) acts elementwise except a softmax output activation, which couples the coordinates of \(\mathbf{a}^{(L)}\). For an input \(\mathbf{x} \in \mathbb{R}^D\) with \(D\) features we have \(d_0 = D\). The output of the network is denoted by \(\hat{\mathbf{y}} = h_{\boldsymbol{\theta}}(\mathbf{x}) = \mathbf{z}^{(L)}\), and the parameters (weights and biases) are collected as \[ \boldsymbol{\theta} = \{ \boldsymbol{\theta}_\ell \}_{\ell=1}^L, \quad \boldsymbol{\theta}_\ell = \{ W^{(\ell)}, \mathbf{b}^{(\ell)} \}. \]

Note. Input data is typically stored as an \( N \times D \) design matrix, where each row corresponds to a data point and each column to a feature. This is referred to as structured data or tabular data. In contrast, for unstructured data such as images or text, different architectures are used:

In particular, the first large language models (LLMs) were built upon the Transformer architecture. From the late 2010s, the Transformer's self-attention mechanism largely replaced RNNs in natural language processing because it models global dependencies in parallel. It thereby bypasses the sequential bottleneck of recurrent designs and connects every pair of positions directly, instead of through the long chains of multiplications along which gradients vanish in a recurrent network.

Are RNNs Outdated?

Recurrent computation keeps advantages that attention lacks. State Space Models (SSMs) combine hardware-aware parallel algorithms for training with the efficient inference characteristic of RNNs.

  • Inference Efficiency:
    Standard self-attention costs \(O(T^2)\) time in the sequence length \(T\), and the memory it keeps for past tokens during generation grows linearly with \(T\). In contrast, SSMs process a sequence in linear time \(O(T)\), with constant \(O(1)\) time and memory per step during inference. This makes them well suited to embedded devices and real-time control systems where memory bandwidth is a bottleneck.
  • Infinite Horizons and State Compression:
    By updating a fixed-size latent state, recurrent systems can theoretically process continuous data streams indefinitely without the memory growth of attention over an ever-longer context. The physical context window is effectively unbounded, but the information density is constrained by the fixed state size. The model must therefore selectively forget or compress older information as the sequence progresses.

Activation Functions

Without a nonlinear activation function, a neural network composed of multiple layers would reduce to a single affine transformation. For example, a two-layer linear network computes: \[ \begin{align*} f(\mathbf{x} ; \boldsymbol{\theta}) &= W^{(2)}(W^{(1)} \mathbf{x} + \mathbf{b}^{(1)}) + \mathbf{b}^{(2)} \\\\ &= (W^{(2)}W^{(1)})\mathbf{x} + (W^{(2)} \mathbf{b}^{(1)} + \mathbf{b}^{(2)}) \\\\ &= \tilde{W}\mathbf{x} + \tilde{\mathbf{b}}. \end{align*} \] This composition remains an affine transformation of \( \mathbf{x} \), and therefore is incapable of representing nonlinear decision boundaries. Nonlinear activation functions are necessary to break this linearity.

Historically, a common choice was the sigmoid (logistic) activation function: \[ \sigma(a) = \frac{1}{1+e^{-a}}. \] However, sigmoid functions saturate for large positive or negative inputs: \( \sigma(a) \to 1 \) as \( a \to +\infty \), and \( \sigma(a) \to 0 \) as \( a \to -\infty \). In these regions, the gradient \(\sigma'(a) = \sigma(a)(1-\sigma(a))\) becomes very small (approaching zero). This small derivative is the source of the vanishing gradient problem.

Even away from saturation, \(\sigma'(a) \leq \tfrac{1}{4}\) for every \(a\), since \(s(1-s) \leq \tfrac{1}{4}\) for \(s \in (0,1)\). During backpropagation, each sigmoid layer contributes such a factor, multiplied together with the transposed weight matrix of that layer, so the gradients reaching the early layers can shrink exponentially with depth. Learning therefore becomes slow or unstable in deep networks.

To address this, deep networks often use the Rectified Linear Unit (ReLU): \[ g(a) = \max(0, a) = a \mathbb{I}(a \gt 0), \] where \(\mathbb{I}(a \gt 0)\) equals \(1\) if \(a \gt 0\) and \(0\) otherwise. Because ReLU's derivative is 1 for positive inputs and 0 for negative inputs, an active unit passes gradients backward without the factor of at most \(\tfrac{1}{4}\) that each sigmoid layer contributes. ReLU is nonlinear yet piecewise linear and computationally simple, and it helps maintain gradient flow during training, which is why it is widely used in deep architectures.

Note. Strictly speaking, the ReLU function is not differentiable at \( a = 0 \). Because ReLU is convex, its subgradients at \( 0 \) are exactly the numbers \(s \in [0, 1]\), since the inequality \(\max(0, z) \geq s z\) holds for all \(z \in \mathbb{R}\) precisely when \(0 \leq s \leq 1\). In practice, implementations assign one of these values as the derivative at exactly \( a = 0 \), most commonly \( 0 \) (sometimes \( 1 \)).

Note. While ReLU mitigates the vanishing gradient problem, it can suffer from the "dying ReLU" problem, in which neurons become inactive (always output zero) and stop learning. This occurs when a neuron's pre-activation is negative on every training input. The ReLU derivative is then \(0\) on every example, so the loss gradient never updates the weights and bias feeding into that neuron.

Learning in Neural Networks

Training the network means finding parameters \( \boldsymbol{\theta} \) that minimize the empirical risk, the average loss over the training pairs \((\mathbf{x}_i, y_i)\), \(i = 1, \ldots, N\): \[ J(\boldsymbol{\theta}) = \frac{1}{N} \sum_{i=1}^N \mathcal{L}(y_i, \hat{\mathbf{y}}_i), \] where \(\hat{\mathbf{y}}_i = h_{\boldsymbol{\theta}}(\mathbf{x}_i)\) is the network's prediction.

For binary classification, the network has a single sigmoid output unit, so \(\hat{\mathbf{y}}\) is a scalar \(\hat{y} \in (0, 1)\), the predicted probability of class \(1\). The standard loss is then binary cross-entropy, the cross-entropy between the label distribution \((y, 1 - y)\) and the predicted distribution \((\hat{y}, 1 - \hat{y})\) for a target \(y \in \{0, 1\}\): \[ \begin{align*} \mathcal{L}(y, \hat{y}) &= \mathbb{H}\bigl((y, 1 - y), (\hat{y}, 1 - \hat{y})\bigr) \\\\ &= -y \log \hat{y} - (1 - y) \log(1 - \hat{y}). \end{align*} \] This is the same loss that logistic regression minimizes on the classification page, now with \(\hat{y}\) produced by a whole network.

Optimization is performed by a gradient-based optimization method, which iteratively updates parameters in the direction of the negative gradient: \[ \boldsymbol{\theta} \leftarrow \boldsymbol{\theta} - \alpha \nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta}) \] where \(\alpha\) is the learning rate. In practice, computing the full-batch gradient at every step is prohibitive when \(N\) is large, so training uses mini-batch stochastic gradient descent, and our demo does the same. Each iteration averages the gradient over a small random subset of the data. This average is an unbiased estimator of the full gradient, with lower variance than a single-example gradient.

The gradients are computed efficiently by the backpropagation algorithm, which applies the chain rule starting from the gradient of the loss with respect to the output and working backward. For a single training pair, the algorithm computes the gradients of \(\mathcal{L}\) with respect to all parameters in just two passes through the network:

Algorithm: BACKPROPAGATION Consider an MLP with \(L\) layers and a loss function \(\mathcal{L}\) Input: Data point \((\mathbf{x}, y)\) // Forward Pass  \(\mathbf{z}^{(0)} = \mathbf{x}\);  for \(\ell = 1 : L\) do     \(\mathbf{a}^{(\ell)} = W^{(\ell)} \mathbf{z}^{(\ell-1)} + \mathbf{b}^{(\ell)}\);     \(\mathbf{z}^{(\ell)} = g_\ell(\mathbf{a}^{(\ell)})\);  \(\hat{\mathbf{y}} = \mathbf{z}^{(L)}\);  Compute Loss \(\mathcal{L}(y, \hat{\mathbf{y}})\); // Backward Pass  \(\mathbf{u}^{(L)} = \nabla_{\hat{\mathbf{y}}} \mathcal{L}\); // Gradient of the loss wrt the network output  for \(\ell = L : 1\) do     \(\boldsymbol{\delta}^{(\ell)} = \mathbf{u}^{(\ell)} \odot g_\ell'(\mathbf{a}^{(\ell)})\); // Elementwise multiplication     \(\nabla_{W^{(\ell)}} \mathcal{L} = \boldsymbol{\delta}^{(\ell)} (\mathbf{z}^{(\ell-1)})^\top\);     \(\nabla_{\mathbf{b}^{(\ell)}} \mathcal{L} = \boldsymbol{\delta}^{(\ell)}\);     \(\mathbf{u}^{(\ell-1)} = (W^{(\ell)})^\top \boldsymbol{\delta}^{(\ell)}\); // Gradient wrt the input of the current layer Output: \(\{\nabla_{W^{(\ell)}} \mathcal{L}, \nabla_{\mathbf{b}^{(\ell)}} \mathcal{L}\}_{\ell=1}^L\)

The gradient of \(J\), or of a mini-batch average, is the corresponding average of these per-example gradients. The step \(\boldsymbol{\delta}^{(\ell)} = \mathbf{u}^{(\ell)} \odot g_\ell'(\mathbf{a}^{(\ell)})\) is the chain rule for an elementwise activation, whose Jacobian is the diagonal matrix \(\operatorname{diag}(g_\ell'(\mathbf{a}^{(\ell)}))\). For ReLU, the value of \(g_\ell'\) at \(0\) is fixed by the convention in the note on subgradients.

A softmax output layer is not elementwise, and there \(\boldsymbol{\delta}^{(L)}\) must be computed with the full softmax Jacobian. For the pairings used in practice, sigmoid with binary cross-entropy and softmax with cross-entropy, both collapse to \[ \boldsymbol{\delta}^{(L)} = \hat{\mathbf{y}} - \mathbf{y}, \] where \(\mathbf{y}\) is the one-hot target (the scalar \(y\) in the binary case). This is the same residual, prediction minus target, that weights each example in the logistic regression gradient on the classification page.

With the architecture and training procedure of neural networks established, a natural question arises: how exactly are the gradients \(\nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})\) computed for arbitrary computational graphs? Our treatment of automatic differentiation generalizes backpropagation beyond sequential layers to the full framework of reverse-mode automatic differentiation on directed acyclic graphs.

Neural Networks Demo

This interactive demo showcases how a simple neural network can learn to classify nonlinear patterns. Datasets can be generated, model parameters adjusted, and the training process visualized in real time. On load, the demo verifies its own gradient computations against finite differences and refuses to render if any check fails.

Try Adjusting:

Development of Deep Learning

The modern revolution in deep learning has been driven not only by algorithmic advances, but also by dramatic improvements in hardware. The most consequential of these was the rise of graphics processing units (GPUs). Originally designed to accelerate matrix-vector computations for real-time rendering in video games, GPUs turned out to be ideally suited for the linear algebra operations at the heart of neural networks.

Around 2010, researchers showed that GPUs could speed up deep learning training by orders of magnitude compared to traditional CPUs. This enabled the training of large neural networks on large labeled datasets such as ImageNet. The resulting models led to breakthroughs in computer vision, speech recognition (converting spoken language to text), and broader natural language processing (NLP) tasks including translation, summarization, and question answering. GPUs subsequently became a core component of ML research and development, as well as of scientific computing, complex simulations, and even cryptocurrency mining.

At a still more basic level, GPUs themselves rely on foundational advances in semiconductor technology. Semiconductors are materials whose conductivity can be precisely controlled. This controllability makes them the backbone of all modern electronics, from GPUs and CPUs to memory chips and mobile devices. By using advanced fabrication techniques and nanometer-scale engineering, manufacturers can pack billions of transistors onto a single chip. This density of computation made the era of foundation models, including LLMs, possible.