Intro to Deep Neural Networks

Convolutional Neural Networks (CNNs)

In our pages on neural networks and automatic differentiation, we introduced the multilayer perceptron, backpropagation, and the generalization of gradient computation to arbitrary computational graphs. With these foundations in place, we now survey the major architectural innovations that have shaped modern deep learning. The survey runs from convolutional networks for spatial data to the Transformer architecture, the basis of the large language models that followed.

Over several decades, deep neural networks (DNNs) evolved from early image classifiers into large language and multimodal models through a series of foundational innovations. The DNNs surveyed on this page are feedforward networks, in which information flows in a single direction, from input to output, without any loops or recurrence. Without recurrence, the network is a fixed composition of layers, which simplifies both its evaluation and its differentiation.

Around 1990, Convolutional Neural Networks (CNNs) emerged as an architecture for processing grid-like data such as images. A CNN uses convolutional layers that apply learnable filters across local spatial regions. Mathematically, a 2D convolution operation is given by: \[ (f * x)(i, j) = \sum_m \sum_n f(m, n) \cdot x(i - m, j - n) \] where \(f\) is the filter (or kernel), \(x\) is the input image, and the sums run over the support of the filter. This operation captures local spatial patterns while sharing parameters across the image. Deep learning libraries actually compute the cross-correlation \(\sum_m \sum_n f(m, n)\, x(i + m, j + n)\), which differs from the convolution only by a reflection of the filter. Since the filter is learned, the two define the same class of layers.

Deep CNNs trained on GPUs and applied to large labeled datasets transformed computer vision. Learned convolutional features proved to outperform traditional vision pipelines by a wide margin. A subsequent architectural milestone was the introduction of residual connections, which enabled training networks of unprecedented depth and set the stage for modern vision architectures.

ResNet: Residual Connections

ResNet is a feedforward architecture built from residual blocks:

Unrolling this recursion expresses the activations at the final layer \(L\) in terms of any earlier layer \(\ell\): \[ \mathbf{z}_L = \mathbf{z}_{\ell} + \sum_{i = \ell}^{L - 1} \mathcal{F}_i (\mathbf{z}_i; \boldsymbol{\theta}_i), \] where \(\boldsymbol{\theta}_i\) denotes the parameters of the transformation at layer \(i\).

This formulation mitigates the vanishing gradient problem in deep neural networks and so enables the training of very deep models.

We adopt the denominator-layout convention. For a vector \(\mathbf{y}\) depending on a vector \(\mathbf{x}\), the derivative \(\partial \mathbf{y}/\partial \mathbf{x}\) denotes the transpose of the Jacobian of \(\mathbf{y}\) with respect to \(\mathbf{x}\). The gradient of a scalar is then a column vector, and the chain rule reads \(\partial \mathcal{L}/\partial \mathbf{x} = (\partial \mathbf{y}/\partial \mathbf{x})\,(\partial \mathcal{L}/\partial \mathbf{y})\), with the factor nearest the loss on the right. Parameters are flattened into vectors. Since \(\boldsymbol{\theta}_{\ell}\) enters the network only through \(\mathbf{z}_{\ell+1} = \mathcal{F}_\ell(\mathbf{z}_\ell; \boldsymbol{\theta}_\ell) + \mathbf{z}_\ell\), the gradient of the loss \(\mathcal{L}\) with respect to the parameters at layer \(\ell\) is \[ \begin{align*} \frac{\partial \mathcal{L}}{\partial \boldsymbol{\theta}_{\ell}} &= \frac{\partial \mathbf{z}_{\ell+1}}{\partial \boldsymbol{\theta}_{\ell}} \frac{\partial \mathcal{L}}{\partial \mathbf{z}_{\ell+1}} \\\\ &= \frac{\partial \mathbf{z}_{\ell+1}}{\partial \boldsymbol{\theta}_{\ell}} \frac{\partial \mathbf{z}_L}{\partial \mathbf{z}_{\ell+1}} \frac{\partial \mathcal{L}}{\partial \mathbf{z}_{L}} \\\\ &= \frac{\partial \mathbf{z}_{\ell+1}}{\partial \boldsymbol{\theta}_{\ell}} \left(\mathbf{I} + \sum_{i = \ell+1}^{L-1} \frac{\partial \mathcal{F}_i (\mathbf{z}_i ; \boldsymbol{\theta}_i)}{\partial \mathbf{z}_{\ell+1}}\right) \frac{\partial \mathcal{L}}{\partial \mathbf{z}_{L}} \\\\ &= \frac{\partial \mathbf{z}_{\ell+1}}{\partial \boldsymbol{\theta}_{\ell}} \frac{\partial \mathcal{L}}{\partial \mathbf{z}_{L}} + \text{other terms} \end{align*} \]

Here \(\partial \mathcal{F}_i(\mathbf{z}_i; \boldsymbol{\theta}_i)/\partial \mathbf{z}_{\ell+1}\) denotes the derivative of \(\mathcal{F}_i\) as a function of \(\mathbf{z}_{\ell+1}\), through its dependence on \(\mathbf{z}_i\). The second equality applies the chain rule through \(\mathbf{z}_L\), and the third differentiates the unrolled expression for \(\mathbf{z}_L\) with \(\ell + 1\) in place of \(\ell\). The first term carries the loss gradient \(\partial \mathcal{L}/\partial \mathbf{z}_L\) back to layer \(\ell\) without passing through the Jacobians of the later blocks. The other terms capture indirect paths through deeper residual blocks. They may shrink in very deep networks but are generally nonzero. Residual connections do not remove these terms. They ensure that these terms are not the only path for gradient flow.

Layer Normalization

Let \(z_i\) be the \(i\)th element of a tensor. For example, in 2D images, the index \(i\) has four components, indicating batch, height, width, and channel: \[ i = (i_N, i_H, i_W, i_C). \]

For each element \(z_i\), the mean and variance are computed by pooling statistics across other dimensions of the tensor but not across examples in the batch: \[ \begin{align*} &\mu_i = \frac{1}{|\mathcal{S}_i|} \sum_{k \in \mathcal{S}_i} z_k \\\\ &\sigma_i = \sqrt{\frac{1}{|\mathcal{S}_i|}\sum_{k \in \mathcal{S}_i} (z_k - \mu_i)^2 + \epsilon} \end{align*} \] where \(\mathcal{S}_i\) is the set of elements over which statistics are computed for index \(i\), and \(\epsilon\) is a small constant for numerical stability. For layer normalization, \(\mathcal{S}_i = \{k : k_N = i_N\}\) consists of all elements belonging to the same example as \(i\).

The normalized and scaled output is given by: \[ \begin{align*} &\hat{z}_i = \frac{(z_i - \mu_i)}{\sigma_i} \\\\ &\tilde{z}_i = \gamma_c \hat{z}_i + \beta_c \end{align*} \] where \(c\) is the channel index corresponding to \(i\), and \(\gamma_c, \beta_c\) are learnable parameters for scaling and shifting.

Unlike batch normalization, which pools across the batch dimension, layer normalization pools only over feature dimensions for each input example. This makes it especially effective in settings where batch-level statistics are unstable or unavailable, as in autoregressive models, with small batch sizes, or with variable-length sequences.

Attention Mechanisms

Recurrent architectures process sequences step by step, which limits parallelism and makes modeling long-range dependencies challenging. For example, consider a reader trying to understand what each word in a sentence refers to. On reaching the word "it," the reader looks back at previous words to determine its meaning. Attention mechanisms formalize this process by computing a weighted sum of all input feature vectors. The weights let the model focus dynamically on the most relevant parts of the sequence, regardless of distance.

In traditional feedforward neural networks, each layer computes hidden activations as a linear transformation of input activations followed by a nonlinearity: \[ \mathbf{Z} = \varphi (\mathbf{X}\mathbf{W}) \in \mathbb{R}^{m \times v'} \] where \(\mathbf{X} \in \mathbb{R}^{m \times v}\) is a matrix of input (or hidden) feature vectors, and \(\mathbf{W} \in \mathbb{R}^{v \times v'}\) is a fixed weight matrix learned during training.

This formulation uses the same set of weights \(\mathbf{W}\) for every input position. To allow more flexibility, we can instead make the weights depend on the input: \[ \mathbf{Z} = \varphi (\mathbf{X}\mathbf{W}(\mathbf{X})), \] where the transformation matrix \(\mathbf{W}(\mathbf{X})\) is itself computed from the input, so the model adapts its computation to the context. Such input-dependent multiplicative interactions are central to attention mechanisms.

In more general settings, the input-dependent matrix is computed from separate sets of learned representations called queries, keys, and values. This leads to a broader formulation of attention: \[ \mathbf{Z} = \mathbf{W}(\mathbf{Q}, \mathbf{K})\,\mathbf{V} \] where:

• \(\mathbf{Q} \in \mathbb{R}^{m \times d}\) is a set of queries
• \(\mathbf{K} \in \mathbb{R}^{m \times d}\) is a set of keys
• \(\mathbf{V} \in \mathbb{R}^{m \times v}\) is a set of values

The matrix \(\mathbf{W}(\mathbf{Q}, \mathbf{K}) \in \mathbb{R}^{m \times m}\) is a function of the queries and keys. It multiplies \(\mathbf{V}\) from the left, so it mixes the value vectors, the rows of \(\mathbf{V}\), across positions, whereas \(\mathbf{W}(\mathbf{X})\) above mixed the features within each position. The nonlinearity now sits inside \(\mathbf{W}(\mathbf{Q}, \mathbf{K})\), in the softmax introduced below. In practice, \(\mathbf{Q}\), \(\mathbf{K}\), and \(\mathbf{V}\) are often obtained by applying learned linear projections to the input sequence.

In attention, the core idea is to compute each output vector \(\mathbf{z}_j\) as a weighted sum over all value vectors \(\mathbf{v}_i\), where the weights \(\alpha_{ij}\) are determined by a similarity between the query \(\mathbf{q}_j\) and each key \(\mathbf{k}_i\): \[ \mathbf{z}_j = \sum_{i}\alpha_{ij}\mathbf{v}_i \] where \(0 \leq \alpha_{ij} \leq 1\) and \(\sum_i \alpha_{ij} = 1\). In matrix form, \(\mathbf{Z} = \mathbf{W}(\mathbf{Q}, \mathbf{K})\,\mathbf{V}\) with \(\mathbf{W}(\mathbf{Q}, \mathbf{K})_{ji} = \alpha_{ij}\). The model can thus attend selectively to the relevant parts of the input.

A common similarity function is the inner product between query and key vectors. If \(\mathbf{q}, \mathbf{k} \in \mathbb{R}^d\) are independent random vectors whose components are independent with zero mean and unit variance, then their inner product \(\mathbf{q}^\top \mathbf{k} = \sum_{l=1}^{d} q_l k_l\) is a sum of \(d\) uncorrelated terms of variance 1. The inner product therefore has zero mean and variance \(d\). To prevent large variance from pushing the softmax into saturation regions where gradients vanish, we normalize it by \(\sqrt{d}\): \[ a(\mathbf{q}, \mathbf{k}) = \frac{\mathbf{q}^\top \mathbf{k}}{\sqrt{d}}. \]

The quantity \(a(\mathbf{q}, \mathbf{k})\) is the scaled dot-product attention score. Under the assumptions above, the scale factor \(\frac{1}{\sqrt{d}}\) keeps the variance of the score equal to 1 for every dimension \(d\). Applying the softmax function to these scores yields the attention weights \(\alpha_{ij}\).

Stacking \(n\) query vectors into a matrix, we compute attention for all of them at once.

Self-Attention

Self-attention is a mechanism that allows each position \(i\) in a sequence to attend to all positions in the same sequence, including itself. Given a sequence of input tokens \(\mathbf{x}_1, \ldots, \mathbf{x}_n \in \mathbb{R}^d\), a sequence of outputs is given by: \[ \mathbf{y}_i = \operatorname{Attention}(\mathbf{x}_i, (\mathbf{x}_1, \mathbf{x}_1), \ldots, (\mathbf{x}_n, \mathbf{x}_n)) \in \mathbb{R}^d \] where the query is \(\mathbf{x}_i\) and each pair \((\mathbf{x}_j, \mathbf{x}_j)\) supplies a key and a value. Stacking the tokens as the rows of \(\mathbf{X} \in \mathbb{R}^{n \times d}\), the outputs \(\mathbf{y}_i\) are the rows of \(\operatorname{Attention}(\mathbf{X}, \mathbf{X}, \mathbf{X})\) in the notation of the definition above. Transformers first pass the queries, keys, and values through learned projections, as in multi-head attention below.

This mechanism captures contextual relationships between tokens by allowing information to flow from any position to any other in a single layer. Each output \(\mathbf{y}_i\) depends only on the inputs \(\mathbf{x}_1, \ldots, \mathbf{x}_n\) and not on the other outputs, so all outputs can be evaluated in parallel. When a model with a causal mask is trained to predict each token of a known target sequence from the tokens before it, this parallelism extends over the whole sequence, which removes the sequential bottleneck of recurrent networks.

Multi-Head Attention

While a single self-attention mechanism can capture dependencies between tokens, using only one set of projections may limit the model's expressiveness. Multi-head attention (MHA) addresses this by computing multiple self-attention operations in parallel using different learned projections, called "heads."

The query and key projections share dimension \(d_k\) so that their inner product is well-defined. The value projection dimension \(d_v\) may differ. Inside each head, the attention scores are scaled by \(1/\sqrt{d_k}\). With \(h\) heads and \(d_k = d_v = d/h\), the total computational cost is comparable to that of single-head attention with full dimension \(d\).

By using multiple heads, the model can attend to information from different representation subspaces and capture a richer set of relationships in the data. Each head may focus on different positions or interaction patterns. Such specialization of heads is often observed in trained models, and it is the usual explanation for the empirical benefit of multiple heads.

Positional Encoding

Standard self-attention is permutation-equivariant. Reordering the input tokens simply reorders the outputs in the same way, so the mechanism itself carries no information about token order. To address this, positional encoding is added to the input embeddings to inject information about the position of each token. A widely used method is to define fixed, deterministic positional encodings using sine and cosine functions of varying frequencies.

This formulation assigns to each sine-cosine pair \((2j, 2j+1)\) its own frequency \(\omega_j = C^{-2j/d}\). Distinct positions receive distinct encoding vectors. Already the pair \(j = 0\) gives \((\sin i, \cos i)\), which determines \(i\) modulo \(2\pi\), and distinct integers are never congruent modulo \(2\pi\) because \(\pi\) is irrational. Moreover, a shift by \(k\) positions acts linearly on the encodings. By the addition formulas for sine and cosine, each pair rotates by the angle \(\omega_j k\): \[ \begin{bmatrix} p_{i+k,\,2j} \\ p_{i+k,\,2j+1} \end{bmatrix} = \begin{bmatrix} \cos(\omega_j k) & \sin(\omega_j k) \\ -\sin(\omega_j k) & \cos(\omega_j k) \end{bmatrix} \begin{bmatrix} p_{i,\,2j} \\ p_{i,\,2j+1} \end{bmatrix}, \] so \(\mathbf{p}_{i+k} = \mathbf{M}_k \mathbf{p}_i\) for a block-diagonal matrix \(\mathbf{M}_k\) that depends on \(k\) but not on \(i\).

Example. Suppose the model dimension is \(d = 4\) and \(C = 10{,}000\), and we compute the positional encoding for position \(i = 2\). Its components are: \[ \begin{align*} &p_{2,0} = \sin \left( \frac{2}{10{,}000^{0/4}} \right) = \sin(2) \\\\ &p_{2,1} = \cos \left( \frac{2}{10{,}000^{0/4}} \right) = \cos(2) \\\\ &p_{2,2} = \sin \left( \frac{2}{10{,}000^{2/4}} \right) = \sin \left( \frac{2}{100} \right) \\\\ &p_{2,3} = \cos \left( \frac{2}{10{,}000^{2/4}} \right) = \cos \left( \frac{2}{100} \right) \end{align*} \]

The positional encoding vector for position 2 is therefore \[ \mathbf{p}_2 = \left[ \sin(2),\ \cos(2),\ \sin(0.02),\ \cos(0.02) \right]. \]

These encodings are added element-wise to the token embeddings at each position. The model thus incorporates position information without relying on recurrence or convolution. Moreover, the use of periodic functions enables the model to potentially generalize to sequences longer than those seen during training.

Transformers

The Transformer architecture replaces recurrence and convolution with a mechanism based entirely on attention. In both the encoder and decoder, the core components are multi-head self-attention layers, feedforward networks, residual connections, and layer normalization. The decoder adds a cross-attention step, in which its queries attend to the encoder output. In the decoder, a causal mask restricts each position to attend only to itself and earlier positions. The original design applies layer normalization after each residual connection, as in the pseudocode below, whereas many later models apply it to the input of each sublayer. The design enables full parallelization across sequence positions during training, and allows the model to capture long-range dependencies through direct pairwise interactions between tokens. By using learned attention weights that vary depending on the input, the model dynamically determines which parts of the sequence to emphasize at each layer.

Transformer-based models have been applied across domains such as text, vision, and multimodal learning. In particular, decoder-only large language models (LLMs) are built on stacked Transformer decoder blocks trained on massive text corpora. These models have no encoder, so their blocks omit the cross-attention step of the DecoderBlock below. They exploit the scalability and expressiveness of the Transformer to generate coherent, context-aware output across a wide range of tasks.

The pseudocode uses the following components:

• MHA(...): multi-head attention, which is self-attention when Q, K, and V are the same sequence and cross-attention otherwise.
• + (...): the residual connection, which adds the input of a sublayer back to its output.
• LayerNorm(...): layer normalization after the residual connection (post-norm).
• FeedForward(...): a position-wise fully connected feedforward network, applied to each token vector independently.
• Embed(...): the map sending each token to a high-dimensional embedding vector.
• POS(...): the positional encoding, added to the embeddings to inject token order information.
• N: the number of copies of the block.

Symmetry as Architectural Principle