What is backpropagation doing, and what does a vanishing gradient look like in practice?

Backpropagation is the chain rule with bookkeeping

A network is a composition of functions. Training needs the partial derivative of a scalar loss with respect to every parameter, and the chain rule already tells you what each of those derivatives is. The insight in backpropagation is purely about the order of evaluation: because the loss is a single scalar, working from the output backwards lets you compute one gradient vector per layer and reuse it for every layer beneath, so the whole gradient costs roughly the same as one forward pass rather than one pass per parameter.

The mechanism is that the forward pass caches each layer's inputs and activations, and the backward pass consumes them. At each layer you receive the derivative of the loss with respect to that layer's output, multiply by the layer's local derivative to get the derivative with respect to its input, and multiply by the cached input to get the derivative with respect to its weights. The first quantity is passed further back; the second is what the optimiser uses. That caching is why training memory scales with depth and batch size, and why gradient checkpointing trades recomputation for memory.

Say explicitly that backpropagation computes gradients and nothing else — the update is the optimiser's job, and conflating the two makes the rest of the conversation imprecise.

Why depth makes the gradient shrink

The gradient reaching layer one is a product of terms, one for each layer above it: the local derivative of each activation and each weight matrix along the path. A product of many numbers whose magnitudes are consistently below one falls off geometrically in the number of factors. Nothing is broken; the arithmetic is doing what multiplication does.

Saturating activations are the classic source of small factors. The logistic sigmoid squashes its input into zero to one, and its derivative is largest at the centre and approaches zero as the input grows in either direction. Tanh has the same shape with a maximum derivative of one at the origin. Once a unit is saturated its derivative is near zero, so it contributes almost nothing to any gradient passing through it — the unit is not learning and it blocks learning behind it. Poor initialisation causes this directly, because weights that are too large drive pre-activations into the flat region on the very first batch.

ReLU addresses the problem rather than solving it. Its derivative is exactly one for positive inputs, so it does not shrink the gradient along active paths, but it is exactly zero for negative inputs, so a unit whose pre-activation is negative for every example passes nothing back at all and is effectively dead. Leaky variants keep a small negative slope for that reason.

Residual connections attack the multiplicative structure itself. When a block computes its input plus a transformation of its input, the derivative of the block is the identity plus the derivative of the transformation, so the gradient has an additive path to the early layers that is never multiplied down. That structural change, rather than any choice of activation, is what made very deep networks trainable.

The symptoms in a real training run

Described in the abstract this sounds like something you would notice immediately. In practice it looks like a network that trains, just badly, which is why it is worth knowing the specific signs.

The training loss falls for a short while and then flattens well above where you expect, and it flattens rather than oscillating. Raising the learning rate does not move it much, which distinguishes it from a rate that is simply too small. The last one or two layers keep improving, so the metrics creep along and the model does not look frozen, but the early layers are barely changing: log the norm of each layer's gradient and you see it fall by orders of magnitude from output to input. A network behaving like this is effectively a shallow model with a large random feature extractor bolted underneath, so adding depth makes it worse rather than better — a useful confirmation.

With saturating activations there is a further tell, which is that the activations in a layer collect at the extremes of the range rather than spread across it. In recurrent networks the same mechanism appears as a model that cannot use context beyond a short window, because the gradient is multiplied once per timestep and the number of factors is now the sequence length.

# Log the gradient norm per layer after loss.backward(). A monotone decay from
# output to input by several orders of magnitude is the diagnostic.
for name, p in model.named_parameters():
    if p.grad is not None:
        print(name, p.grad.norm().item())

Do not diagnose it from the loss curve alone

A plateau has several causes and they take opposite fixes. A rate too low plateaus, and so does a rate too high, because the updates bounce around a minimum instead of descending into it. A model short of capacity plateaus, and so do dead ReLUs. All of these look similar on the one plot most people look at, and guessing between them by swapping optimisers is how a week disappears.

The distinguishing evidence is per-layer, not global. If the gradient decays smoothly with depth, the problem is structural and so are the fixes: better initialisation, non-saturating activations, normalisation layers, residual connections. If gradients are healthy throughout and the loss still will not move, stop changing activations, because the problem is the optimiser or the data.