Backpropagation is the chain rule with bookkeeping
A network is a composition of functions. Training needs the partial derivative of
a scalar loss with respect to every parameter, and the chain rule already tells
you what each of those derivatives is. The insight in backpropagation is purely
about the order of evaluation: because the loss is a single scalar, working from
the output backwards lets you compute one gradient vector per layer and reuse it
for every layer beneath, so the whole gradient costs roughly the same as one
forward pass rather than one pass per parameter.
The mechanism is that the forward pass caches each layer's inputs and
activations, and the backward pass consumes them. At each layer you receive the
derivative of the loss with respect to that layer's output, multiply by the
layer's local derivative to get the derivative with respect to its input, and
multiply by the cached input to get the derivative with respect to its weights.
The first quantity is passed further back; the second is what the optimiser uses.
That caching is why training memory scales with depth and batch size, and why
gradient checkpointing trades recomputation for memory.
Say explicitly that backpropagation computes gradients and nothing else — the
update is the optimiser's job, and conflating the two makes the rest of the
conversation imprecise.
Why depth makes the gradient shrink
The gradient reaching layer one is a product of terms, one for each layer above
it: the local derivative of each activation and each weight matrix along the
path. A product of many numbers whose magnitudes are consistently below one falls
off geometrically in the number of factors. Nothing is broken; the arithmetic is
doing what multiplication does.
Saturating activations are the classic source of small factors. The logistic
sigmoid squashes its input into zero to one, and its derivative is largest at the
centre and approaches zero as the input grows in either direction. Tanh has the
same shape with a maximum derivative of one at the origin. Once a unit is
saturated its derivative is near zero, so it contributes almost nothing to any
gradient passing through it — the unit is not learning and it blocks learning
behind it. Poor initialisation causes this directly, because weights that are too
large drive pre-activations into the flat region on the very first batch.
ReLU addresses the problem rather than solving it. Its derivative is exactly one
for positive inputs, so it does not shrink the gradient along active paths, but it
is exactly zero for negative inputs, so a unit whose pre-activation is negative
for every example passes nothing back at all and is effectively dead. Leaky
variants keep a small negative slope for that reason.
Residual connections attack the multiplicative structure itself. When a block
computes its input plus a transformation of its input, the derivative of the block
is the identity plus the derivative of the transformation, so the gradient has an
additive path to the early layers that is never multiplied down. That structural
change, rather than any choice of activation, is what made very deep networks
trainable.
The symptoms in a real training run
Described in the abstract this sounds like something you would notice
immediately. In practice it looks like a network that trains, just badly, which
is why it is worth knowing the specific signs.
The training loss falls for a short while and then flattens well above where you
expect, and it flattens rather than oscillating. Raising the learning rate does
not move it much, which distinguishes it from a rate that is simply too small.
The last one or two layers keep improving, so the metrics creep along and the
model does not look frozen, but the early layers are barely changing: log the norm
of each layer's gradient and you see it fall by orders of magnitude from output to
input. A network behaving like this is effectively a shallow model with a large
random feature extractor bolted underneath, so adding depth makes it worse rather
than better — a useful confirmation.
With saturating activations there is a further tell, which is that the activations
in a layer collect at the extremes of the range rather than spread across it. In
recurrent networks the same mechanism appears as a model that cannot use context
beyond a short window, because the gradient is multiplied once per timestep and
the number of factors is now the sequence length.
for name, p in model.named_parameters():
if p.grad is not None:
print(name, p.grad.norm().item())
Do not diagnose it from the loss curve alone
A plateau has several causes and they take opposite fixes. A rate too low
plateaus, and so does a rate too high, because the updates bounce around a minimum
instead of descending into it. A model short of capacity plateaus, and so do dead
ReLUs. All of these look similar on the one plot most people look at, and guessing
between them by swapping optimisers is how a week disappears.
The distinguishing evidence is per-layer, not global. If the gradient decays
smoothly with depth, the problem is structural and so are the fixes: better
initialisation, non-saturating activations, normalisation layers, residual
connections. If gradients are healthy throughout and the loss still will not move,
stop changing activations, because the problem is the optimiser or the data.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).