Backpropagation, optimisers and learning-rate behaviour, normalisation and regularisation, attention and transformers, and when a neural net is not the answer.
3 questions in Machine Learning.
Backpropagation applies the chain rule backwards through a network, reusing each layer's cached activations to get every parameter's gradient in one pass. Gradients vanish when many small factors multiply, and it shows as a stalled loss with early-layer weights barely moving while the last layers still train.
Start with AdamW because it needs less tuning, and consider SGD with momentum where you have the budget to tune it and want its generalisation behaviour. The schedule matters more than the choice: a short warmup, then decay, and read the loss curve to tell a rate that is too high from a model that has stopped learning.
On small and medium tabular data, where gradient-boosted trees usually match it for a fraction of the effort; where a decision must be explained exactly rather than approximately; and where a tight CPU latency or memory budget makes it unservable. The simpler model then wins on every axis that matters.