mediumConceptScenarioMidSeniorStaff
How do you choose an optimiser and a learning-rate schedule?
Choosing an optimizer and learning rate schedule starts with the learning rate, then the optimizer. AdamW is a strong default, SGD with momentum can generalise well when tuned, and warmup plus decay usually matters more than the optimizer name. Use this optimisers answer to show the decision, trade-off, and evidence rather than a memorised definition.
5 min readUpdated 2026-08-15
Answer
Short answer
Tune the learning rate first because it controls every step. Start with AdamW for a practical default, consider SGD with momentum when you can tune longer, and use warmup plus decay while reading the loss curve for divergence or plateau.
Was this answer helpful?
Likely follow-ups
- Why is weight decay in AdamW handled differently from an L2 penalty added to the loss?
- You double the batch size. What do you do to the learning rate, and why?
- What does a loss curve that is spiky but trending down tell you?
- How would you find a starting learning rate for an architecture you have never trained?
Related questions
- What is backpropagation doing, and what does a vanishing gradient look like in practice?mediumAlso on training-diagnostics4 min
- Your test lifts conversion, but something else moved the wrong way. What are guardrail metrics, and when do they stop a launch?hardSame kind of round: concept5 min
- A method with REQUIRES_NEW was meant to save an audit row even when the main transaction fails. When does it work, and when does it deadlock?hardSame kind of round: concept5 min
- What does a forward deployed engineer interview test that a standard software engineer loop does not?mediumSame kind of round: concept5 min