mediumConceptDesignMidSeniorStaff
What has to be captured for a training result to be reproducible?
4 min readUpdated 2026-08-15
Answer
Short answer
Five things: the data version, the code commit, the resolved configuration, the pinned environment and the seeds. Even with all five, GPU kernels are often nondeterministic, so what you can promise is a statistically equivalent rerun and a recoverable artefact, not a bitwise identical one.
Was this answer helpful?
Likely follow-ups
- Your training data lives in a mutable warehouse table. How do you version it without copying terabytes?
- Which sources of randomness exist in a typical training run, and which of them does a single seed call miss?
- If a rerun gives an AUC 0.004 lower, how do you decide whether that is noise or a real difference?
- What would you log so that a failed reproduction attempt tells you which input diverged?
Related questions
- Why isn't a notebook a record of an experiment?mediumAlso on experiment-tracking and reproducibility4 min
- Six months after a decision, a regulator asks how your model arrived at it. What must you have logged at the time?hardAlso on experiment-tracking and reproducibility5 min
- When would you choose deterministic lockstep over an authoritative server, and what does each model cost you?hardAlso on determinism5 min
- Walk me through what rollback netcode does on a single frame, and what it costs to run it.hardAlso on determinism5 min