What has to be captured for a training result to be reproducible?

The five inputs that determine the result

A training run is a function of five things, and if any one of them is unrecorded the result is not reproducible, it is merely remembered.

The first is the data version. This is the one most often waved past, because the training query reads a warehouse table and the table looks like a stable thing. It is not: rows arrive, late-arriving events backfill yesterday's partition, a nullable column gets defaulted, and the same query run a week later returns a different dataset. You need a way to say "this exact snapshot", whether that is an immutable partition path, a table format's time-travel snapshot identifier, or a content hash of the materialised extract.

The second is the code commit, and the important part is not just recording it but recording whether the working tree was clean. A run tagged with a commit hash while the author had uncommitted edits is worse than an untagged run, because it lies with authority. Log the hash, log the dirty flag, and if the tree was dirty log the diff.

The third is the configuration: hyperparameters, feature list, target definition, train and validation split boundaries, early-stopping criteria, class weights. Config resolution is where silent divergence hides. If the effective config is assembled from a YAML file, environment variables, and command-line overrides, then log the resolved config after merging, not the file you started from.

The fourth is the environment. A requirements.txt with unpinned versions is not an environment, it is a wish. What you want is the fully resolved dependency set — every transitive package at an exact version — plus the interpreter version, and for accelerated training the CUDA and driver versions and the accelerator model, because numerics differ across them. A container image digest captures most of this in one field and is the cheapest thing to record.

The fifth is the seeds, plural. A run typically draws randomness from several independent generators: the framework's own, the numerical library underneath it, the language's standard library, the data loader's shuffling and its worker processes, and any augmentation pipeline. Setting one and assuming the rest follow is a common and quiet mistake.

What logging this looks like in practice

run.log_params({
    # An immutable snapshot identifier, not "the customers table"
    "dataset_snapshot": snapshot_id,
    "git_commit": commit_sha,
    "git_dirty": tree_is_dirty,          # a dirty tree invalidates the commit
    "image_digest": container_digest,     # pins interpreter + libs + CUDA together
    "seed": seed,
    "accelerator": device_name,           # numerics differ across hardware
})
run.log_params(resolved_config)          # after merging file, env and CLI

The pattern to notice is that every value here is an identifier of something immutable. If a logged field points at something that can change underneath you, it records a name rather than a state.

Why bitwise reproduction is usually unavailable

This is the part that separates an honest answer from a rehearsed one. Even with all five inputs pinned, GPU training is frequently nondeterministic. Many accelerated kernels — reductions, scatter-adds, some convolution and attention implementations — accumulate in an order that depends on how work is scheduled across thousands of threads, and floating-point addition is not associative. Two runs sum the same numbers in different orders and get results differing in the last bits. Those differences then compound through gradient updates over thousands of steps.

Frameworks expose deterministic modes that swap in order-stable kernels, and they do help, but they come with a real throughput cost and some operations have no deterministic implementation available, in which case the framework raises rather than silently drifting. Multi-worker data loading and distributed all-reduce ordering add further nondeterminism above the kernel level. Atomic non-determinism in mixed-precision accumulation is the same story again.

So state the guarantee in layers rather than overclaiming. You can guarantee artefact recovery: the exact weights that produced a given evaluation are stored and retrievable, so the model that was assessed is the model you can serve. You can guarantee statistical reproducibility: a rerun with the same five inputs lands within a known tolerance, and you know that tolerance because you measured it by rerunning. You can guarantee auditable lineage: for any artefact, the data, code and config that produced it are named. You cannot generally guarantee bitwise equality on accelerated hardware, and a candidate who promises it has either not tried or is not being asked hard enough.

Measuring your own tolerance is the step people skip

If you cannot reproduce bitwise, then "close enough" needs a number, and that number is empirical. Run the same configuration several times varying only the seed, and record the spread of your headline metric. That spread is your noise floor. It tells you two useful things: whether a candidate model's improvement over the incumbent is real or within seed noise, and whether a failed reproduction attempt indicates a genuine divergence in inputs or ordinary variance. Teams that never measure this end up shipping seed lottery winners and calling them progress.

Reproducibility is not one property. Separate what you can guarantee — the artefact, the lineage, a measured tolerance — from what the hardware will not let you promise, and say which is which before the interviewer asks.