Why isn't a notebook a record of an experiment?

Execution order is not stored

A notebook file stores a list of cells and, for each, an execution counter and the output that was captured. What it does not store is the sequence in which those cells were run relative to each other over the life of the kernel. The counter tells you a cell ran fourteenth at some point, but a cell that ran, was edited, and ran again shows only the last count. A cell that was deleted after contributing to the state leaves no trace at all.

The consequence is concrete rather than aesthetic. The saved file, executed top to bottom in a fresh kernel, may throw, may produce different numbers, or may produce nothing at all — and the outputs sitting in the file are still there, still looking authoritative. This is why "restart and run all" is the only honest test of a notebook, and why so many notebooks fail it.

Hidden state has no representation in the file

The kernel holds every name ever bound, including ones whose defining cell no longer exists. You import a helper, later delete the import cell, and the name stays live for the rest of the session. You define a threshold, edit the cell to a different value, and a downstream cell you have not rerun still holds a model fitted against the old one. You mutate a dataframe in place in a cell you then rerun, and the second run operates on already-mutated data.

The pathological version of this happens when someone tunes a hyperparameter by editing a cell and re-running only the fit and the evaluation. The reported metric belongs to the last configuration, the cell shows whatever value was typed last, and the intermediate results — the actual experiment — were never recorded anywhere. The notebook now claims a number it cannot reconstruct.

Cell [3]  threshold = 0.5      <- edited to 0.7 later, then edited back
Cell [7]  model = fit(X, threshold)
Cell [8]  print(score(model))  -> 0.912   <- which threshold produced this?

The file offers no way to answer the question in that comment. Nothing in it is wrong; it simply is not a record.

The inputs are usually untracked too

Even a notebook that survives restart-and-run-all is often not reproducible, because of what it reads. A path to a CSV in the author's home directory pins nothing, since the file can be overwritten. A query against a live warehouse table returns different rows next week. A model or encoder loaded from a local pickle carries no provenance. And the environment is whatever was installed in that kernel, which is rarely written down anywhere the notebook can see.

Combine this with the version-control problem — the JSON format means output blobs and execution counters churn on every save, so diffs are unreadable and merges are painful — and the reviewability of the artefact is close to zero. A reviewer cannot see what changed, so nobody reviews it.

What to do instead, without banning notebooks

Notebooks are excellent at what they are for: looking at data, plotting a distribution, checking a hunch in ninety seconds. Prohibiting them costs real velocity and gets quietly ignored. The workable position is a boundary rather than a ban.

Draw the line at the moment a result becomes something you would defend. Up to that point, explore freely. Past it, the run has to be promoted: the logic moves into a module or script that takes its configuration as parameters, is committed, and emits a tracked run recording the data snapshot, commit, resolved config, seeds and environment. The notebook then becomes a thin caller of that module, which has a pleasant side effect — the notebook gets shorter and the logic becomes testable.

Three habits make this cheap in practice. Instrument the tracking call inside the shared training function, so a notebook that calls it gets logged whether or not the author thought about it. Keep the heavy transformation code in importable modules from the start, so promoting a result is a matter of committing and running rather than rewriting. And put a restart-and-run-all check on anything that is being shared, even informally, because the failure it catches is the one that later becomes an argument about what the number was.

Where this goes wrong in review

The mistake in an interview answer is to reach for the tooling question — which notebook-diffing tool, which conversion utility — when the interviewer is probing whether you understand the state model. The specific failure is that a notebook's outputs and its source are two independent things that happen to sit in one file, and nothing enforces that they correspond. Every remedy, from restart-and-run-all to promotion into scripts, exists to re-establish that correspondence. Say that, and the tool choice becomes an implementation detail.

A notebook is a transcript of a conversation with a kernel, and the kernel is gone. Anything you intend to defend six months from now has to be logged by something that was watching at the time.