How do you turn a training script that works on your laptop into a scheduled pipeline?

Cut the script where the intermediate output has value

The wrong instinct is to split by function: one task per Python function, twelve tasks, a diagram that looks industrious. The right criterion is what would you want to skip on a re-run. Every stage boundary costs you a serialisation of the intermediate state and a scheduling hop, so a boundary only earns its place if the thing on its left is expensive to recompute and stable enough to reuse.

For a typical training job that produces four or five stages. Extraction materialises an immutable snapshot of the source data. Validation checks that snapshot. Feature construction turns it into a training matrix. Training fits the model and writes the artefact. Evaluation scores it and produces a report. Registration publishes it if the report passes.

The reason this decomposition is the common one is that the failure modes cluster at those seams. Extraction fails because the warehouse is busy. Validation fails because upstream changed. Training fails because a node died. Each of those has a different correct response, and you cannot express different responses for stages you have not separated.

Idempotency is a property of where you write

A scheduled pipeline will re-run stages. The scheduler retries on a transient failure, an operator clears a task, a backfill replays a fortnight of days. If re-running a stage produces a different result or corrupts the previous one, every one of those routine events becomes an incident.

Idempotency in practice means the output location is a pure function of the inputs, and the write is atomic. A stage that appends to a table is not idempotent — retry it and yesterday's rows appear twice. A stage that writes to output/latest/ is not idempotent across concurrent backfill runs, because two of them race for the same path. A stage that writes to features/dt=2026-07-28/run=<inputs-hash>/ and then publishes a pointer only after the write completes is idempotent, and it also gives you the ability to see that a run's output already exists and skip the work.

# Deterministic destination keyed on everything that affects the output.
out = f"s3://ml/features/dt={logical_date}/v={feature_code_version}"

if exists(out + "/_SUCCESS"):        # a completed prior run
    return out                        # skipping is now safe, not a gamble

write_atomically(build_features(snapshot_ref), out)

The logical_date in that path matters more than it looks. Stages must key off the run's logical execution date, never now(). A stage that computes "the last thirty days" from wall-clock time produces different data when you backfill it in three months, and your backfilled history silently disagrees with the history that was produced live.

Validation gates that fail rather than warn

A data validation step that logs a warning is worse than no validation step, because it manufactures the impression of a control while allowing the bad run to proceed. The point of a gate is that it stops the pipeline.

Place the main gate between extraction and training, and check the things that actually break models. Schema conformance: expected columns present, types unchanged, no silently added nullable column. Volume: row count within a band derived from history, because a partial upstream load presents as a small but valid dataset. Null rates per column against their historical range, since a feature that quietly goes ninety per cent null still trains a model. Distribution checks on key features against the previous accepted snapshot. Range and category constraints from domain knowledge — a negative age, a currency code that is not in the enumeration. And label sanity: the positive rate within a plausible band, since a broken label join usually shows up here first.

Two design decisions decide whether anyone trusts the gate. Thresholds should come from the historical distribution of the check rather than from someone's guess, or the gate will either never fire or fire constantly until it is disabled. And a failing gate must leave the previous model serving, untouched — the correct behaviour on bad data is to skip the retrain and alert, never to train on it and promote something. Stating that explicitly is worth real credit, because the fail-open version of this pipeline is the one that ships a model trained on a half-loaded table.

flowchart LR
  A[Extract snapshot] --> B[Validate]
  B -->|pass| C[Build features]
  B -->|fail| G[Alert<br/>incumbent keeps serving]
  C --> D[Train]
  D --> E[Evaluate vs incumbent]
  E -->|meets gates| F[Register candidate]
  E -->|below| G

The branch worth looking at is that both failure paths land in the same place and neither of them touches production. A pipeline where a failure path leads anywhere near the serving model is the one that causes outages.

Passing artefacts by reference

Stages must not pass data structures to each other, and they must not share a mutable working directory. Each stage writes its output to durable storage at an immutable path and passes the path forward; the orchestrator carries small strings between tasks, not payloads. This is what makes stages independently retryable, independently runnable for debugging, and portable across machines — and it is also what makes the lineage of the final model recoverable, because every intermediate is still sitting there under a path that names its inputs.

The pointer to "the current best" belongs in a registry rather than in a filesystem convention, so that promotion is an explicit recorded act rather than the side effect of a stage overwriting a well-known path.

Retry semantics belong per stage

One global retry policy is wrong in both directions. Extraction should retry with exponential backoff, because its failures are overwhelmingly transient contention. Validation should retry zero times, because a schema violation will not fix itself and a retried failure just delays the alert while looking like flakiness. Training should retry only if it can resume from a checkpoint, otherwise a retry burns another eight hours to fail the same way. Registration should retry, because it is a small idempotent API call.

Two more things belong in the same discussion. Every stage needs a timeout, or a hung task holds a slot indefinitely and the next scheduled run either queues behind it or, worse, runs concurrently with it. And concurrency has to be bounded deliberately: decide whether two runs of this pipeline may overlap, and if they may not, say so in the pipeline definition rather than relying on the schedule being wider than the runtime.

Every stage boundary should be justified by something you would want to re-run independently, and every stage should be safe to re-run. Retries, backfills and cleared tasks are routine, so a pipeline that is only correct on its first attempt is not scheduled, it is merely automated.