When does distributed training earn its complexity?

Scale up before you scale out

The honest first answer, and the one a good interviewer is waiting for, is that a bigger single machine is usually cheaper than a cluster. Not cheaper in instance price — cheaper in total cost, because the alternative carries engineering time, a new class of failure, and a communication tax on every step.

The physical reason is bandwidth. Accelerators inside one host communicate over a dedicated high-bandwidth interconnect; accelerators in different hosts communicate over a network that is typically an order of magnitude or more slower, and with higher latency. Since data-parallel training synchronises gradients on every single step, the gap between those two regimes shows up in every step of the run. Eight accelerators in one box and eight accelerators in eight boxes are not the same system.

The operational reason is that a single machine has one failure mode and a cluster has many. A worker that dies, a worker that is merely slow, a network partition mid-all-reduce, a rank that never joins the process group, a hang with no error message that holds every accelerator in the job. Each needs detection and a response. That work is real and it is not part of the model.

So the ordering is: get more out of one accelerator first — mixed precision, a more efficient data loader, gradient accumulation for effective batch size, checkpointing activations to trade compute for memory — then move to the largest single multi-accelerator host you can get, and only then across hosts.

The two things you can split, and what each fixes

Data parallelism replicates the whole model on every worker and splits the batch. Each worker computes gradients on its shard, the gradients are averaged across all workers, and every replica applies the same update. It relieves a time constraint: more samples processed per step, so fewer steps of wall-clock time for the same number of epochs. It does nothing for memory, because every worker still holds the entire model, its gradients and its optimiser state.

Model parallelism splits the model itself across workers. In its tensor form, individual layers are partitioned so a single matrix multiplication is computed cooperatively. In its pipeline form, consecutive groups of layers live on different workers and micro-batches flow through the stages. It relieves a memory constraint: it is what you do when the model plus its optimiser state does not fit on one accelerator. It costs more than data parallelism because the workers exchange activations within the forward and backward pass rather than gradients at the end of it, and pipeline forms have inherent idle time at the start and end of each batch while the pipeline fills and drains.

There is a middle option worth knowing: sharding the optimiser state, gradients and parameters across data-parallel workers rather than replicating them, gathering each shard only when needed. It keeps the programming model of data parallelism while relieving much of the memory pressure, at the price of extra communication. Naming that as the thing to try before hand-partitioning a model is a good signal.

Constraint you have	What to reach for
Fits on one accelerator, run is too slow	Bigger machine, then data parallelism
Fits, but only just, and batch size is tiny	Sharded optimiser state, or gradient accumulation
Does not fit at all	Tensor or pipeline parallelism, usually combined with data parallelism
Dataset too large to read fast enough	Neither. Fix the input pipeline.

That last row is the most common misdiagnosis. A job whose accelerators sit at low utilisation is starved by its data loader, and adding workers multiplies the starvation across more expensive hardware.

Communication overhead scales with the model, not the data

The intuition to fix is that distributing a bigger dataset costs more communication. It does not. In data-parallel training the synchronised quantity is the gradient, which is the size of the parameters, and it is exchanged once per step regardless of how many samples were in the batch. So communication per step is a function of parameter count, and the useful ratio is compute-per-step against bytes-exchanged-per-step.

That gives you a clear prediction about where data parallelism works well. A large model with a large batch does plenty of computation per step against a fixed gradient exchange, so the communication hides behind the compute — especially since gradients for early layers can be exchanged while later layers are still computing their backward pass. A small model with a small batch does very little computation per step against the same exchange, and the workers spend their time waiting. Scaling efficiency therefore falls as you add workers with the batch per worker held constant, and the point at which it becomes unacceptable is something you measure on your own model and network rather than predict from a rule.

Two second-order effects belong in a senior answer. Synchronous training runs at the speed of the slowest worker on every step, so a single straggler ten per cent slower slows the entire job by roughly ten per cent — the cluster has no ability to average over it. And more workers at a fixed per-worker batch means a larger global batch, which changes the optimisation problem: fewer updates per epoch, and a learning rate that was tuned for the smaller batch is now wrong. Scaling the learning rate with the batch and warming it up is standard practice for exactly this reason, but past some batch size the returns fade and you are buying throughput while losing sample efficiency. A team that reports "we doubled the workers and it converged worse" has usually met this and diagnosed it as a bug.

The failure mode of reaching for it early

The specific error is treating distribution as a scaling technique when it is a constraint-relief technique. Asked "how would you train this faster", the weak answer adds workers; the strong one asks what the bottleneck is. If accelerator utilisation is at forty per cent, distribution multiplies an idle resource. If the model is small and the batch is small, distribution adds communication to a step that had almost no compute to hide it behind. If the run takes four hours and is not on anyone's critical path, distribution buys nothing anyone will notice while adding a system that can hang at 03:00.

Profile the single-accelerator step first and know where the time goes — data loading, forward, backward, optimiser, host-device transfer. Then measure scaling efficiency at two workers and at four before committing to sixteen. A team that can state its own scaling curve is in a completely different position from one quoting a framework's marketing.

Distributed training relieves a specific constraint: not enough memory for the model, or not enough time for the schedule. If you cannot name which of those you are hitting, the cheaper answer is a bigger machine and a faster data loader.