Two replicas of the same deployment are running different code, and both were started from the same image tag. How did that happen and what do you change?

Establish what is running before theorising about why

The first move is evidence, and the evidence is not the tag. Ask each replica which image content it is actually running, which for a Kubernetes pod means the resolved image identifier reported in the container status rather than the image reference written in the spec. The spec says what you asked for; the status says what was obtained. When the two replicas report different digests behind the same reference, the diagnosis is finished and everything after that is about mechanism and prevention.

Do the same on the registry side and list the digests that reference has pointed at, along with when each was pushed. Between those two facts you can usually reconstruct the timeline exactly: a push replaced the pointer, some nodes resolved it afterwards and some did not.

A tag is a mutable pointer, and that is the whole cause

An image is identified by the digest of its manifest, which is a content hash and therefore immutable: the same digest always names the same layers and the same config. A tag is a separate, mutable entry in the registry that maps a human-friendly name to one of those digests, and nothing stops a later push from repointing it. app:1.4.2 is not a version, it is a label that currently happens to point at a version.

So the moment your deployment references a tag, the identity of what runs is decided at pull time rather than at deploy time. Two pulls separated by a push get different content from an identical reference, and no part of the system is misbehaving when that happens. This is also why the same problem produces a second, more confusing symptom: because the pod spec did not change when the tag was re-pushed, the orchestrator sees no reason to do anything, so the rollout you assumed happened never happened at all. Replicas that were recreated for unrelated reasons picked up the new content; the rest kept the old.

Pull policy explains which node got which

The remaining half of the mechanism is caching. Each node keeps images it has already pulled, so whether a given node fetches fresh content depends on the pull policy. Pulling always means the node checks the registry for the reference every time a container starts, which gets you the current content at the cost of a registry round trip on every start and a hard dependency on registry availability during recovery. Pulling only if absent means a node with a cached copy under that reference will use it indefinitely, which is fast and registry-independent and precisely how two nodes end up disagreeing.

Kubernetes makes the trap easier to fall into by defaulting on the shape of the reference: if you omit the policy and the tag is latest, the policy becomes always; if you omit it with any other tag, it becomes if-not-present. That is a reasonable default and it is also why a team that carefully moved off latest to versioned tags can quietly acquire stale-cache behaviour they did not have before.

The important thing to say next is that setting the policy to always is not a fix. It narrows the window rather than closing it, because two containers starting either side of a push still get different content, and it makes every container start depend on the registry, which is a bad thing to add to your recovery path. Correctness has to come from the reference, not from the cache policy.

Deploy by digest and let the tag be documentation

The fix is to make the deployed reference immutable. Have the build pipeline push the image, capture the digest the registry returns, and write that digest into the manifest that gets applied. The tag still exists for humans and for browsing, but nothing in production resolves it.

# The tag is a label that can be repointed by anyone with push access.
image: registry.example.com/app:1.4.2
# The digest names exact content, so every replica runs the same bytes.
image: registry.example.com/app@sha256:<digest emitted by the build>

Three good things follow. A change to the deployed content is now necessarily a change to the manifest, so it is reviewable, diffable and attributable, and it triggers a rollout because the spec genuinely differs. Rollback becomes meaningful, because the previous digest is a specific artefact rather than whatever the tag pointed at last week. And the cache stops being a correctness risk, since a digest can be served from cache forever without being wrong, which means you can prefer if-not-present and keep the registry off your critical recovery path.

Rolling back to a tag is not a rollback

This is the part that separates a strong answer, because it is where the same defect becomes an incident rather than a curiosity. Under tags, your rollback target is a pointer, and pointers move. You redeploy app:1.4.1 expecting last month's known-good build and get whatever 1.4.1 resolves to now, which may be a rebuild of the same source against a base image that has since changed, or in the worst case a tag that was overwritten by mistake. You have not restored a known state; you have made a fresh request whose answer you cannot predict.

Worse, the failure is silent. The deployment succeeds, the pods become ready, and the only evidence that you did not get the artefact you tested is a digest nobody looked at. Tag-based rollback is therefore a rollback you cannot verify, and an unverifiable rollback is not one you should be relying on during an incident. The related habit is retention: keep the digests you might roll back to, and know what your registry's cleanup policy would delete, because a garbage-collected rollback target is no target at all.

Making it stick

A convention that depends on discipline decays, so put the guarantee somewhere mechanical. Have continuous integration resolve tag to digest at build time, and treat that resolution as the only sanctioned source of an image reference for production. Enable immutable tags in the registry if it supports them, so a push cannot silently repoint an existing tag. Then use admission control to reject any workload whose image reference is a tag rather than a digest, which turns the rule from a review comment into a property of the cluster.

The cost is real and worth stating: digests are unreadable, so your manifests get harder for humans to scan, and picking up a patched base image now requires a build and a manifest change rather than happening invisibly on the next pull. That invisible upgrade was never a feature, though. It was the same mechanism that let two replicas disagree, arriving on a day you happened to like the outcome.

One further detail is worth having ready, because interviewers use it as a depth check. For a multi-architecture image the tag usually points at an index that lists per-platform manifests, and pinning the index digest still resolves to the right platform on each node. Pinning a single platform's manifest digest instead is how a deployment ends up working on one node pool and failing to pull on another.

A tag names a pointer and a digest names content, so anything you want to be able to reproduce or roll back to must be referenced by digest. Pull policy only decides how long a stale copy survives; it cannot make a mutable reference into a stable one.