You need to add one field to an event and remove another, and five teams consume it. How does that roll out?

Adding is easy, removing is a project

These two changes are usually presented as one ticket and they have almost nothing in common. Adding a field is safe as long as an old reader encountering it ignores it, and as long as a new reader encountering a message without it has a defined value to fall back on. Removing a field is unsafe by construction, because you cannot know from your own repository whether some consumer's mapping code dereferences it, and a producer deploy that stops populating it takes effect for every consumer at once with no rollback that recovers the messages already published.

So the rollout is asymmetric. The addition ships in one release. The removal ships in three: stop populating the field's meaning while still emitting it, prove nothing reads it, then delete it from the schema. That middle step is the one candidates skip, and it is the one that actually costs calendar time.

Get the deploy order right, because the compatibility mode dictates it

If you use a schema registry, you are choosing a compatibility mode whether you think about it or not, and the mode determines who deploys first. Confluent's default is BACKWARD, which means a new schema can read data written with the previous schema. Under that mode you may add a field only if it has a default, and you may delete a field freely.

Read the deploy consequence off that definition rather than guessing it. BACKWARD protects readers against old data, so it is safe when consumers upgrade before producers. FORWARD means old readers can handle new data, which is what you need when producers go first. Teams deploy the producer first out of habit — it is their own service, and the change originates there — and then discover that the mode they configured assumed the opposite order. Being able to say "we run BACKWARD, so consumers roll first, and the removal step is therefore the safe one and the addition is the one that needs sequencing" is the answer that reads as having done it.

FULL gets you both directions and costs you the ability to make most changes at all, which is why it is rarely the default. Whatever you pick, the registry's compatibility check in CI is the guardrail that matters: the point is not the mode, it is that an incompatible schema fails a build rather than a consumer at 3am.

What the serialisation format lets you do

The mechanics differ enough that naming your format is part of the answer.

In Protobuf, identity is the field number, and the tombstone is the mechanism.

message OrderPlaced {
  reserved 4;                  // was promo_code - the number must never be reused
  reserved "promo_code";       // and neither must the name, so nobody re-adds it

  string order_id    = 1;
  int64  customer_id = 2;
  string currency    = 5;      // new; absent in old messages, so it reads as ""
}

reserved exists because reusing field number 4 for something else would make an old message decode into the new field with the wrong meaning, silently and with no error. That is the single most damaging failure mode in this whole area, and it is why the number is the contract and the name is documentation. Note the other proto3 wrinkle: a plain scalar has no way to distinguish "absent" from its zero value, so currency == "" cannot tell you whether an old producer omitted it or a new producer sent an empty string. Marking the field optional, which proto3 regained in protobuf 3.15, restores explicit presence and is worth doing for any field where absent and empty mean different things.

In Avro, the reader's schema and the writer's schema are resolved against each other at decode time, and a field added without a default cannot be resolved when reading older data — which is exactly the rule the registry is enforcing. With JSON Schema the equivalent discipline is refusing to set additionalProperties: false on an event you intend to evolve, because that turns every future addition into a validation failure at the consumer.

The log remembers, so "everyone has upgraded" is not a state you reach

The habit that separates event-driven experience from request-response experience is remembering that a retained topic is not a channel, it is a database of past messages. A consumer that resets its offset to replay six months of history will encounter every schema version that ever existed. So there is no point at which you can delete the code that handles the old shape, unless you have also compacted or expired the messages containing it — and if the topic is the system of record for a rebuildable projection, you never will.

Two consequences follow. Deserialisation must never throw on an unrecognised field, which is the tolerant-reader principle stated as a hard requirement rather than a nicety: read what you understand, ignore what you do not, and never fail on the presence of something new. And handling of a missing field must be an explicit decision at the consumer, written as a default in the schema or a branch in the code, rather than whatever your language does with a null.

How you prove nobody reads the field

"We asked the five teams and they said no" is not evidence, because the answer comes from memory rather than from the code. There are three things that are.

Consumer-driven contract tests are the strongest: each consumer publishes the shape it depends on, the producer's build verifies its output satisfies every published expectation, and a removal that breaks a consumer fails in the producer's pipeline before it is merged. This inverts the usual dependency and it is the only mechanism that turns a coordination problem into a build failure.

Failing that, search the consumers' repositories for the field name — cheap, imperfect, and much better than asking. And instrument the producer: emit the field with a sentinel or log a counter when it is populated, then look for consumer-side errors. For a removal that genuinely cannot be verified, the honest fallback is a long deprecation window with the field still present but documented as unmaintained, and an announced date.

Where the version boundary should be

Evolving in place is right for the changes above. A new topic — or a version suffix on the subject — is right when the change is not expressible as an evolution at all: the field's type changes, one event splits into two, or the semantics of an existing field change while its shape stays identical. That last case is the dangerous one, because no compatibility checker in existence will catch it. Changing amount from gross to net passes every schema check and silently corrupts every downstream calculation, which is why a semantic change should be shipped as a new field with a new name and the old one deprecated, never as a redefinition.

Run both topics in parallel, let consumers migrate on their own schedule, and retire the old one when its consumer group list is empty. The cost is dual publishing for a while and a period where two topics describe the same reality; the benefit is that no consumer's timeline is coupled to yours, which is the property you split the services up to get.