Schema Evolution Framework

Schemas are contracts between systems that change at different speeds. Evolution is safe only when every change is classified, checked, and rolled out in an order that keeps old and new participants working together.

How it works

Pick a compatibility mode per subject. Backward compatibility means new consumers can read old data, so consumers upgrade first. Forward compatibility means old consumers can read new data, so producers upgrade first. Full compatibility guarantees both and is the safest default for long-lived streams. A registry stores every version and rejects a proposed schema that violates the mode.

Implementation steps

1. Inventory shared schemas and record the owner, consumers, and retention of each.
2. Choose a serialisation format with explicit evolution rules, such as Avro, Protobuf, or JSON Schema with conventions.
3. Set a compatibility mode and enforce it in the build pipeline.
4. Classify changes: adding an optional field is safe; removing a field, renaming, or changing a type is breaking.
5. For breaking changes, run expand and contract: publish both shapes, migrate consumers, observe zero use of the old shape, then retire it.
6. Document semantics, units, and nullability, since tools cannot check meaning.
7. Test consumers against recent real messages, not only the latest schema.

Common pitfalls

• Reusing a numeric field tag with a new meaning.
• Changing the meaning of a field while keeping its name and type.
• Making a field required after data without it already exists.
• Deleting old versions from the registry while messages still exist.
• Letting consumers fail hard on unknown fields instead of ignoring them.

Metrics to watch

• Count of schema versions active per subject.
• Consumer deserialisation error rate.
• Compatibility check failures in CI.
• Age of deprecated fields still in use.
• Share of consumers pinned to old versions.

Example configuration

{
  "subject": "orders.order-placed-value",
  "compatibility": "FULL_TRANSITIVE",
  "rules": {
    "addField": "optional with default",
    "removeField": "deprecate for 2 release cycles, then remove",
    "renameField": "forbidden, use add and deprecate"
  }
}

Practical rollout

Governance works best when it is light. Give schema owners a short checklist for each change: is it additive, are defaults defined, who consumes it, and when will deprecated fields be removed. Publish a changelog next to the registry so consumer teams can subscribe to changes. Run a periodic cleanup review to remove fields with no observed readers, which keeps additive-only discipline from producing permanent clutter. Share the plan early with consumer teams.

Key takeaways

• Default to additive changes and full transitive compatibility.
• Use expand and contract for anything breaking.
• Enforce compatibility automatically in CI.
• Document semantics because tooling only checks structure.