Event Schema Registry

Overview

A schema registry turns event formats into managed contracts. Every message announces which schema wrote it by a compact id.

Compatibility checking at registration time moves breakage from runtime to the build pipeline.

Choosing a compatibility mode depends on who upgrades first. Backward compatibility means new consumers can read old data, which suits consumers that upgrade before producers or that replay history. Forward compatibility means old consumers can read new data, which suits producers that upgrade first. Full compatibility requires both and is the safest default for shared topics.

How it works

Producers serialize with a registered schema and prepend its id. Consumers resolve the id through a cached registry client.

wire format:  [magic byte][schema id: 4 bytes][avro payload]

producer: id = registry.register("orders-value", schema)
consumer: schema = cache.get(id) or registry.fetch(id)
rule:     BACKWARD  (new schema reads old data)

Safe evolution has simple rules. Add optional fields with defaults, never reuse a field name for a different meaning, and use deprecation periods before removal. Treat schema subjects as owned artifacts with reviewers. For JSON events without a binary format, JSON Schema in a registry still gives compatibility checks, though messages are larger.

Pitfalls

• Removing a required field or changing a type, which breaks even under lax modes.
• Adding a field without a default under backward compatibility.
• Letting producers auto-register schemas in production without review.
• Not caching schemas, making the registry a hot path.

Key takeaways

• Enforce compatibility in CI, not in production incidents.
• Add optional fields with defaults.
• Pick the compatibility mode to match who upgrades first.
• Cache schemas and plan for registry outages.