Kappa Architecture

Overview

Kappa architecture, proposed by Jay Kreps, argues that a replayable log makes the batch layer of lambda redundant. Batch is simply a stream processed from an early offset at high speed.

How it works

Producers append to a partitioned log. One or more stream jobs maintain materialized views. Logic changes are handled by running two job versions side by side.

sources --> log (retained) --> job v1 --> table_v1 --> serving
                          \--> job v2 (from offset 0) --> table_v2
                                         switch serving to table_v2 when caught up

The new job reads historical offsets as fast as the cluster allows, builds a complete table, and only then does the serving layer repoint. The old job and table are deleted after validation. Tiered storage, as offered by Kafka and Pulsar, makes keeping months or years of events affordable by moving old segments to object storage.

Determinism is essential: the job must produce the same output from the same input, which means using event time rather than wall-clock time, and externalizing randomness.

Pitfalls

• Underestimating replay time: reading a year of events may take days and compete with live consumers; give replay its own resources or quotas.
• Evolving schemas: old events must remain readable. Use a schema registry with compatibility rules.
• Side effects during replay: jobs that call external APIs or send notifications will repeat them. Keep sinks idempotent or switch effects off.
• Compaction versus history: a compacted topic keeps only the latest value per key, so it cannot replay full history.

Key takeaways

• One streaming codebase replaces two, if the log retains enough history.
• Reprocessing means running a new version in parallel and swapping outputs.
• Determinism and idempotent sinks make replay safe.
• Large analytical scans may still belong in a warehouse fed by the same log.