Kafka and the Log: What the Design Means in Production

The paper in brief

"Kafka: a Distributed Messaging System for Log Processing" by Jay Kreps, Neha Narkhede and Jun Rao was presented at the NetDB workshop in 2011. It describes a system built at LinkedIn to collect and deliver high volumes of activity and operational data. The authors wanted something with the throughput of a log aggregator and the consumption model of a messaging system. Jay Kreps later wrote an influential essay, "The Log: What every software engineer should know about real-time data's unifying abstraction", which explains the broader idea.

Key ideas

A partitioned, append-only log. A topic is split into partitions. Each partition is an ordered sequence of messages that is only appended to. Each message has an offset, its position in the partition. Ordering is guaranteed within a partition and not across partitions.

Consumers pull and track their own position. Consumers fetch from a broker at an offset of their choosing. Because the log is retained for a configured period, rather than deleted on acknowledgment, a consumer can rewind and re-read. Different consumer groups read the same data independently without affecting each other.

Lean on the operating system. The paper describes relying on the file system page cache rather than maintaining a large cache inside the process, using sequential disk access, and using the sendfile system call to move data from file to socket without extra copies. Sequential disk I/O is far faster than random I/O, and the design is built around that fact.

Simple broker, smart consumer. Brokers do not track per-message acknowledgment state. Consumers record how far they have read. In the original design, coordination among consumers in a group used ZooKeeper.

Delivery semantics. The paper describes at-least-once delivery as the typical behavior, with duplicates possible after a consumer failure.

What breaks in practice

The original paper predates much of what people now rely on. It did not describe the replication that later versions added, where each partition has a leader and followers and an in-sync replica set determines when a write is considered committed. Production durability depends on settings such as replication factor, acknowledgment level and minimum in-sync replicas, and defaults or misconfiguration can lose data that operators assumed was safe.

Ordering is only per partition. Teams that need strict ordering for an entity must route its messages to one partition by key, and then a hot key becomes a hot partition. Changing partition counts later changes the key to partition mapping, which breaks ordering assumptions for existing keys.

At-least-once means consumers must be idempotent, or must use the transactional and idempotent producer features added in later versions. Treating a queue as exactly once without that work produces duplicates after rebalances and retries.

Consumer group rebalancing can stall consumption. Slow processing that exceeds liveness timeouts triggers rebalances, and a cluster of slow consumers can churn endlessly.

Retention is a cost and compliance decision. Keeping data for days or weeks is the feature, and also the disk bill.

Modern descendants

Kafka became an Apache project and the center of a large ecosystem of stream processing, connectors and log compaction for keeping the latest value per key. The project has since replaced ZooKeeper with KRaft, its own Raft-based metadata quorum, as covered in our Raft article. The log abstraction also influenced systems such as Apache Pulsar and cloud services including Amazon Kinesis and Google Pub/Sub, which make different design choices around storage and delivery, and event sourcing practices in application architecture.

The deeper idea is that a replicated, ordered log is a general primitive. Databases use it internally for recovery and replication, and Kafka exposes it as a product.

Key takeaways

• Kafka's core is a partitioned append-only log with offsets, retained after consumption.
• Ordering exists within a partition only, so key choice and partition count are design decisions.
• Performance comes from sequential I/O, the page cache and zero-copy transfers.
• Durability and delivery guarantees come from later replication and producer features, not from the 2011 paper alone.
• Consumers should be idempotent, since at-least-once delivery is the baseline.