Stackbook Logo
← Back to System Architectures

Designing a Distributed Message Queue

Design a Kafka-style durable message queue with partitioned logs, replication, consumer groups and delivery guarantees for millions of messages per second.

4 min read

A message queue decouples producers from consumers, smooths bursts and enables replay. This design uses a partitioned, replicated commit log, the model behind Kafka and Pulsar.

Requirements

Functional

  • Producers publish messages to named topics.
  • Consumers read in order within a partition and track progress.
  • Consumer groups share a topic's load; multiple groups read independently.
  • Configurable retention (time or size) with replay from any offset.

Non-functional

  • 1 million messages per second ingest, average 1 KB.
  • End-to-end p99 latency under 20 ms for normal acknowledgements.
  • Durability: acknowledged messages survive the loss of any single broker.
  • Ordering guaranteed per partition key.
  • 99.95 percent availability.

Capacity estimation

1 million messages per second at 1 KB is 1 GB/s ingest. With replication factor 3, the cluster writes 3 GB/s. Seven-day retention: 1 GB/s times 86,400 seconds times 7 is about 600 TB, or 1.8 PB replicated. At 20 TB of disk per broker, that needs about 90 brokers. With consumers reading two times the ingest, outbound is about 2 GB/s, which a 10 Gbps NIC per broker handles easily.

High-level design

 Producers --> [ Broker 1 ] [ Broker 2 ] [ Broker 3 ] ...
                 topic T:
                  P0 leader   P0 follower  P0 follower
                  P1 follower P1 leader    P1 follower
                       |
        Metadata quorum (controller, Raft)
                       |
 Consumer group G: C1 reads P0, C2 reads P1 (offsets committed)

Each topic is split into partitions. Each partition is an append-only log with one leader and several followers.

Data model

A partition is a sequence of segment files (for example 1 GB each) named by base offset, plus a sparse offset index and a time index. A message holds offset, timestamp, optional key, value, headers and a CRC. Consumer offsets live in an internal compacted topic keyed by group, topic and partition.

Key deep-dives

Sequential I/O. Appending to log files and reading sequentially let the OS page cache and disk readahead do the work. Zero-copy transfer (sendfile) moves bytes from page cache to the socket without user-space copies, which is why a single broker can sustain hundreds of MB/s.

Partitioning and ordering. Producers choose the partition by hashing the key, so all events for one entity stay ordered. More partitions raise parallelism but also increase file handles, leader election time and end-to-end latency. A partition is consumed by at most one consumer in a group, so partition count caps consumer parallelism.

Replication. Followers fetch from the leader. The in-sync replica set (ISR) contains followers within a lag bound. With acks=all and a minimum ISR of 2, a write commits once every ISR member has it; losing the leader promotes an ISR member with no committed data lost.

Delivery semantics. At-most-once: commit offset before processing. At-least-once: commit after processing, so crashes cause redelivery; consumers must be idempotent. Effectively-once: idempotent producers (producer id plus sequence number deduplicate retries) and transactions that atomically write output and offsets.

Consumer group rebalancing. A coordinator assigns partitions and detects dead members through heartbeats. Rebalances pause consumption, so use cooperative incremental rebalancing and static membership for rolling deploys.

Backpressure and retention. Pull-based consumers control their own pace, so slow consumers do not hurt producers. Retention deletes whole old segments; compacted topics keep only the latest record per key for changelog use cases.

Failure modes and trade-offs

  • Broker crash: the controller elects new leaders from the ISR in seconds; producers retry with idempotence to avoid duplicates.
  • Unclean leader election: enabling it favors availability but can lose committed messages; disable for critical data.
  • Poison messages: one bad record can block a partition. Route to a dead-letter topic after N retries.
  • Consumer lag: monitor lag per group; scale consumers up to partition count, beyond which add partitions (which breaks key-to-partition mapping for existing keys).
  • Throughput versus latency: batching and compression (linger of 5 ms, LZ4) increase throughput several-fold at small latency cost.
  • Queue versus log: per-message acknowledgement and priority queues (RabbitMQ style) are easier for task dispatch; a log is better for streaming and replay.

Key takeaways

  • A replicated append-only partitioned log gives high throughput through sequential I/O and horizontal scaling.
  • Ordering is per partition; choose the key carefully and accept that partition count limits consumer parallelism.
  • ISR replication with acks=all and a minimum ISR trades a little latency for no-loss failover.
  • Exactly-once is built from idempotent producers plus transactions, but consumers should still be idempotent.
  • Retention and offset-based replay turn the queue into a durable buffer, not just a pipe.