Consumer Groups and Partitioning

Overview

Partitioned logs such as Kafka combine ordering and scale. Order is guaranteed only inside a partition, so the producer picks a key that keeps related events together.

A consumer group is a team of workers that divide the partitions between them. More groups mean more independent readers of the same data.

Choose the key by asking which events must be seen in order. Customer id suits account state, while a random key maximizes balance but gives up ordering. Estimate peak throughput per partition and divide to get the partition count, leaving headroom, since you can add partitions later but doing so changes key placement.

How it works

The producer hashes the key to choose a partition. The group coordinator assigns each partition to one member, and members commit offsets after processing.

key=cust42 -> hash % 4 = partition 2

topic orders: P0 P1 P2 P3
group billing:   c1(P0,P1)  c2(P2,P3)
group analytics: c1(P0..P3)

Rebalancing is the main operational cost. When a member joins or leaves, partitions move and processing pauses briefly. Cooperative rebalancing, static membership and sensible session timeouts reduce disruption. Monitor consumer lag per partition, because an average lag hides a single stuck or hot partition that holds back one slice of your keys.

Pitfalls

• More consumers than partitions leaves extras idle.
• Committing offsets before the work completes, which loses messages on crash.
• Choosing a low-cardinality key that creates hot partitions.
• Increasing partitions later, which reroutes keys and breaks ordering for in-flight data.

Key takeaways

• Order exists per partition, so key choice defines ordering.
• Partition count caps consumer parallelism.
• Commit offsets after processing.
• Plan partition counts up front.