Apache Kafka Guide

What it is and when to use it

Apache Kafka is a distributed, append-only commit log. Producers write records to topics, and consumers read them at their own pace. Because records are retained on disk for a configurable period, many independent consumers can read the same stream and even replay it from the past.

Use Kafka when you need high-throughput event streaming, decoupling between many services, event sourcing, change data capture, or durable buffering between systems that run at different speeds. Prefer a simpler queue such as RabbitMQ when you need per-message routing, priorities or low-volume task distribution.

Quick start

Modern Kafka runs in KRaft mode, so no ZooKeeper is needed. Using the official image:

docker run -d --name kafka -p 9092:9092 apache/kafka:3.8.0

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --create \
  --topic orders --partitions 3 --replication-factor 1

docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic orders

docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 --topic orders \
  --from-beginning --group demo

Core concepts

Topic and partition. A topic is split into partitions. Each partition is an ordered, immutable sequence of records identified by offsets. Ordering is guaranteed only within a partition, so choose a record key (for example the order id) so related events land together.

Replication. Each partition has a leader and followers. With a replication factor of 3 and min.insync.replicas=2, the cluster tolerates one broker failure without losing acknowledged writes.

Consumer groups. Consumers sharing a group.id split the partitions between them. Parallelism is capped by the partition count. Each group tracks its own committed offsets, so groups do not interfere with each other.

Retention and compaction. Delete-based retention drops old segments by time or size. Log compaction keeps only the latest record per key, which suits changelog and state topics.

Reliable producer and consumer configuration

# producer
acks=all
enable.idempotence=true
compression.type=lz4
linger.ms=10
batch.size=65536

# consumer
group.id=billing-service
enable.auto.commit=false
auto.offset.reset=earliest
max.poll.records=500

With acks=all and idempotence enabled, retries do not create duplicates inside a partition. On the consumer side, commit offsets only after processing succeeds. That gives at-least-once delivery, so make handlers idempotent. Exactly-once processing across topics is possible with transactions, at the cost of more complexity.

Inspect lag regularly:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group billing-service

Pitfalls

• Too few partitions. You can add partitions later, but doing so changes key-to-partition mapping and breaks ordering for existing keys. Plan for peak consumer parallelism up front.
• Too many partitions. Thousands per broker increase recovery time and memory use.
• Auto-commit with slow processing. Offsets may be committed before work finishes, which loses messages on a crash.
• Rebalance storms. Long processing between polls exceeds max.poll.interval.ms and kicks the consumer out of the group. Tune it or process in smaller batches.
• Huge messages. The default limit is about 1 MB. Store large payloads in object storage and send a reference.
• Replication factor of 1 in production. A single disk failure loses data.
• Hot keys. A skewed key distribution overloads one partition while others sit idle.

How it fits an architecture

Kafka commonly sits at the center of an event-driven platform. Services publish domain events, stream processors such as Kafka Streams or Flink enrich them, and sinks load them into search, a warehouse or a cache. Kafka Connect with Debezium captures database changes into topics without touching application code. Pair it with a schema registry using Avro or Protobuf so producers and consumers can evolve contracts safely, and export broker and consumer lag metrics to your monitoring stack.

Key takeaways

• Kafka is a durable log, not just a queue: consumers replay by resetting offsets.
• Ordering holds per partition, so pick keys deliberately.
• Use acks=all, idempotent producers and manual offset commits for reliability.
• Partition count sets the ceiling on consumer parallelism.
• Monitor consumer lag and under-replicated partitions above all else.
• Keep messages small and use a schema registry for compatibility.