RabbitMQ Guide

What it is and when to use it

RabbitMQ is a mature message broker implementing AMQP 0-9-1, with plugins for MQTT, STOMP and streams. Producers publish to exchanges, which route messages to queues according to bindings, and consumers receive them from queues.

Reach for RabbitMQ when you need background job distribution, flexible routing, request-reply patterns, priorities, delayed retries or per-message acknowledgement. If you need very high-volume event replay by many independent readers, a log-based system like Kafka fits better, although RabbitMQ Streams now cover part of that space.

Quick start

docker run -d --name rabbit -p 5672:5672 -p 15672:15672 \
  rabbitmq:3.13-management

# management UI at http://localhost:15672 (guest / guest, localhost only)
docker exec rabbit rabbitmqctl list_queues name messages consumers

A minimal publisher and consumer with the Python pika library:

import pika

conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.queue_declare(queue="emails", durable=True)

ch.basic_publish(
    exchange="",
    routing_key="emails",
    body=b"welcome user 42",
    properties=pika.BasicProperties(delivery_mode=2),  # persistent
)

def handle(channel, method, props, body):
    print("sending", body)
    channel.basic_ack(delivery_tag=method.delivery_tag)

ch.basic_qos(prefetch_count=10)
ch.basic_consume(queue="emails", on_message_callback=handle)
ch.start_consuming()

Core concepts

Exchanges decide routing. A direct exchange matches the routing key exactly, a topic exchange matches patterns such as orders.*.created, a fanout exchange copies to every bound queue, and a headers exchange matches message headers.

Queues store messages. Classic queues are fine for transient work, but for replicated durability use quorum queues, which are Raft-based and the recommended default for data you cannot lose.

Acknowledgements. A message stays unacked until the consumer confirms it. If the consumer dies, the broker redelivers. Publishers use publisher confirms to learn the broker has taken responsibility for a message.

Prefetch. The prefetch_count limits unacked messages per consumer, which balances load across workers.

Dead-letter queues and retries

Poison messages should not loop forever. Configure a dead-letter exchange so rejected or expired messages are parked for inspection:

rabbitmqadmin declare exchange name=dlx type=direct
rabbitmqadmin declare queue name=emails.dead durable=true
rabbitmqadmin declare binding source=dlx destination=emails.dead routing_key=emails

rabbitmqadmin declare queue name=emails durable=true \
  arguments='{"x-queue-type":"quorum","x-dead-letter-exchange":"dlx","x-dead-letter-routing-key":"emails","x-delivery-limit":5}'

With a quorum queue, x-delivery-limit caps redelivery attempts before the message is dead-lettered.

Pitfalls

• Unlimited prefetch. The default can send the whole backlog to one consumer. Always set a prefetch value.
• Auto-ack mode. Messages are lost if the consumer crashes mid-processing. Use manual acks.
• Non-durable queues or transient messages vanish on broker restart.
• Unbounded queues. A slow consumer lets memory and disk grow until the broker raises a flow-control alarm and blocks publishers. Set max-length or a TTL, and alert on queue depth.
• Connection churn. Open one long-lived connection per process and use channels, not a new connection per message.
• Nack with requeue loops. Requeueing a failing message immediately creates a hot loop. Use a delivery limit or dead-letter path.
• Default guest account only works from localhost. Create real users and permissions with rabbitmqctl add_user and set_permissions.

How it fits an architecture

RabbitMQ typically sits between a web tier and workers: the API publishes a job and returns quickly, while a pool of workers drains the queue. A topic exchange lets several services subscribe to the events they care about without the publisher knowing them. For resilience, run a three-node cluster with quorum queues spread across availability zones, put a load balancer or client-side node list in front, and scrape the built-in Prometheus plugin for queue depth, unacked counts and publish rates.

Key takeaways

• Producers publish to exchanges, never directly to queues in the AMQP model.
• Use quorum queues, durable messages, publisher confirms and manual acks for safety.
• Always set prefetch and a delivery limit with a dead-letter exchange.
• Choose the exchange type that matches your routing needs: direct, topic or fanout.
• Watch queue depth and consumer count, and keep connections long-lived.