Stackbook Logo
messagingestablished · medium operational burden

Competing Consumers

Also known as: work queue, worker pool

Intent

Scale message processing by letting multiple consumers pull from the same queue so each message is handled by exactly one of them.

Problem

A single consumer cannot keep up with bursts of messages, and one slow message blocks everything behind it. Scaling by running more consumers risks double processing unless the queue arbitrates.

Forces

  • •Throughput must scale horizontally
  • •Each message should be processed by one worker
  • •Workers crash mid-processing
  • •Load varies and workers should balance themselves

Solution

Overview

Competing consumers is the classic work-queue pattern. Many identical workers compete for messages on one queue, and the broker guarantees each message goes to a single worker at a time.

It turns a throughput problem into a deployment problem: add workers until the backlog drains.

It pairs naturally with autoscaling. Queue depth shows how much work waits, but the age of the oldest message shows whether you are meeting your latency target, so scale on both. Keep consumers stateless so any instance can take any message and instances can be replaced freely during deploys.

How it works

A worker receives a message, which becomes invisible to others for a lease period. On success it acknowledges; on crash or timeout the lease lapses and the message reappears.

queue --> worker A  (msg 1, leased 30s)
      --> worker B  (msg 2, leased 30s)
      --> worker C  (idle)
ack(msg 1)   -> deleted
A crashes    -> msg 2 reappears after lease

Tune the lease carefully. If a task takes up to two minutes, a thirty second lease causes duplicates, while an hour-long lease delays recovery after a crash. Long tasks should extend the lease with a heartbeat. Limit prefetch so one worker does not hoard messages while others sit idle, and cap retries before routing to a dead-letter queue.

Pitfalls

  • Visibility timeout shorter than worst-case processing, producing duplicates. Extend leases with heartbeats.
  • Non-idempotent handlers. Delivery is at-least-once.
  • Ignoring poison messages that fail forever; route them to a dead-letter queue after a retry cap.
  • Scaling consumers beyond what the downstream database can absorb.

Key takeaways

  • One queue, many identical workers, one winner per message.
  • Leases plus acknowledgements give crash recovery.
  • Scale on backlog age, not just depth.
  • Pair with idempotency and a dead-letter queue.

✓ When to Use

  • •Work items are independent and order is unimportant
  • •Load is spiky and workers should scale with backlog
  • •Tasks are long enough that parallelism pays off

✗ When Not to Use

  • •Messages for the same entity must be processed in order
  • •Handlers are not idempotent and redelivery is unsafe
  • •Work requires affinity to cached local state

Pros

  • +Near-linear horizontal scaling
  • +Natural load balancing
  • +Resilient to individual worker failure

Cons

  • −No ordering guarantees across workers
  • −Redelivery means duplicates
  • −Visibility timeout tuning is fiddly

Cost Profile

Infrastructure

Low - queue plus worker pool

Operational

Medium - autoscaling and backlog alerts

Cognitive

Low - simple model

Failure Modes

  • Visibility timeout shorter than processing time causes duplicate work

  • Poison message crash-loops all workers

  • Autoscaling lag lets backlog age past SLA

Real-World Examples

Alternatives

    Related Patterns

    • dead-letter-queue

    Competency Domains

    distribution communicationreliability ops