Outbox Pattern Implementation Guide

The outbox pattern solves the dual-write problem. A service cannot atomically update its database and publish to a message broker, so it records the intent to publish inside the database itself and lets a separate process deliver it. The database transaction becomes the single source of atomicity, and the broker becomes an eventually consistent delivery channel.

How it works

Each business transaction inserts one or more rows into an outbox table in the same transaction as the domain change. Either both commit or neither does. A relay then reads committed rows in insertion order and publishes them. Because the relay can crash after publishing but before marking the row as sent, the guarantee is at-least-once. Every event therefore carries a stable identifier so that consumers can detect duplicates.

Implementation steps

1. Create an outbox table with an event id, aggregate id, event type, payload, created timestamp, and a published marker.
2. Insert outbox rows from the same transaction and data access layer as the domain change.
3. Choose a relay. A polling publisher is simple: select unpublished rows ordered by id, publish, then mark them. Log-based CDC reads the database write-ahead log and avoids polling load.
4. Key messages by aggregate id so a partitioned broker preserves per-aggregate ordering.
5. Make consumers idempotent by recording processed event ids or using natural idempotency.
6. Delete or archive published rows on a schedule.

Common pitfalls

• Using multiple relay instances without row locking, which causes duplicates and reordering. Use skip-locked selection or a single leader.
• Relying on auto-increment ids for ordering when concurrent transactions commit out of order. Prefer CDC or a commit-ordered sequence.
• Putting large payloads in the outbox. Store references and keep rows small.
• Letting the table grow unbounded, which slows polling queries.
• Changing event schemas without versioning, which breaks consumers that replay.

Metrics to watch

• Outbox lag: age of the oldest unpublished row.
• Unpublished row count and growth rate.
• Relay publish error rate and retry count.
• Consumer duplicate-detection rate.
• Outbox table size and cleanup duration.

Example configuration

CREATE TABLE outbox (
  id            BIGSERIAL PRIMARY KEY,
  event_id      UUID NOT NULL UNIQUE,
  aggregate_id  TEXT NOT NULL,
  event_type    TEXT NOT NULL,
  payload       JSONB NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at  TIMESTAMPTZ
);

-- Relay query: safe with several workers
SELECT * FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;

Key takeaways

• Write the event in the same transaction as the state change, never as a second step.
• Accept at-least-once delivery and make consumers idempotent.
• Monitor outbox lag as a first-class service level indicator.
• Prefer CDC at scale, polling when simplicity matters more.