Exactly-Once Delivery Guide

Exactly-once delivery cannot be guaranteed across a network, but exactly-once effects can. The approach is to accept duplicates in transport and make processing idempotent, so that a message delivered twice changes state once.

How it works

Producers attach an idempotency key to each logical operation, reused on retries. The consumer checks whether the key was already processed, and if not, performs the effect and records the key in one atomic step. Broker features such as idempotent producers suppress duplicates caused by producer retries, and transactional consume-transform-produce loops commit offsets and output records together. These guarantees stop at the boundary of the broker, so effects in databases or third-party APIs need their own protection.

Implementation steps

1. Identify operations with non-idempotent effects and list every retry point.
2. Generate keys deterministically from business identity, such as order id plus step, or let the client supply a unique key per intent.
3. Create a processed-keys table with a unique constraint and write to it in the same transaction as the effect.
4. For external calls, pass the idempotency key to the provider, and persist the result keyed by it.
5. Enable producer idempotence and, for stream pipelines, transactional processing with read-committed consumers.
6. Choose a dedup retention window longer than the maximum retry and replay horizon.
7. Test with fault injection that kills consumers after the effect but before the acknowledgement.

Common pitfalls

• Recording the key and applying the effect in separate transactions, which allows loss or duplication.
• Generating a new key on every retry.
• Expiring dedup records before late redeliveries stop.
• Assuming broker exactly-once covers database writes and email sends.
• Ignoring out-of-order duplicates that arrive after newer state.

Metrics to watch

• Duplicate detection rate and its trend.
• Dedup table size and cleanup lag.
• Retry and redelivery counts.
• Transaction abort rate for transactional pipelines.
• Reconciliation mismatches between source and target.

Example configuration

CREATE TABLE processed_messages (
  idempotency_key TEXT PRIMARY KEY,
  processed_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  result          JSONB
);

-- In one transaction: insert key (fails if duplicate), then apply effect.
-- ON CONFLICT DO NOTHING plus a rowcount check means "already handled".

Practical rollout

Apply the effort where duplicates hurt most. Payments, inventory changes, and notifications that trigger external actions deserve full idempotency design, while analytics counters may tolerate approximation. Document the idempotency contract in every public API, including how long keys are honoured and what response a repeated request receives. Return the stored original result for a duplicate rather than an error, so clients can retry safely without special handling.

Key takeaways

• Aim for exactly-once effects through idempotent processing, not exactly-once transport.
• Store the dedup key atomically with the effect.
• Size the dedup window to the longest retry horizon.
• Verify guarantees with fault injection.