Exactly-Once Processing

Overview

Exactly-once is better described as effectively-once. The message may arrive many times, but its effect is applied once.

The standard recipe is at-least-once delivery plus idempotent handling, with the dedup record and the state change committed atomically.

Think in terms of boundaries. Inside a single database transaction, exactly-once effect is achievable. Across a broker and a database, you need either transactional features in both or an inbox table. Across third-party APIs, you depend on their support for idempotency keys, and otherwise you must reconcile afterward.

How it works

On receipt, the handler opens a transaction, inserts the message id into a processed table, applies the change, and commits. A unique-key violation means a duplicate and the handler just acknowledges.

BEGIN;
  INSERT INTO processed(msg_id) VALUES (:id);  -- fails if seen
  UPDATE account SET balance = balance - :amt WHERE id = :acct;
COMMIT;
ack(message)

Some streaming platforms offer transactional writes that make consuming input, updating state and producing output atomic within the platform. That covers pipelines that stay inside the platform, but the guarantee ends at the edge. When a pipeline writes to an external store, make that write an idempotent upsert keyed by a deterministic id derived from the input.

Pitfalls

• Checking for the id and then acting in separate steps, which races.
• Generating a new id on each producer retry, defeating dedup.
• Assuming broker transactions cover calls to external HTTP APIs.
• Keeping dedup records for less time than the maximum redelivery delay.

Key takeaways

• Exactly-once delivery does not exist; exactly-once effect does.
• Dedup and state change must share a transaction.
• Use stable ids and idempotency keys.
• Size dedup retention to maximum redelivery delay.