Exactly-Once Illusion

Overview

Exactly-once delivery over an unreliable network is impossible: it is a variant of the Two Generals problem. What systems offer is exactly-once effect, built from at-least-once delivery plus idempotent handling. The illusion is that each message was delivered once; the reality is that duplicates arrived and were harmlessly ignored.

Think of it as two separate guarantees: the transport promises nothing is lost, and the receiver promises repeats change nothing. Together they produce the observable behavior of exactly one effect per message.

How it works

The sender retries until it sees an acknowledgment. The receiver tracks which messages it has already applied.

producer --msg(id=42)--> consumer
   (ack lost)
producer --msg(id=42)--> consumer   retry

consumer:
  BEGIN
    if exists(processed, 42) then skip
    else apply(effect); insert(processed, 42)
  COMMIT
  send ack

The effect and the processed marker must commit atomically. For HTTP APIs, clients send an idempotency key and the server returns the stored result on repeats. Kafka's idempotent producer and transactions apply the same principle inside the Kafka ecosystem, but external side effects still need their own protection.

Pitfalls

• Marking the message processed before the effect, or vice versa, outside a transaction.
• Expiring dedupe keys sooner than the longest possible retry or redelivery delay.
• Calling a non-idempotent third party inside the handler with no key of its own.
• Treating broker settings as end-to-end guarantees; they cover only the broker boundary.
• Ordering assumptions: duplicates can arrive after newer messages.

Key takeaways

• Exactly-once delivery is impossible; exactly-once effect is achievable.
• Use at-least-once delivery with idempotent consumers.
• Commit the dedupe marker atomically with the effect.
• Size the dedupe window to the longest retry horizon.