Idempotent Batch Jobs

Overview

A job is idempotent if running it once or many times with the same inputs leaves the same final state. For batch pipelines this converts failure handling into a single rule: just run it again.

How it works

The two main techniques are partition overwrite and keyed merge, combined with atomic publish.

run(date = D):
  input  = raw.events where event_date = D        # fixed snapshot
  out    = transform(input)                        # pure function
  write out to staging/D/
  atomically replace curated/event_date=D with staging/D

With a table format that supports transactions, use a single statement:

INSERT OVERWRITE TABLE curated.daily_sales PARTITION (sale_date = :d)
SELECT ... FROM raw.sales WHERE sale_date = :d;

For upserts, MERGE on the primary key so replays update instead of insert. A run is identified by its logical date, not the time it started, so a retry at noon for yesterday's data processes yesterday's data. Success markers, such as a manifest file written last, let consumers wait for complete partitions.

Pitfalls

• Using the current timestamp or latest state in logic. Use the logical run date and as-of snapshots.
• Appending without dedup keys; retries create duplicates. Add unique business keys.
• Non-atomic publish: consumers read half a partition. Stage then swap or commit.
• Late-arriving data in a closed partition is ignored on rerun unless the lookback window is defined.
• External side effects inside the job run twice. Isolate them behind idempotency keys.

Key takeaways

• Deterministic input plus overwrite or merge equals safe reruns.
• Publish atomically and mark completion explicitly.
• Parameterize by logical date, never by wall clock.
• Idempotence is the foundation for retries and backfills.