Backfill Strategy

Overview

A backfill reruns pipeline logic over past time ranges. It is routine in analytics: logic changes, sources arrive late, outages leave gaps. A planned strategy prevents these reruns from becoming incidents.

How it works

Make the job a pure function of a partition key, usually a date, and have it replace that partition's output. The orchestrator then generates one run per partition.

for d in 2026-01-01 .. 2026-03-31 (max 4 parallel):
    run job --date d --output orders_v2   # overwrite partition d
validate: row counts, sums vs orders_v1 per day
swap: point view orders to orders_v2
trigger: downstream models for the range

Writing to a new versioned table lets you compare before exposing results. Chunking limits memory and allows resume from the last successful partition. Throttle reads from operational sources or use a replica or snapshot export. Pin the code version and input snapshot so the rerun is reproducible.

For streaming systems, the equivalent is replaying the log from an offset into a new output, as in kappa architecture.

Pitfalls

• Non-idempotent writes: appending during backfill duplicates rows. Overwrite partitions or merge on keys.
• Competing with production for the same resources; cap concurrency and run off-peak.
• Reading current reference data when history needs as-of values, which silently changes past facts. Use snapshots or slowly changing dimensions.
• Forgetting downstream dependencies, leaving dashboards built on stale aggregates.
• Triggering notifications or external calls during rerun; disable side effects.

Key takeaways

• Build jobs partition-parameterized and idempotent from the start.
• Backfill into a shadow output, validate, then swap.
• Bound parallelism to protect live workloads.
• Plan downstream refresh and communicate changes.