reliability-opsestablished · medium operational burden

Reconciliation Job

Also known as: reconciler, repair-job, audit-reconciliation

Intent

Periodically compare two sources of truth, detect drift, and repair or flag differences so that eventual consistency actually converges.

Problem

Events get lost, handlers fail halfway, and external systems change independently. Over time, copies of data diverge silently and nobody notices until a customer or an audit does.

Forces

•Real-time propagation is never perfectly reliable
•External systems are outside your transaction boundary
•Drift must be found before it is costly
•Automated repair can itself cause damage

Solution

Overview

A reconciliation job is the safety net under asynchronous designs. It does not trust that events arrived; it compares what each system holds, finds differences, and fixes or reports them. Payment platforms run these daily against bank and processor reports, and the same idea underlies anti-entropy repair in distributed databases.

How it works

Pick a stable comparison window, read both sides, and diff by key.

for each window (e.g. yesterday, closed):
  A = read(source_of_truth, window)
  B = read(replica_or_provider, window)
  missing  = keys(A) - keys(B)
  extra    = keys(B) - keys(A)
  mismatch = keys in both where checksum differs
  for k in missing: enqueue idempotent repair(k)
  for k in mismatch: send to review queue with both versions
  emit metrics: counts, oldest age

Use closed windows or snapshots so that in-flight writes do not look like drift. Compare checksums per batch first, and drill down only where they differ.

Pitfalls

Comparing live data without a cutoff, producing noise that teaches people to ignore alerts.
Letting the job repair in the wrong direction. Define the authoritative side explicitly.
Non-idempotent repairs; a rerun after partial failure doubles the fix.
Silent failure. Alert on the job not running, not only on mismatches.
Treating recurring mismatches as normal instead of finding the upstream defect.

Key takeaways

Reconciliation closes the loop on eventual consistency.
Define the source of truth and compare closed windows.
Auto-repair only safe cases, idempotently; review the rest.
Monitor job health and mismatch trends.

✓ When to Use

•Payments and ledgers versus provider reports
•Derived stores such as search indexes and caches
•Event-driven systems where consumers may miss or fail on events

✗ When Not to Use

•Systems with strong, transactional single-source consistency
•When drift cannot be defined because no authoritative side exists
•As a replacement for fixing the root cause of recurring loss

Pros

+Catches drift that real-time paths miss
+Provides audit evidence
+Closes the loop on eventual consistency

Cons

−Detection is delayed until the next run
−Adds load on both systems
−Repair logic can introduce new errors

Cost Profile

Infrastructure

Medium: compute and read load

Operational

Medium: triage of exceptions

Cognitive

Medium: matching and repair design

Failure Modes

Job compares moving data and reports false mismatches
Auto-repair overwrites the correct side
Failed runs go unnoticed so drift accumulates

Real-World Examples

Alternatives

anti-entropy-merkle-trees

Related Patterns

eventual-consistency
anti-entropy-merkle-trees
idempotent-batch-jobs
outbox-pattern
audit-logging
backfill-strategy

Competency Domains

reliability opsdata state