Stackbook Logo
data-consistencyestablished · medium operational burden

Hinted Handoff

Also known as: sloppy-quorum-hints, write-hints

Intent

Keep writes available during a replica outage by storing them temporarily on another node and replaying them when the replica returns.

Problem

When a replica for a key is down, writes either fail or leave fewer copies than intended. Failing writes hurts availability, while dropping the missed update leaves the replica permanently stale until repaired. Short outages should not require full repair.

Forces

  • •Write availability during transient node failures
  • •Replicas must eventually receive missed writes
  • •Hint storage must be bounded
  • •Hints do not count as full durability for reads

Solution

Overview

Hinted handoff is a write-availability technique from Dynamo, used by Cassandra and Riak. A healthy node temporarily accepts a write meant for a down replica and forwards it later. It smooths over brief failures without immediate repair.

How it works

key k -> replicas A, B, C ; C is down
write k=v: A ok, B ok, D stores (k, v, hint: C)
...C returns...
D detects C alive, replays (k, v) to C, deletes hint

The coordinator counts a hinted write toward its write quorum when using a sloppy quorum, so the client sees success. Hints are stored in a dedicated structure with a target id and timestamp. Replay is throttled, and a maximum hint window, for example three hours in Cassandra by default, bounds storage.

Pitfalls

  • Hints do not restore the strict quorum overlap: a read at quorum can miss a write whose copies are on A and a hinting node outside the read set.
  • If the hint holder crashes before replay, that copy is lost; durability relies on the remaining real replicas.
  • Long outages exceed the window, so a node returning after that must be repaired with anti-entropy or rebuilt.
  • Unbounded hints can fill disks during large failures; cap and monitor them.
  • Replaying many hints at once can saturate the recovered node; throttle delivery.

Key takeaways

  • Hinted handoff trades read consistency for write availability.
  • It handles short outages; anti-entropy handles everything else.
  • Bound hint retention, monitor backlog and throttle replay.

✓ When to Use

  • •Dynamo-style leaderless stores with high write availability goals
  • •Frequent short node outages or restarts
  • •Systems using sloppy quorums

✗ When Not to Use

  • •Strict quorum consistency requirements
  • •Long outages where hints would pile up
  • •Single-leader systems with replication logs

Pros

  • +Writes succeed despite replica failures
  • +Fast catch-up after short outages
  • +Reduces repair work

Cons

  • −Reads may miss data held only as hints
  • −Hint storage grows during outages
  • −Replay can overload a recovering node

Cost Profile

Infrastructure

Disk for hint queues

Operational

Medium: tune windows and replay throttles

Cognitive

Medium: sloppy quorum semantics

Failure Modes

  • Hint node fails before replay, losing the only extra copy

  • Hint expiry window passes and replica stays stale until repair

  • Replay storm overloads the recovering node

Real-World Examples

Alternatives

    Related Patterns

    • quorum-reads-writes
    • replication

    Competency Domains

    data statereliability ops