Stackbook Logo
coordinationestablished · low operational burden

Fencing Tokens

Also known as: fencing, epoch-numbers, generation-tokens

Intent

Prevent a stale lock holder or old leader from corrupting shared resources by attaching monotonically increasing tokens to its requests.

Problem

A client holding a lock can pause for garbage collection or network delay, lose its lease, and then resume believing it still holds the lock. Meanwhile another client has acquired it, so both write to the resource. The lock service alone cannot stop a delayed write.

Forces

  • •Clients cannot reliably know their own lease expired
  • •Pauses and delays are unbounded
  • •The resource itself is the last line of defense
  • •Tokens must increase strictly across grants

Solution

Overview

A fencing token is a number that increases every time a lock or leadership is granted. The protected resource uses it to refuse work from stale holders. Martin Kleppmann popularized the pattern in his analysis of distributed locks; ZooKeeper zxids and etcd revisions can serve as tokens.

How it works

Client 1: acquires lock, token=33, then pauses (GC)
Lease expires
Client 2: acquires lock, token=34, writes to storage (max=34)
Client 1: wakes, writes with token=33
Storage: 33 < 34, reject

The storage layer keeps a record of the highest token seen, atomically with the write:

if request.token < stored.maxToken: reject
else: stored.maxToken = request.token; apply write

Leader epochs in Raft and Kafka controller epochs use the same idea: messages from lower terms are ignored.

Pitfalls

  • The check must be atomic with the write, for example a conditional update, or a race remains.
  • Token generation must survive failover. A lock backed by a single non-replicated Redis can reissue lower numbers.
  • Resources without versioning, such as third-party APIs, cannot enforce fencing; use idempotency keys or accept the risk.
  • Persist the max token durably, otherwise a restart forgets it.
  • Do not rely on lease timeouts and clock checks on the client as a substitute.

Key takeaways

  • Leases limit staleness; fencing tokens make it harmless.
  • The resource, not the client, enforces exclusion.
  • Use a strictly monotonic source backed by consensus.
  • Log rejected stale tokens; a rising count signals pauses or clock trouble in lock holders worth investigating.

✓ When to Use

  • •Distributed locks protecting storage or external side effects
  • •Leader election where old leaders may linger
  • •Any lease-based exclusion where correctness matters

✗ When Not to Use

  • •Efficiency-only locks where occasional duplication is harmless
  • •Resources that cannot check or store a token
  • •Single-process mutual exclusion

Pros

  • +Safety even under arbitrary pauses
  • +Simple monotonic comparison
  • +Works with leases, consensus or database sequences

Cons

  • −Resource must participate by checking tokens
  • −Needs a reliable monotonic source
  • −Does not provide mutual exclusion by itself

Cost Profile

Infrastructure

Small token store or counter

Operational

Low: token source is a lock service feature

Cognitive

Medium: end-to-end thinking

Failure Modes

  • Token source not monotonic after failover, such as a Redis lock with lost state

  • Resource accepts writes without checking tokens

  • Token reset on resource restart accepts old holders

Real-World Examples

Alternatives

    Related Patterns

    • distributed-lock
    • leader-election

    Competency Domains

    distribution communicationreliability ops