Problem
In a distributed system a crashed node does not announce its death, and a slow network looks identical to a dead peer. Peers need a timely, tunable way to decide a node is gone so work can be reassigned.
resilienceestablished · medium operational burden
Heartbeat
Also known as: keep-alive-signal, liveness-lease
Intent
Detect failed nodes by having them periodically announce they are alive, and treating silence as failure.
Forces
- •Detection speed vs false suspicion
- •Network delays and clock pauses look like failure
- •Heartbeat traffic scales with cluster size
- •Failures must lead to a consistent decision
Solution
Overview
A heartbeat is a periodic proof of life. Because a crashed process sends nothing, absence of the signal is the only available evidence of failure, and the design question is how long to wait before believing it.
How it works
Each worker sends a heartbeat to a coordinator, which records the last-seen time.
worker: every 1s -> send(id, seq)
coordinator: if now - lastSeen(id) > 5s
mark SUSPECT, reassign tasks
A common rule is a timeout of three to five intervals so that one or two lost packets do not trigger action. Gossip-based systems such as Cassandra spread heartbeats peer to peer and use phi accrual detection, which outputs a continuous suspicion value that adapts to network jitter.
Pitfalls
- A paused process may wake up believing it is still the owner. Use leases with expiry and fencing tokens on writes.
- Heartbeats sent from a separate thread can keep succeeding while the real work thread is deadlocked, so tie the beat to progress.
- Aggressive timeouts cause flapping under load, which adds more load.
- An all-to-all topology costs quadratic messages as the cluster grows.
- If the monitor is partitioned, require a quorum before declaring failures.
Key takeaways
- Silence is the failure signal, so set the timeout from observed variability.
- Tie heartbeats to real progress, not just process existence.
- Pair detection with fencing to stay safe after false suspicion.
- Use quorum or gossip to avoid a single point of judgment.
✓ When to Use
- •Cluster membership and leader election
- •Worker pools holding leases on tasks
- •Detecting dead connections in long-lived sessions
✗ When Not to Use
- •Systems where request-level errors already reveal failures quickly
- •Very large clusters without gossip or hierarchy
- •Stateless request handlers behind a load balancer using health checks
Pros
- +Simple and widely implemented
- +Tunable detection time
- +Enables automatic task reassignment
Cons
- −Cannot distinguish slow from dead
- −Generates constant background traffic
- −Needs fencing to avoid split brain
Cost Profile
Infrastructure
Low, small periodic messages
Operational
Medium, tune intervals per environment
Cognitive
Medium, failure detection subtleties
Failure Modes
Long GC pause triggers false failure and duplicate work
Monitor itself fails or is partitioned and declares everyone dead
Clock skew corrupts timeout calculation
Real-World Examples
Alternatives
Related Patterns
- health-check-endpoint
Competency Domains
distribution communicationreliability ops