Self-Healing Watchdog

Overview

A watchdog turns operational knowledge into automation: if a component stops making progress, restart it. The approach comes from hardware watchdog timers and is the core of Erlang supervisors and Kubernetes restart policies.

How it works

The monitored component must regularly prove progress, and the supervisor acts when it does not.

loop:
  do work
  watchdog.kick()          # resets countdown

supervisor:
  if no kick for 30s:
      dump threads and heap
      restart component
      backoff = min(backoff * 2, 5m)
      if restarts > 5 in 10m: quarantine and page

Kick the watchdog from the place that proves useful work, such as after processing a message, not from an independent timer thread.

Pitfalls

• A background thread that kicks on a timer reports health even if the worker is deadlocked.
• Restart loops hide root causes and can overload dependencies on startup. Use backoff and a ceiling, then escalate.
• Collect a core dump or thread dump before restarting so the bug can still be diagnosed.
• A fleet-wide shared fault can trigger every instance to restart simultaneously. Limit concurrent restarts.
• Restarting a stateful process requires safe recovery from its journal or snapshot.

Key takeaways

• Tie the watchdog to real progress.
• Capture evidence before restarting.
• Use backoff, limits, and escalation to prevent loops.
• Treat automatic restarts as a metric to investigate, not a fix.
• Alert on restart counts per component, because a rising restart rate is an early warning of a leak or defect even when users notice nothing.