Health Check Strategy

Health checks are how infrastructure decides whether to send traffic to an instance or restart it. A strategy defines what each check means, who consumes it, and how failures are interpreted. Mixing these roles is the most common source of check-induced outages.

How it works

Use three distinct signals.

• Startup: has the application finished initializing, such as loading caches and running migrations? Until it passes, other checks are suspended.
• Liveness: is the process able to make progress? It should check only the process itself, for instance that the event loop responds. Failure leads to restart.
• Readiness: can this instance serve requests right now? Failure removes it from load balancing but does not restart it.

Dependency health is exposed separately for humans and dashboards. If a database is down, every instance would fail a dependency-based readiness check simultaneously, leaving zero capacity even for requests that could be served degraded.

Implementation steps

1. Add a liveness endpoint that returns immediately without touching external systems.
2. Add a readiness endpoint that reflects local conditions: initialization complete, not draining, connection pool not exhausted, queue below limit.
3. Cache any expensive probe result for a few seconds so checks do not amplify load.
4. Configure consumers with thresholds, for example three consecutive failures to eject and two successes to restore.
5. Implement graceful shutdown: on termination, flip readiness to failing, wait for the load balancer to drain, then stop.
6. Add a deep check endpoint that exercises dependencies, protected and used by monitoring only.
7. Include a synthetic probe from outside that tests a real user path end to end.

Common pitfalls

• Liveness that queries the database, so a database slowdown restarts the entire fleet.
• Timeouts shorter than garbage collection pauses, causing flapping.
• Probes hitting expensive code paths and adding measurable load.
• No panic threshold: when most instances fail readiness, the balancer should keep sending traffic to all rather than none.
• Health endpoints requiring authentication that the prober lacks.

Metrics to watch

• Count of ejections and restarts per service.
• Flap rate, meaning state changes per instance per hour.
• Ratio of healthy to total instances.
• Probe latency and error rate.
• Gap between check status and real user error rate.

Example configuration

containers:
  - name: api
    startupProbe:
      httpGet: { path: /startup, port: 8080 }
      failureThreshold: 30
      periodSeconds: 2
    livenessProbe:
      httpGet: { path: /livez, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet: { path: /readyz, port: 8080 }
      periodSeconds: 5
      timeoutSeconds: 1
      failureThreshold: 3
      successThreshold: 2

Key takeaways

• Liveness answers only whether to restart; keep it local and cheap.
• Readiness reflects ability to serve, without gating on shared dependencies.
• Use thresholds and hysteresis to avoid flapping.
• Drain gracefully by failing readiness before shutdown.
• Back checks with end-to-end synthetic probes.