resilienceestablished · low operational burden

Health Check Endpoint

Also known as: health-endpoint-monitoring, liveness-readiness-probe

Intent

Expose a lightweight endpoint that reports whether an instance is alive and ready so orchestrators and load balancers can route around failures.

Problem

A process can be running yet unable to serve traffic, for example while starting, deadlocked, or cut off from its database. Without a signal, load balancers keep sending requests to broken instances.

Forces

•Probes must be cheap and fast
•Deep checks catch more but risk cascading failures
•Liveness and readiness mean different things
•False positives remove healthy capacity

Solution

Overview

A health check endpoint lets infrastructure ask an instance whether it should receive traffic or be replaced. The value comes from separating the questions rather than from a single status flag.

How it works

Three probes are common in Kubernetes, and the same split works elsewhere.

GET /healthz/live    process responsive?      fail -> restart
GET /healthz/ready   can serve right now?     fail -> remove from LB
GET /healthz/startup finished booting?        fail -> keep waiting

The orchestrator calls each on an interval, for example every five seconds, and acts after a threshold of consecutive failures. Readiness typically checks local state, such as completed warm-up, open connection pools, and loaded config. It reports degraded dependencies in the body for humans but avoids failing on optional ones.

Pitfalls

If every instance checks the same database and fails when it is slow, the whole fleet leaves rotation at once. Prefer local checks or tolerate dependency failure.
Using one endpoint for liveness and readiness causes restarts when the right action was only to pause traffic.
Expensive checks run by many probers become a load source.
Checks that never touch the request path can report healthy while handlers are stuck.
Unauthenticated endpoints should not leak internal topology.

Key takeaways

Separate liveness, readiness, and startup.
Keep checks fast, local, and cheap.
Require consecutive failures before acting.
Never let a shared dependency decide fleet-wide health.
Version and document probe semantics for every team, so a new service does not copy a deep check that was only suitable for a different architecture.

✓ When to Use

•Containers behind Kubernetes or load balancers
•Auto-scaling groups that replace unhealthy instances
•Rolling deployments needing readiness gating

✗ When Not to Use

•Short-lived batch jobs
•Single-instance systems with no router to act on the result
•When checks would be so deep they overload dependencies

Pros

+Automatic removal of broken instances
+Safer rolling deployments
+Standard tooling support

Cons

−Poor checks cause outages themselves
−Adds endpoint to maintain
−Shallow checks miss real faults

Cost Profile

Infrastructure

None, tiny endpoint

Operational

Low, tune thresholds

Cognitive

Low, widely understood

Failure Modes

Readiness check calls a shared database, so a database blip removes the whole fleet
Liveness probe too strict restarts healthy pods under load
Check passes while real request path is broken