Load Shedding Playbook

Load shedding is the practice of refusing some requests so that the rest succeed. Under overload, doing all work badly is worse than doing the important work well. This playbook covers detection, priority, rejection, and recovery.

How it works

A shedder sits at the entry of a service, ideally before parsing and authentication cost are paid. It computes an overload signal and compares it to thresholds.

• Signals: in-flight request count, queue wait time, event loop lag, CPU saturation, or the adaptive concurrency limit derived from observed latency.
• Priority: each request carries a class, often set by the edge from route and caller, for example critical, standard, and sheddable.
• Action: when overloaded, reject the lowest class with HTTP 503 or gRPC RESOURCE_EXHAUSTED, including a Retry-After hint. As severity increases, move up the classes.

Queue delay is usually a better signal than CPU because it measures the harm directly: requests waiting longer than their useful life.

Implementation steps

1. Define priority classes with product owners and write down which routes belong to which.
2. Run a load test to find the saturation point and the concurrency at which p99 latency breaks the objective.
3. Set a concurrency or queue-delay limit just below that point.
4. Add the check as early middleware; keep the rejection path allocation-free.
5. Propagate a deadline header so downstream tiers discard work whose budget has expired.
6. Mark rejections as non-retryable by default for non-critical clients, or require jittered backoff.
7. Document the manual override, such as a flag that lowers the threshold during an incident.

Common pitfalls

• Using CPU alone, which lags behind queue growth and misses I/O-bound overload.
• Shedding after expensive work has been done, wasting the effort.
• Health checks being shed, causing the load balancer to remove healthy instances and shrink capacity.
• Clients retrying rejections immediately, defeating the shedding.
• Testing only the happy path and discovering at 3 a.m. that the shedder itself is slow.

Metrics to watch

• Shed rate per priority class and per route.
• Goodput, meaning successful responses per second, versus offered load.
• Queue delay and p99 latency of admitted requests.
• Proportion of requests dropped by deadline expiry downstream.
• Client retry rate immediately after rejections.

Example configuration

load_shedder:
  signal: queue_delay_ms
  classes:
    critical:
      shed_above: never
    standard:
      shed_above: 150
    sheddable:
      shed_above: 50
  max_concurrency: 800
  response:
    status: 503
    retry_after_seconds: 2
  exempt_paths: ["/healthz", "/readyz"]
  deadline_header: x-request-deadline

Key takeaways

• Shed early, cheaply, and by priority, never uniformly at random.
• Measure harm with queue delay rather than relying on CPU.
• Exempt health endpoints and propagate deadlines.
• Tell clients what to do after rejection, or retries will undo the benefit.
• Load test the policy before an incident, not during one.