resilienceestablished · medium operational burden

Timeout Budget

Also known as: latency-budget, end-to-end-timeout

Intent

Allocate a fixed end-to-end time limit across the hops of a request so the total never exceeds what the caller will wait.

Problem

Each service in a call chain sets its own timeout independently, so the sum of downstream timeouts can far exceed the client timeout. Work continues long after the user has given up, and slow dependencies hold threads and connections.

Forces

•Callers have a hard upper bound on acceptable latency
•Downstream hops have different typical and tail latencies
•Retries multiply total elapsed time
•Too-tight timeouts cause false failures

Solution

Overview

A timeout budget treats time like a shared resource. The edge of the system decides how long the user will wait, and every hop spends from that single allowance instead of choosing a timeout in isolation.

How it works

Start with the client limit, subtract local processing, and divide the rest among dependencies using their observed p99 latency. Keep a small reserve for variance.

client budget: 800 ms
  gateway work          20 ms
  auth service         100 ms  (p99 70)
  catalog service      300 ms  (p99 220)
  pricing service      250 ms  (p99 180)
  reserve              130 ms

Each service computes its outgoing timeout as the minimum of its allocation and the time left on the clock. If a retry would not fit in the time left, it is skipped. Pair this with deadline propagation so downstream services know the remaining time rather than guessing.

Pitfalls

Using averages instead of p99 produces timeouts under normal load.
Allowing retries to ignore the remaining budget turns a bounded call into an unbounded one.
Parallel fan-out is bounded by the slowest branch, not the sum, so budget it as a max.
Budgets rot. A dependency that gets slower silently eats the reserve, so alert on budget exhaustion rates.
Queue wait before the handler starts also counts against the budget.

Key takeaways

Choose the total first, then divide it, never the reverse.
Base allocations on tail latency and leave explicit slack.
Every retry must check the remaining budget.
Monitor how often each hop exhausts its share.

✓ When to Use

•Multi-hop synchronous request chains
•User-facing APIs with a latency SLO
•Fan-out calls where the slowest dependency dominates

✗ When Not to Use

•Long-running batch jobs with no caller waiting
•Purely asynchronous workflows
•Single-hop calls where one timeout is enough

Pros

+Bounds worst-case latency predictably
+Frees resources for abandoned work sooner
+Makes latency ownership explicit per dependency

Cons

−Budgets need ongoing re-tuning
−Tight splits can cause spurious timeouts
−Requires agreement across teams

Cost Profile

Infrastructure

None, configuration and libraries

Operational

Medium, budgets drift with traffic and code

Cognitive

Low to medium, simple concept

Failure Modes

Budgets set from averages rather than tail latencies cause frequent timeouts
Retries ignore the remaining budget and overrun it
Stale budgets after a dependency slows down cause an outage that looks like a client bug