Retry Budget

Overview

A retry budget answers the question that per-request retry counts cannot, namely how much extra load retries are allowed to add in total. Envoy and gRPC both ship the idea as configuration.

How it works

Maintain a token bucket per destination. Successful calls deposit a fraction of a token, and each retry withdraws a whole one.

ratio = 0.1
on success:  tokens = min(max, tokens + ratio)
on failure:
  if tokens >= 1 then tokens -= 1; retry
  else fail fast

With a ratio of 0.1, retries can never exceed ten percent of successful traffic in steady state. During a full outage the bucket drains and retries stop, so the client falls back to a single attempt per request. A small minimum allowance keeps low-traffic clients functional.

Pitfalls

• If three tiers each retry three times, one failure can produce 27 calls. Retry at one layer and propagate errors upward.
• Counting retries per endpoint but sharing the budget per host can hide a hot endpoint.
• A budget does not replace backoff, since synchronized retries still spike without jitter.
• Retrying non-idempotent calls remains unsafe whatever the budget.
• Do not count rejected retries as errors twice in dashboards.

Key takeaways

• Bound retries by ratio, not just by count.
• Use token buckets fed by successes.
• Retry in exactly one layer.
• Combine with backoff, jitter, and circuit breakers.
• Expose the remaining budget as a metric so operators can see when retries are being suppressed and correlate it with downstream error rates.