Retry Budget Framework

Retries convert transient failures into successes, but they also multiply traffic precisely when a dependency is weakest. A retry budget makes the amplification explicit and bounded: you may retry, but only up to a fixed share of normal traffic.

How it works

Consider a call chain of four tiers where each makes up to three attempts. In the worst case the bottom tier sees 3 to the power of 4, or 81, times the original request. A budget removes the exponent.

The client tracks two counters over a sliding window of, say, ten seconds: original requests and retries. A retry is permitted only if retries are less than a set ratio of original requests, such as 0.1, plus a small minimum allowance so low-traffic clients can still retry. Once the ratio is reached, failures are surfaced immediately.

The budget works alongside standard hygiene:

• Exponential backoff with full jitter to spread retries.
• Retry only idempotent calls, or those with idempotency keys.
• Retry only retryable codes, such as timeouts and 503, not 400 or 404.
• Respect an overall deadline that spans all attempts.

Implementation steps

1. Map call chains and decide which single layer owns retries; others make one attempt.
2. Choose a budget ratio, starting at ten percent, and a minimum of a few retries per second.
3. Implement a shared token or counter mechanism in the client library or sidecar.
4. Classify errors into retryable and not; log the classification.
5. Add jitter and a maximum backoff; propagate the deadline.
6. Emit metrics for retries attempted and retries denied.
7. Test with fault injection: make a dependency fail completely and confirm total load stays near 1.1 times normal.

Common pitfalls

• Retrying at the client, the gateway, and the service mesh simultaneously.
• Budgets tracked per connection rather than per client, so each connection has its own allowance.
• Retrying non-idempotent writes and creating duplicates.
• Backoff with no jitter, producing synchronized waves.
• Treating budget exhaustion as an error to alert on without context, or ignoring it entirely.

Metrics to watch

• Retry ratio, meaning retries divided by original requests.
• Count of retries denied by the budget.
• Success rate of retries; a low value means retries are wasted.
• Request amplification factor at each tier.
• Latency contributed by backoff.

Example configuration

retry_policy:
  budget:
    ratio: 0.1
    min_retries_per_second: 3
    window: 10s
  max_attempts: 3
  backoff:
    type: exponential
    base: 100ms
    max: 5s
    jitter: full
  retry_on: [timeout, connect_failure, http_503]
  idempotent_only: true
  overall_deadline: 2s

Key takeaways

• Bound retries as a share of traffic, not as a per-call count.
• One layer should own retrying; others should not.
• Use jitter, deadlines, and idempotency.
• Verify with fault injection that amplification stays near the budget.
• Retry success rate shows whether retries are helping at all.