Request Hedging Guide

Hedging reduces tail latency by speculatively duplicating a request that is taking suspiciously long. It exploits the fact that slowness is usually transient and replica-specific: a different replica will probably answer quickly.

How it works

The client sends the request to replica A and starts a timer equal to the hedge delay. If A responds first, the timer is cancelled and nothing else happens. If the timer fires, the client sends the same request to replica B, then accepts the first successful response and cancels the loser.

The hedge delay is the key parameter. Set it to the observed p95 latency of the operation, and only about five percent of requests are hedged, adding roughly five percent load while sharply cutting p99. A lower delay hedges more traffic and costs more for diminishing return.

A variation, tied requests, sends both copies immediately but tags them so each server cancels its copy when the other starts execution. This gives better latency at the cost of added coordination.

Implementation steps

1. Confirm the operation is idempotent and side-effect free, or protected by idempotency keys.
2. Measure latency per operation and replica set; compute the percentile to use as the delay.
3. Implement the hedge in the client with a rolling latency estimate so the delay adapts.
4. Choose a different replica for the second attempt, not the same one.
5. Add a hedging budget, for example hedges at most ten percent of requests, and disable hedging when the backend signals overload.
6. Propagate cancellation, such as gRPC context cancel, so the slower request stops work.
7. Load test with injected slow replicas and compare p99 before and after.

Common pitfalls

• Hedging during overload, doubling load when capacity is already short.
• Sending the second request to the same slow replica.
• Static delay that becomes wrong after a deployment changes latency.
• Ignoring that cancelled requests may still consume resources if servers do not check cancellation.
• Hedging writes, creating duplicates.

Metrics to watch

• Hedge rate and hedge win rate, meaning how often the second request wins.
• p50, p99, and p99.9 latency before and after hedging.
• Total backend request amplification.
• Cancelled request count and wasted work.
• Hedges suppressed by the budget.

Example configuration

hedging:
  enabled: true
  operations: [GetProfile, SearchIndex]
  delay:
    mode: adaptive
    percentile: 95
    min: 20ms
    max: 300ms
  max_extra_load: 0.10
  max_hedges_per_request: 1
  pick: different_replica
  disable_when:
    backend_overload_signal: true

Key takeaways

• Hedge only idempotent reads, after a delay near p95.
• Cap extra load with a budget and turn hedging off during overload.
• Send the duplicate to a different replica and cancel the loser.
• A high win rate for hedges signals a replica or capacity problem worth fixing.
• Verify gains with percentile comparisons, not averages.