Stackbook Logo
scalingestablished · medium operational burden

Tail Latency Reduction with Hedged Requests

Also known as: hedged-requests, tail-at-scale

Intent

Reduce p99 latency by sending a duplicate request to another replica when the first is slow.

Problem

In fan-out systems the slowest backend determines overall latency, so a rare 1 percent slow response per backend becomes common for the end user. Garbage collection pauses, noisy neighbors and queueing cause slow outliers that averages hide.

Forces

  • •Fan-out multiplies the effect of rare slow calls
  • •Duplicate requests add load
  • •Operations must be idempotent to be safely repeated
  • •Slow replicas are usually slow only temporarily

Solution

Overview

Tail latency is the high percentile of response time, for example p99. When one request touches 100 backends and each is slow 1 percent of the time, about 63 percent of user requests hit at least one slow backend. Hedged requests, described in the paper The Tail at Scale, attack this directly.

How it works

t=0     send request to replica A
t=p95   no reply yet -> send same request to replica B
first reply wins -> cancel the other

The hedge delay is set from measured latency, typically the p95 of that call, so only about 5 percent of requests are duplicated. The extra load is small, yet requests stuck behind a pause or a hot queue finish on the second replica. A variation is the tied request: send to two replicas at once, each told about the other, and the first to start executing cancels its twin.

Guard the mechanism with a budget, for example hedges may not exceed 5 percent of requests, tracked with a token bucket. Under broad slowdown, the budget empties and hedging stops, which prevents amplification.

Other tail techniques work alongside hedging: set aggressive per-call deadlines, pick the replica with the shortest queue, split large requests into smaller ones, and isolate background work from serving threads.

Pitfalls

  • Hedging writes or other non-idempotent calls.
  • Ignoring cancellation, so both replicas finish the work and waste capacity.
  • Static hedge delays that drift as latency changes. Recompute from recent percentiles.
  • Hedging into the same failing replica set or the same zone.
  • Measuring average latency, which hides the problem. Track p99 and p99.9 per dependency.

Key takeaways

  • Fan-out turns rare slowness into common slowness.
  • Hedge after roughly p95 latency, with a strict budget.
  • Use only for idempotent calls and cancel the loser.
  • Watch tail percentiles, not averages.

✓ When to Use

  • •Read-only fan-out queries across many shards
  • •Latency-sensitive paths with multiple equivalent replicas
  • •Services where p99 dominates user experience

✗ When Not to Use

  • •Non-idempotent writes
  • •Systems already near capacity, where duplicates worsen overload
  • •Backends with a single replica or shared bottleneck

Pros

  • +Cuts p99 and p99.9 sharply for little extra load
  • +Masks transient slowness without detection
  • +Simple to add in the client or proxy

Cons

  • −Increases total request volume
  • −Needs idempotency and cancellation support
  • −Can amplify load during widespread slowdowns

Cost Profile

Infrastructure

Small extra capacity for duplicates

Operational

Medium: tune delay and budget

Cognitive

Medium: idempotency and cancellation

Failure Modes

  • Hedging during overload doubles traffic and triggers collapse

  • Duplicate execution of a non-idempotent call

  • Hedge delay set too low, so most requests are duplicated

Real-World Examples

Alternatives

    Related Patterns

    • retry-with-backoff
    • load-shedding
    • circuit-breaker

    Competency Domains

    reliability opsdistribution communication