Stackbook Logo
resilienceestablished · medium operational burden

Hedged Requests

Also known as: backup-requests, speculative-retry

Intent

Cut tail latency by sending a duplicate request to another replica when the first has not answered within a short delay.

Problem

In a fan-out system a small fraction of slow responses dominates overall latency, because the slowest call determines the result. Slowness is often transient and specific to one replica, such as a GC pause or noisy neighbor.

Forces

  • •Tail latency is driven by rare slow replicas
  • •Extra requests add load
  • •Duplicated work must be safe to repeat
  • •Delay choice trades latency against cost

Solution

Overview

Hedged requests, described in Google's work on tail latency, treat slowness as a probabilistic event. Rather than waiting for a slow replica, the client races a second attempt.

How it works

The client issues the first request, waits for a hedge delay, then issues a backup to a different replica. The first successful reply wins and the other is cancelled.

t=0     send to replica A
t=95ms  no reply yet (p95), send to replica B
t=110ms B replies, cancel A, return result

A good hedge delay is the observed p95 or p99 of the operation. A token bucket limits the share of hedged traffic, for example ten percent of requests, so the mechanism cannot run away. Servers that support cancellation stop work early, which keeps the cost close to the duplicate-rate.

Pitfalls

  • Hedging writes or other non-idempotent calls causes duplicate side effects.
  • Without a budget, hedging under overload creates a retry storm in disguise.
  • Choosing replicas that share a failure domain removes the independence the technique relies on.
  • Ignoring cancellation wastes server capacity on results nobody reads.
  • Delay set too low hedges almost every request and doubles load.

Key takeaways

  • Hedge only idempotent operations.
  • Set the delay near p95 and cap the hedge rate.
  • Send the backup to a different replica or zone.
  • Disable hedging when the system is shedding load.
  • Measure the hedge rate and the win rate of backup requests. If backups rarely win, the delay is too low or the slowness is not replica specific, and the extra load is wasted.

✓ When to Use

  • •Idempotent reads against replicated data
  • •Fan-out queries where p99 dominates
  • •Systems with spare capacity and variable replica latency

✗ When Not to Use

  • •Non-idempotent writes
  • •Systems already near saturation
  • •Failures caused by overload rather than replica slowness

Pros

  • +Large reduction in p99 and p999 latency
  • +Small average load increase
  • +Masks transient replica slowness

Cons

  • −Extra load and cost
  • −Needs idempotency and cancellation support
  • −Can amplify overload if unbounded

Cost Profile

Infrastructure

Low to medium, around five percent extra requests

Operational

Medium, tune delay and budget

Cognitive

Medium, concurrency and cancellation

Failure Modes

  • Hedging during overload doubles traffic and deepens the outage

  • Non-idempotent operation executed twice

  • Both replicas slow for the same reason, so hedge adds load without benefit

Real-World Examples

Alternatives

    Related Patterns

    • retry-with-backoff
    • circuit-breaker

    Competency Domains

    reliability opsdistribution communication