Request Coalescing

Overview

Request coalescing, also called single-flight or request collapsing, ensures that only one execution of an identical operation runs at a time. It targets the window in which a cache cannot help yet: the moment between the first miss and the first stored result.

How it works

inflight = {}

func get(key):
  if key in inflight: return inflight[key].wait()
  call = start(loader, key)
  inflight[key] = call
  try:    return call.wait()
  finally: delete inflight[key]

Go's singleflight package implements this idea directly. CDNs and reverse proxies do the same with request collapsing: when 500 clients request an uncached object, one request goes to the origin and the other 499 wait for its response.

Choose the key carefully: normalize parameters and exclude per-request noise such as trace ids. Run the shared call with its own timeout, detached from any individual caller's context, so one impatient caller cannot cancel the work for the rest.

Coalescing works best with a cache. The shared result is stored, so later arrivals hit the cache instead of starting a new flight.

Pitfalls

• Sharing a call across users with different authorization. Include the identity in the key when results differ.
• Propagating a caller's cancellation into the shared call.
• Forgetting to remove the entry on error, which poisons the key.
• Assuming it works across instances. In-process coalescing reduces calls by instance count, not to one. Use a shield tier for global collapse.
• Unbounded waiter lists on very slow calls. Time out waiters separately.

Key takeaways

• One in-flight call per key, shared by all concurrent callers.
• Use it for idempotent reads, never for writes.
• Pair with caching and a decoupled timeout.
• Keep the key tight and authorization-aware.