Shadow Traffic

Overview

Shadow traffic, also called mirroring or dark launching, sends a duplicate of production requests to a candidate version. Users are served only by the current version, so a defective candidate costs nothing but compute.

How it works

The proxy tee's each request. The primary response goes back to the client; the mirror is fire-and-forget.

client --> proxy --> stable v1 --> response to client
              |
              +--(copy, async)--> candidate v2 --> discarded
                                        |
                                  diff + metrics store

In Envoy or Istio this is a mirror policy with a sample percentage. A comparison job normalises volatile fields, such as timestamps and generated ids, then reports mismatch rates and latency percentiles.

Pitfalls

• Mirroring writes. Candidate must run with a sandboxed database, dry-run mode or stubs for third-party calls.
• Forwarding authentication tokens or personal data to environments with weaker controls.
• Synchronous mirroring, which makes the primary wait for the shadow.
• Comparing without normalisation, producing thousands of false diffs.
• Forgetting that the shadow stack needs production-scale capacity to be a valid load test.
• Starting at 100 percent; ramp the sample rate gradually.

Key takeaways

• Mirror requests, discard responses, compare results.
• Isolate every side effect before mirroring.
• Keep mirroring asynchronous and sampled.
• Normalise nondeterminism in comparisons.
• Treat it as validation before a canary, not a replacement for one.