Traffic Mirroring Framework

Traffic mirroring, also called shadowing, sends a copy of production requests to a second system while the original request is served normally. It lets teams observe how a new version behaves under real conditions before it affects anyone.

How it works

A proxy, load balancer, or service mesh receives each request, forwards it to the primary, and in parallel sends a duplicate to the shadow target. The copy is fire-and-forget: its response is discarded and its latency or failure cannot delay or break the user's request.

Analysis happens out of band:

• Error and latency comparison between primary and shadow for the same requests.
• Response diffing after normalizing volatile fields such as timestamps, request ids, and ordering.
• Capacity observation: CPU, memory, and downstream load of the shadow at the mirrored rate.

Mirrored requests are tagged with a header so the shadow knows it is not serving a real user and can suppress side effects.

Implementation steps

1. Choose the scope: read-only endpoints first, with a sampling percentage such as five.
2. Deploy the shadow with its own resources and, where it writes, its own datastore or a stub.
3. Configure the mirror in the proxy, with a mirror header and a timeout that cannot affect the primary path.
4. Strip or mask personal data and secrets from mirrored payloads, and confirm the shadow environment meets the same data policies.
5. Build the comparison pipeline with field normalization and mismatch classification.
6. Raise the sample rate gradually while watching shared dependencies, such as caches and databases.
7. Use findings to fix mismatches, then proceed to canary release.

Common pitfalls

• Shadow performing real writes, sending emails, or charging cards.
• Mirroring consuming proxy resources until the primary path slows.
• Shared downstream databases being hit twice, causing contention.
• Comparing responses without normalization, producing thousands of false mismatches.
• Non-idempotent reads that mutate state, such as counters or caches.

Metrics to watch

• Mismatch rate by endpoint and by category.
• Shadow versus primary p50 and p99 latency.
• Shadow error rate.
• Added load on shared dependencies.
• Proxy overhead on primary requests.

Example configuration

route:
  match: /api/v2/search
  primary: search-v1
  mirror:
    target: search-v2-shadow
    sample_percent: 5
    timeout: 500ms
    add_headers:
      x-shadow: "true"
    strip_headers: [authorization, cookie]
  comparison:
    ignore_fields: [timestamp, request_id]
    sort_arrays: true

Key takeaways

• Mirror asynchronously and discard shadow responses.
• Start with idempotent reads and a small sample.
• Isolate or stub every side effect and sanitize data.
• Normalize before diffing, or the signal drowns in noise.
• Treat mirroring as a precursor to canary, not a replacement.
• Review mirrored data handling with security before enabling it, and keep an audit trail of what was copied.
• Repeat the exercise after each significant change to the system, because results go stale.