Synthetic Monitoring

Overview

Synthetic monitoring, also called black-box monitoring, tests a system from the outside the way a user would. It complements white-box metrics, which only report what the system knows about itself. A probe fails when the user path fails, regardless of whether internal dashboards are green.

How it works

Probes run from multiple regions on a fixed interval and record success, latency, and step timings.

every 60s, from eu, us, ap:
  GET  /health/deep           expect 200 under 500ms
  POST /login (test account)  expect token
  GET  /cart                  expect 200
  POST /checkout (sandbox)    expect order id
  check TLS expiry            warn at 14 days

alert if: 2 of 3 regions fail 3 consecutive runs

Probe results become an availability SLI: successful runs divided by total runs. Use a dedicated header or account so requests are identifiable in logs and excluded from product metrics. Journey steps should reuse the same code paths as real traffic, including the CDN and public DNS name, not an internal endpoint.

Pitfalls

• Probing only a shallow health endpoint. It can return 200 while the database is unreachable; probe real journeys.
• Running from one location, which turns that location's problems into false incidents.
• Letting scripts rot. Treat them as code with tests and owners.
• Forgetting cleanup of created data, which grows tables and skews reports.

Key takeaways

• Probes catch failures that internal metrics cannot see.
• Run from several locations and require agreement before paging.
• Use dedicated test accounts and tag the traffic.
• Combine with real user monitoring for full coverage.