Canary Analysis

Overview

Canary analysis automates the verdict on a canary release. Systems such as Kayenta (Spinnaker) and Flagger compare canary metrics with a baseline and return a score, turning a judgement call into a pipeline step.

How it works

Three groups run side by side: production, a new baseline built from the current version, and the canary built from the candidate. Comparing canary to a fresh baseline, rather than to long-running production, removes warm-up and age effects.

traffic 5% --> canary   (v2) \
traffic 5% --> baseline (v1) / compare for 30 min
rest       --> production (v1)

score = weighted pass rate of metrics
score >= 90 pass | 75-90 inconclusive | below 75 fail

Each metric is tagged by direction (lower is better for errors and latency) and criticality. A critical metric failing fails the whole run regardless of overall score.

Pitfalls

• Choosing only technical metrics. Add business measures such as checkout conversion.
• Windows too short to cover caching, garbage collection and periodic jobs.
• Treating inconclusive as success. Extend the window or require human review.
• Comparing against old production processes with different uptime.
• Over-tuned thresholds that cause constant false failures, so teams bypass the gate.
• Insufficient traffic share to reach statistical power.

Key takeaways

• Compare canary with a same-age baseline, not with all of production.
• Define metrics, direction and critical flags up front.
• Treat inconclusive as not-yet-safe.
• Include business and tail latency signals.
• Revisit thresholds regularly using past releases.