Stackbook Logo
deployment

Progressive Delivery Framework

Intent

Release changes gradually to growing audiences with automated analysis, limiting the blast radius of defects.

Problem

Deploying a change to all users at once turns every release into a gamble, and problems are discovered by customers rather than by monitoring. Teams respond by slowing releases down, which increases batch size and makes failures even larger.

Forces

    Solution

    Progressive Delivery Framework

    Progressive delivery extends continuous delivery by controlling who sees a change and when. Instead of one irreversible switch, a release is a sequence of measured steps.

    How it works

    A new version is deployed alongside the stable one with no traffic. A rollout controller shifts a small share of requests, collects metrics for both versions, and runs analysis queries. If the canary stays within thresholds for the configured duration, the controller moves to the next weight. A breach aborts the rollout and returns all traffic to the stable version. Feature flags add a second layer by gating functionality independently of the binary.

    Progressive delivery has two complementary layers. Traffic-based rollouts control which version of a binary receives requests, and are best for infrastructure-level risk such as performance regressions. Flag-based rollouts control which users see a behaviour, and are best for product-level risk such as a new checkout flow. Using both lets a team ship code dark, ramp exposure by cohort and turn a feature off instantly without redeploying, while keeping the deployment pipeline itself simple and fast.

    Implementation steps

    1. Establish baseline SLIs for each service: error ratio, latency percentile and saturation.
    2. Choose a traffic splitting layer, such as a service mesh, ingress or load balancer weights.
    3. Define stages, for example 1, 5, 25 and 100 percent, with minimum durations and minimum request counts.
    4. Write analysis rules comparing canary and stable, not absolute values only.
    5. Automate rollback and notification on failure.
    6. Ensure database and API changes are backward compatible across versions.
    7. Retire old flags and rollout configurations after completion.

    Pitfalls

    • Promoting based on time alone without enough traffic to detect a problem.
    • Sticky sessions or caches that skew the canary population.
    • Ignoring business metrics such as conversion, which technical metrics miss.
    • Leaving flags in place indefinitely.
    • Schema changes that break the older version still serving traffic.

    Metrics to watch

    • Percentage of releases auto-promoted, paused and rolled back.
    • Time from first canary to full rollout.
    • Change failure rate and customer-reported incidents per release.
    • Mean time to rollback.
    • Number of stale flags older than ninety days.

    Example Argo Rollouts strategy

    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    metadata:
      name: checkout
    spec:
      replicas: 10
      strategy:
        canary:
          analysis:
            templates:
              - templateName: error-rate-vs-baseline
          steps:
            - setWeight: 5
            - pause: {duration: 10m}
            - setWeight: 25
            - pause: {duration: 15m}
            - setWeight: 50
            - pause: {duration: 15m}
    

    Key takeaways

    • Separate deployment from release to control exposure.
    • Decide promotion and rollback criteria before starting.
    • Compare against a live baseline rather than fixed thresholds.
    • Backward compatibility across versions is a prerequisite.

    ✓ When to Use

    • •Defects in production are costly or hard to reverse
    • •You deploy frequently to a large user base
    • •Reliable service-level indicators already exist
    • •Different customer segments tolerate risk differently

    ✗ When Not to Use

    • •Traffic is too low to compare baseline and canary meaningfully
    • •Changes are irreversible, such as destructive data migrations
    • •No automated health signals exist

    Pros

    • +Limits blast radius of bad releases
    • +Detects regressions using real production traffic
    • +Enables automatic rollback without human latency
    • +Supports frequent small releases

    Cons

    • −Needs good telemetry and traffic routing
    • −Multiple versions run simultaneously, requiring compatibility
    • −Slower to reach 100 percent than a plain deploy
    • −Flag and variant sprawl adds complexity

    Cost Profile

    Infrastructure

    Operational

    Cognitive

    Failure Modes

      Alternatives

        Related Patterns

          Competency Domains

          reliability opsrequirements constraintsdistribution communication