Stackbook Logo
deploymentestablished · low operational burden

Immutable Infrastructure

Also known as: immutable-servers, phoenix-servers

Intent

Never modify running servers; replace them with freshly built images for every change.

Problem

Servers patched in place accumulate unique history. Over time no two are identical, failures cannot be reproduced, and a rollback restores neither the OS nor the packages that were changed along the way.

Forces

  • •Configuration drift causes unreproducible failures
  • •Rollbacks must restore a known good state
  • •Patching live machines is risky and slow to verify
  • •Build and rollout time grows with image size

Solution

Overview

With immutable infrastructure, a running server is a read-only artifact. If something must change, you build a new image, launch replacements and terminate the old servers. Containers made this the default, but it applies equally to virtual machines.

How it works

CI builds an image tagged with the commit. The deployment layer launches instances from it, waits for health checks, shifts traffic, and then destroys the previous generation.

commit --> CI --> image:v42
                    |
        launch v42 instances --> health check --> add to LB
                                                   |
        drain v41 instances <----------------------+
        terminate v41

Anything that must survive replacement lives outside the instance: databases, object storage, attached volumes, or secrets fetched at startup.

Pitfalls

  • Allowing SSH changes in production, which quietly reintroduces drift. Disable shell access or auto-replace touched nodes.
  • Baking secrets into images, which leaks them to anyone with registry access.
  • Building huge images that make every deploy slow. Use minimal bases and layer caching.
  • Forgetting that old images age. Rebuild regularly so base OS patches flow in even without code changes.
  • Treating local disk as durable storage.
  • No image provenance, so you cannot tell what is inside a running instance. Sign images and keep an SBOM.

Key takeaways

  • Replace, do not repair.
  • Rollback equals redeploying the previous image.
  • Externalise all state and secrets.
  • Rebuild images on a schedule for security patches.
  • Prevent manual changes technically, not just by policy.

✓ When to Use

  • •Autoscaled fleets and container platforms
  • •Environments where drift and snowflake servers cause incidents
  • •Security programmes that require frequent, verifiable patching

✗ When Not to Use

  • •Stateful hosts where data lives on local disks with no replication
  • •Very small setups where image pipelines cost more than they save
  • •Hardware that cannot be reprovisioned quickly

Pros

  • +Identical instances and reproducible failures
  • +Simple, reliable rollback
  • +Patching becomes a rebuild and redeploy

Cons

  • −Needs image build pipeline and registry
  • −Debugging requires good telemetry since SSH is discouraged
  • −Stateful workloads need careful redesign

Cost Profile

Infrastructure

Medium - image registry, temporary overlap capacity during rollout

Operational

Low - fewer snowflakes once pipeline exists

Cognitive

Medium - state externalisation discipline

Failure Modes

  • Someone hand-patches a node, recreating drift

  • Local state is lost when an instance is replaced

  • Base image vulnerability is baked into the whole fleet

Real-World Examples

Alternatives

  • rolling-update

Related Patterns

  • infrastructure-as-code
  • blue-green-deployment
  • kubernetes-patterns

Competency Domains

reliability opssecurity compliance