Empirical Studies of Outages and Postmortems: What We Actually Know

Few topics connect research and practice as directly as the study of why production systems fail. Jim Gray's 1986 paper Why Do Computers Stop and What Can Be Done About It examined fault-tolerant systems and found that administration and software had become significant sources of outages as hardware improved. Decades later, researchers and companies still analyze the same question using richer data.

What the research says

David Oppenheimer, Archana Ganapathi and David Patterson studied failures at several large Internet services in the early 2000s and found that operator error and configuration problems were leading contributors to outages, ahead of hardware. Haryadi Gunawi and colleagues, in the 2016 paper Why Does the Cloud Stop Computing, analyzed hundreds of publicly reported cloud service outages and examined root causes, impact and the role of failure recovery, finding that many outages involved failures in the supposedly redundant recovery mechanisms themselves, and that upgrades, configuration and software bugs were common triggers. Later studies of production cloud incidents have likewise found that misconfiguration, deployment changes and bugs matter greatly, with consistent patterns across providers.

Richard Cook's 1998 essay How Complex Systems Fail, drawn from healthcare, offers a conceptual frame often cited by software engineers: complex systems run in a degraded mode, catastrophe usually requires multiple failures, and attribution to a single root cause is often a social choice more than a technical finding. Sidney Dekker's work on just culture and John Allspaw's writing on blameless analysis carry similar themes.

What practitioners see

Public postmortems from companies such as Amazon, Google, Cloudflare and others describe a recurring shape: a routine change, a latent defect or an assumption that had been true until it was not, an amplifying mechanism such as retry storms or cascading overload, and a slow recovery because tools or visibility depended on the broken system. The 2017 S3 outage postmortem, in which an incorrectly entered command removed more capacity than intended, is a widely discussed example, and the response described changes to tooling to limit the blast radius of such commands.

The Google SRE books popularized blameless postmortems with action items, and many organizations have adopted templates. Teams report that the benefit depends on follow-through. A postmortem that produces a document but no completed remediation, or whose actions are generic such as add more monitoring, changes little. Another observed pattern is that the written root cause list hides contributing factors that were uncomfortable to name, such as schedule pressure or understaffed on-call.

Gaps

• Public postmortems are a biased sample: they come from companies willing to publish and from incidents they chose to publish.
• Postmortems are written after the fact, with hindsight bias, and different authors may assign different causes to the same event.
• Categories such as human error are not consistent across studies, so statistics are hard to compare.
• Little rigorous evidence exists on whether postmortem practices reduce future incident frequency, as opposed to improving learning within teams.

How to apply it

Run a blameless review for significant incidents and separate the timeline of facts from the analysis. Ask how the system made the action reasonable at the time, not who made the error. Look for contributing factors in tooling, process and organization, and avoid ending at a single root cause.

Limit action items to a few that have owners and dates, and track completion. Prefer actions that reduce blast radius, add safe defaults or remove manual steps over those that only ask people to be careful.

Keep an incident library and review it for themes quarterly, since patterns across incidents reveal more than any one. Use those themes to choose architecture work, fault injection experiments and on-call improvements.

Key takeaways

• Studies repeatedly point to operations, configuration, changes and failed recovery paths, not only hardware or novel bugs.
• Complex failures have multiple contributing factors, so single root cause statements are simplifications.
• Public postmortems are informative but form a biased sample.
• Blameless reviews help only when action items are completed and meaningful.
• Review incidents in aggregate to find systemic themes.