Incident Response Framework

Incidents are inevitable; chaos during them is not. This framework gives responders a shared script so effort goes into restoring service rather than negotiating who does what.

How it works

Anyone can declare an incident when a trigger is met, for example an SLO fast burn or customer-impacting error. Declaration opens a dedicated channel, a video bridge and a timeline document, and assigns a severity. The incident commander takes charge and delegates: the operations lead investigates and applies changes, the communications lead updates status pages and stakeholders, and the scribe records decisions with timestamps.

The framework scales by adjusting formality rather than changing the roles. A small team may have one person fill several roles at low severity, but the commander and the scribe should remain distinct even then. Larger organisations add area leads for each affected system and a dedicated liaison for customer support and executives, so that responders are never interrupted by status requests while they are working on mitigation. Tooling should reduce friction: one command to open the channel, page the roles and start the timeline.

Implementation steps

1. Define severity levels by customer impact and duration, not by technical cause.
2. Write the declaration rule: when in doubt, declare.
3. Create role cards describing responsibilities and handoff phrases.
4. Automate setup with a chat command that creates the channel, document and pages the roles.
5. Set communication cadence, such as a status update every 30 minutes for the highest severity.
6. Train commanders through tabletop exercises and shadowing.
7. Require a postmortem for every incident above a defined severity.

Pitfalls

• The most senior engineer commands and debugs simultaneously, losing the overview.
• Several parallel changes made without announcing them, hiding the real cause.
• Debating root cause while a known-good rollback is available.
• Updates that describe technical detail but not customer impact.
• No handoff at the two-hour mark, leaving exhausted responders in charge.

Metrics to watch

• Time to detect, time to acknowledge and time to mitigate.
• Incident count by severity and by service.
• Percentage of incidents with a postmortem completed within a week.
• Repeat incidents sharing a contributing cause.
• Responder load and after-hours pages per person.

Example response policy

severities:
  sev1:
    definition: "Core user journey unavailable for many customers"
    page: [commander, ops-lead, comms-lead]
    update_interval_minutes: 30
    postmortem_required: true
  sev2:
    definition: "Degraded experience or partial outage"
    page: [commander, ops-lead]
    update_interval_minutes: 60
    postmortem_required: true
  sev3:
    definition: "Minor impact with a workaround"
    page: [ops-lead]
    update_interval_minutes: 240
    postmortem_required: false
on_declare:
  - create_channel: "inc-YYYYMMDD-short-name"
  - create_timeline_doc: true
  - update_status_page: "investigating"

Key takeaways

• Separate coordination from debugging so someone always holds the big picture.
• Mitigate first, investigate later.
• Declare early and cheaply; de-escalate when safe.
• Practice the process before you need it.