Incidents are inevitable; trauma is optional. The difference between a team that learns from outages and one that quietly rage-quits is process, clear roles, blameless review, and a hard line on toil.
Roles, not heroes
The lone genius firefighting at 3am is a failure mode, not a badge. Split the work so no one person carries the whole load:
Incident Commander, owns decisions and coordination, not keyboards
Comms lead, updates stakeholders so responders are not interrupted
Ops/responders, actually investigate and mitigate
Scribe, timestamps the timeline as it happens
Mitigate first, diagnose later
Stop the bleeding (roll back, fail over, shed load) before root-causing
Communicate on a fixed cadence, even when there is no news
Keep a running timeline, memory is the first casualty of an incident
Declare the incident over explicitly, and hand off if it is long
Blameless postmortems that change something
A postmortem that assigns blame teaches people to hide problems. A blameless one asks how the system let a reasonable person make that call, and produces action items with owners.
# Postmortem: checkout 500s (2026-07-10)
## Impact
- 22 min of elevated errors; ~3% of checkouts failed
## Timeline (UTC)
- 14:02 deploy of api v7
- 14:05 error rate crosses SLO; alert fires
- 14:11 IC declares incident, rollback started
- 14:24 errors back to baseline
## Root cause
- New migration held a lock under load
## Action items (owner, due)
- [ ] Add lock-timeout to migrations (@dba, Jul 17)
- [ ] Canary analysis on error rate (@platform, Jul 21)You do not rise to the occasion during an incident; you fall to the level of your preparation and your process.
Protect your people and the reliability follows. Rotate on-call fairly, budget real time for action items, and treat every incident as the system telling you where to invest next.
