Incident Response: Triage, Mitigate, Verify, Postmortem

Why this matters

An outage pages at night and five engineers debug five theories in parallel while nobody tells users anything. Two fixes collide, the timeline is reconstructed from memory a week later, and the postmortem concludes human error with no action that would stop a recurrence. The failure was not the bug, it was the absence of a practiced shape: one lead, one timeline, mitigation first, learning last. This lesson is that shape, and the L48 lab runs it end to end on a short outage.

Concepts

Triage bounds the incident: what users feel, since when, how broad. Severity follows user impact, not technical drama. One lead coordinates; everyone else works a task or watches. Communication starts early with what is known and what is next, even when the known is small; silence manufactures rumors faster than incidents manufacture facts.

Mitigation restores service by the fastest recorded path: rollback (dp14l4), traffic shift, feature flag, scaled capacity. Diagnosis waits its turn; curiosity during user impact is billed to users. Verification proves restoration with user-shaped evidence (success rates, checkout completions from the M15 toolkit), because green pods with failing checkouts are the incident continuing quietly.

The postmortem is written within days, blameless by construction: what happened (timeline from the recorded log), why each decision made sense at the time, what the causes were (plural, systemic, never a name), and actions with owners and dates. Action items fix systems and guards (alerts, gates, runbooks); training issues get training, not punishment. Unwritten incidents recur; the writing is the prevention.

Runbooks carry the practiced motions: symptom, diagnosis steps with commands, the mitigation path, the verification check, escalation. The M07 runbook lesson returns at incident scale. A runbook that nobody has rehearsed is a rumor; L48 rehearses one.

Worked example

A short staged outage (bad config through the GitOps path) pages the learner. Triage names the impact and start time; rollback-first restores service with the time noted; verification quotes the recovered user signals; the postmortem names the gate waved through (M14's lesson) plus the missing canary, with two owned actions. The whole arc fits one sitting because the shape was practiced, not invented.

Common wrong move

Skipping verification because the graph looks better. Partial recovery with a green dashboard is the incident relapsing on a delay. User-shaped evidence or it did not recover.