DevOps Professional · Module 7: Incident leadership and handover
Evidence, Communication and the Postmortem That Teaches
Learning starts during the incident with preserved evidence and honest updates, and it lands in a postmortem that proves its claims. False causality gets corrected with measurements; every cause gets an owned, dated action.
10 min reading
Objectives
- Collect evidence during the incident, not after it rotates away
- Communicate what is known, what is next and when the next update lands
- Fix postmortems that assert false causality with measurable evidence
- Turn causes into owned actions with dates, not into general advice
Why this matters
A postmortem concludes the deploy caused the outage because it preceded it; the actual cause was a quota exhaustion the deploy merely revealed. The action (slower deploys) misses the fix (quota guard), and the incident recurs on schedule. Post hoc reasoning is the default failure mode of incident learning: sequence mistaken for cause, a plausible story unchallenged by measurement. The L68 lab fixes a postmortem that asserts false causality with measurable evidence; this lesson teaches the correction as a habit.
Concepts
Evidence collection runs parallel to mitigation. Freeze the timeline (entries with timestamps as they happen), snapshot the failing state (logs, metrics windows, configs) before rotation and rollback erase it, record decisions with their reasons in the moment. After-the-fact evidence is reconstruction; reconstruction edits itself toward the plausible. The scribe role exists so evidence survives the pressure; without a scribe, the timeline is the first casualty.
Communication has a shape: impact (who feels what), known (facts only, marked as facts), next (the action in flight with its owner), update time (the next check-in, kept). Unknowns are stated as unknowns; guessed causes announced as fact become the false story the postmortem must later retract. Cadence beats completeness: short updates on the clock outperform one perfect update after the fact.
Postmortem correction is mechanical. Every causal claim faces its measurement: deploy caused it (show the mechanism, not the timestamp), quota caused it (show exhaustion coinciding with failure and recovery on relief). Competing claims are tested, not voted on. Causes stay plural and systemic; the action list converts each into an owned fix with a date (guard, alert, runbook, capacity), and advice-shaped items (be more careful) are rewritten or removed.
Worked example
A fixture postmortem blames the release with timestamp evidence only. The learner reopens it: quota metrics show exhaustion before the deploy finished, recovery follows relief not rollback, and the corrected postmortem names the quota guard plus the misleading-timing note as actions. Same incident, evidence-shaped learning.
Common wrong move
Writing the postmortem from memory a week later. Memory keeps the plausible story and drops the inconvenient measurement. Timeline live, evidence frozen, postmortem within days, or the learning is fiction.
Quick check
An optional 4-question self-check. Answers never leave your device, are not stored, and never count toward any assessment.
Lesson feedback
No published feedback yet.
Log in and complete the lesson to leave feedback.
Exercise
Take a fixture postmortem asserting false causality, challenge each claim with measurements, and rewrite it with systemic causes plus owned dated actions.
Pass criteria
The record shows each challenged claim with its measurement, the corrected causes, and the owned dated actions.