Runbooks Someone Else Can Run

Why this matters

The expert is on leave and the database fills its disk; the runbook says free some space, which the on-call engineer reads as delete the oldest archive, which turns out to be the only backup. Vague verbs destroy systems. Procedures carry exact commands, expected outputs, and the decision tree for when output differs; anything less is advice wearing a uniform.

Concepts

Each procedure has one trigger (which alert, which symptom), preconditions (what must be true before step one), numbered steps with exact commands and expected output quoted, decision points (if output differs, go to step X or escalate), and a done condition plus a rollback pointer. Copy-pasteable commands beat descriptions; expected outputs let a stranger confirm each step instead of hoping. Review procedures against the current system, not memory: every command gets re-run in a drill before it earns trust.

The change log is the incident's memory. Every production change records who, what, when, why, and how to reverse it, in one place the responder can read in a minute. Most incidents are solved by finding the recent change, and the log makes that search seconds instead of archaeology. No log entry, no change: the rule is social before it is tooling, enforced in review.

Handover is the test. Give the runbook to someone who has never run it, watch silently, and fix every place they hesitate, guess, or improvise. Their confusion is the document's defect, not their failure. Then run the repaired procedure for real under supervision. A runbook that survives one handover is worth ten expert-only pages.

Escalation bounds the procedure. Each runbook names its stop conditions: which outputs mean stop and page the expert, which blast radius requires the incident lead, which customer impact triggers status communication. Running past the stop line turns a contained fault into an outage with company. Knowing when to stop is part of the procedure, written at the top, not learned at the bottom of an incident.

Worked example

A broken runbook for disk-full-on-database says restart the service and clear logs. The handover tester deletes the active log the database is writing to, and the database keeps the file handle while disk stays full. Expected reading: restart was never the fix (the writer holds handles), and clear logs named no file, no check, no expected output. The rewrite: check disk by mount, identify the growing file with open handles listed, rotate through the database's own command, verify space plus service health, with the stop condition (replication lag above X means escalate). Verify by the tester running it clean on the first try. Lab L21 requires exactly this: rewrite, handover, repair, run.

The common wrong move

Screenshots of consoles and tribal one-liners as documentation. Consoles move, commands rot, and nobody can copy a screenshot at 3am. Text commands in version control, reviewed with the system changes they describe, drilled on schedule. Documentation that cannot be executed is commentary; runbooks are programs for tired humans.

Lab and next step

Lab L21 takes a broken runbook, rewrites it to the stranger standard, hands it to another person, and runs it. From here the path continues to M08: the security half of operations, least privilege, secrets and findings.