Runbooks the next person can actually run
Updated
A runbook is tested like code: a new person follows it during calm hours and every gap gets fixed.
Most runbooks describe the system. Useful runbooks prescribe the repair. The test is simple: hand it to the next person on a quiet afternoon. If they can run it without calling you, it works. Every call is a defect.
Prescribe, do not describe
Trigger first. The symptoms that say this runbook, not another. Operators pick runbooks under stress; the first lines must disambiguate.
Commands with expected outputs. Every step is copy-pasteable and says what success looks like. A command without an expected output is a guess with formatting.
Rollback at the end. What to do when the runbook does not work: who to call, what to revert, where the next runbook lives.
Worked example: a fictional disk-pressure runbook
The context below is fictional. Fictional team ParcelTrack (fictional) writes a disk-pressure runbook after their first inode incident. First version describes filesystems beautifully and helps nobody.
The rewrite lists the alert symptom, the two meter checks, the identify-then-delete order, and the expected recovered output. A new hire runs it unassisted on a fixture disk during work hours; three questions become three fixes. Next real incident: resolved without waking the author.
Checklist: a runbook that runs
- Trigger symptoms in the first lines.
- Every command copyable with expected output.
- Rollback path when steps fail.
- Tested by a non-author in calm hours.
Related reading
- Hands on: Rewrite the Runbook and Hand Over the Run.
- The disk companion: Disk full is two problems.
Straight answers
Frequently asked questions
What belongs in every runbook?
Symptoms that trigger it, exact commands in order, expected output after each, and the rollback path. Anything else is appendix.
How do I know it works?
Someone who did not write it follows it in calm hours. Every question they ask is a runbook bug to fix.
How long should one be?
One incident, one page. A runbook covering everything covers nothing at 3 AM.