Rollback plans that survive real incidents
Updated
One command, tested, timed: the rollback shape that works when judgment is at its worst.
Deploy plans get the rehearsal; rollback plans get the hope. Reverse that ratio.
The short answer
Every deploy ships with a one-command rollback, abort thresholds in writing, and a drill record with a timer. The owner of the call is named before go-live.
The rollback card
| Field | Example |
|---|---|
| Command | Single documented command restoring the previous signed release |
| Abort thresholds | Error rate above agreed line for 10 minutes, or outcome metric regressing |
| Timebox | Fix-forward attempt capped at 30 minutes, then rollback |
| Owner of the call | Named engineer holding the runbook during the window |
| Verification | Health checks green for 15 minutes plus the key business metric steady |
| Drill record | Date, duration, gaps found and fixed |
Worked example: the Tuesday rollback
A fictional deploy (fictional) trips the error threshold nine minutes into the window. The named engineer calls rollback at minute ten, no meeting held. The previous signed image serves in six minutes; verification passes by minute twenty. The readout entry carries cause, fix and prevention. Boring is the compliment. The constraints post sets the window; the runbook post holds the card.
Checklist: rollback readiness
- Rollback tested in staging for this exact release.
- Abort thresholds and timebox in the brief, signed.
- Owner of the call named and holding the runbook.
- Drill record exists with a date, not a memory.
Related reading
Straight answers
Frequently asked questions
What makes a rollback plan real?
One command, tested in staging, timed, with named abort thresholds decided before the deploy. Plans invented during incidents are wishes.
When do you roll back versus fix forward?
The brief decides before shipping: threshold, timebox, and owner of the call. Without that line, every incident becomes an argument.
How often should rollback be tested?
Every slice that changes production behavior, plus a quarterly drill on the oldest path. Untested paths fail exactly when needed.