Restore versus Rollback: Different Tools, Different Moments

Why this matters

A bad migration corrupts records and the team rolls back the code three times, each rollback perfectly restoring the application that reads the corrupted data perfectly. Hours pass before someone asks whether the data is the patient. Rollback was the reflex; restore was the cure, and the reflex burned the incident window. The decision between them is the highest-leverage minute of a data incident, and it must come from evidence, not from whichever runbook opened first.

Concepts

The decision rule fits one paragraph. If the data is good and the code is bad (new release serves errors, old data reads fine), roll back the code (M14's machinery). If the data is bad (corruption, bad migration, deleted rows), restore the data from the last known good inside RPO, whatever the code version. If both are bad, restore data first (to the point the old code understands), then roll code to match; reversing the order writes new-code shapes into restored data or old-code reads against new shapes.

Evidence decides in minutes. Serve reads from a copy: does the old code read clean (code fault), does every version read corrupt (data fault)? Check the deploy and data timelines for coincidence: failure starting with a release implicates code, with a migration or job implicates data. The check is a query and a timeline, not a debate; time-box it, because the decision delayed is darkness extended.

Sequencing matters when both fire. Restore to a separate target first, verify the data there, then cut traffic to the restored copy with the matching code version. Never restore over the only copy before verifying; never roll code during an unverified restore. Each step leaves a way back until the next step proves itself, the same checkpoint discipline as migrations.

Worked example

A fixture incident corrupts rows through a staged bad job while a new release also serves. The learner tests reads per version, finds data fault plus code fault, restores data to a separate target and verifies, then rolls code to match, verifying each step. The record shows the evidence, the order, and the reason each step preceded the next.

Common wrong move

Restoring over production without verifying on a separate target first. A bad restore destroys the evidence and the fallback together; the second attempt starts from less than the first. Separate target, verification, then cutover, every time.