AI Application Rollback Runbook: Design It Before Launch Day

The short answer

Classic rollback reverts code; AI rollback must also revert behavior - prompts, model versions, retrieval indexes. Design the levers before launch, write the runbook as incident steps, and rehearse it. A rollback you haven't rehearsed is a theory.

The four levers

1. Kill switch (feature flag): route traffic to the non-AI fallback path without deploy. Test the flag path in staging with production-like traffic shapes.
2. Behavior pinning: prompts and model versions are config with version history; rollback = repoint to last-good pair. Never let a prompt be "just edited in prod."
3. Index rollback: if you rebuilt the retrieval index, keep the previous one addressable; index reindex is a deployment with its own rollback.
4. Data reversibility: AI-written records must be traceable (job ID, version) so misbehaving outputs can be identified and corrected/requeued.

The runbook skeleton

INCIDENT CLASS: AI quality degradation / cost spike / outage
S1 DETECT   alert: error rate, latency, quality sample, $/hour
S2 STABILIZE  flag off → fallback path serves (define it!)
S3 DIAGNOSE  which layer: model API? retrieval? prompt change? data drift?
S4 RECOVER   repoint versions / reindex / scale; verify with eval subset
S5 REVIEW    quality incident postmortem within 48h

The drill

Quarterly, in staging: break the model API (chaos proxy), watch detection fire, flip the flag, measure fallback behavior, time each step. Fix what took >5 minutes. Record the drill like you record deploys - see deployment documentation.

What "misbehaving" means here

Not just errors: confidence drift, cost creep, or groundedness decay all trigger the same runbook. Your evaluation suite is the detector - without it, the runbook has no trigger and the customer's users are your monitor.

Continue: POC-to-production checklist · Production delivery