Upgrades That Roll Back

Why this matters

An upgrade changes the chart version and the values at once, traffic breaks, and recovery means reconstructing the previous combination from memory while the incident clock runs. Helm already recorded that combination: every upgrade stores a revision with its values and rendered manifests. The teams that recover in one command are not calmer, they are just reading the history the tool kept. Upgrade discipline is history discipline.

Concepts

A release is a named installation with numbered revisions. Upgrade creates a new revision from the current chart plus the supplied value chain; the previous revisions stay stored. Rollback points the release at a prior revision and reapplies it through the normal path, so the same hooks and verification apply. Status and history commands show which revision is live and what each revision changed; the L33 lab makes reading them the first reflex instead of the last resort.

Change one axis at a time. Upgrading the chart version and the values together doubles the suspect list when traffic breaks. The disciplined sequence is values first with the old chart where possible, then chart with fixed values, verifying traffic after each step. Atomic upgrades (automatic rollback on failure) suit small services; larger ones verify explicitly because automatic rollback can mask a failure worth understanding.

Failed releases need cleanup, not abandonment. A failed upgrade leaves a revision marked failed while the previous one may still serve; pending installs can wedge the release name. Know the uninstall and cleanup motions for your Helm version, and never leave a half-installed release for the next shift to discover. History limits keep the revision store small; keep enough revisions to cover the incident window, not every release since founding.

Worked example

The demo release upgrades to a chart version with a broken default. The learner verifies traffic failing, reads the history to name the last working revision, rolls back to it, and verifies traffic restored. Then the upgrade is repeated with the default corrected: same chain, new revision, traffic verified, history grown by two entries that read like a log.

Common wrong move

Deleting and reinstalling the release to fix a bad upgrade. That erases the revision history, drops the recorded values, and turns a one-command rollback into a reconstruction project. Move through revisions; never around them.