Upgrade a kubeadm Cluster Step by Step

Why this matters

Kubernetes supports skew of one minor version between control plane and kubelets, and kubeadm enforces the path: upgrade the control plane first, then the workers, never skipping a minor version. The exam tests whether you respect that order under time pressure, because every shortcut produces a cluster that looks fine until the first real workload lands on it.

Concepts

Start with a snapshot; upgrades touch etcd data formats and there is no undo button without one. On the first control-plane node, upgrade kubeadm itself, run kubeadm upgrade plan to see the path and required manual steps, then kubeadm upgrade apply to the target version. Remaining control-plane nodes get kubeadm upgrade node. Then each worker in turn: kubectl drain (which evicts pods respectfully, honoring PodDisruptionBudgets unless forced), upgrade kubeadm plus kubelet plus kubectl on the host, kubeadm upgrade node, and kubectl uncordon to accept pods again.

Drain deserves respect: DaemonSet pods and mirror pods are not evicted, and a tight disruption budget will block the drain until you decide deliberately. --ignore-daemonsets plus --delete-emptydir-data are the usual flags, and each one should be a conscious choice, not muscle memory.

Worked example

Control plane 1.35 to 1.36, then one worker:

# snapshot first, always
etcdctl snapshot save /tmp/pre-upgrade.db  # with the cert flags
# control plane node one
sudo apt-mark unhold kubeadm && sudo apt install -y kubeadm=1.36.x
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.36.x
sudo apt install -y kubelet=1.36.x kubectl=1.36.x
sudo systemctl restart kubelet
kubectl get nodes  # control plane Ready at 1.36
# each worker in turn
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
sudo apt install -y kubeadm=1.36.x kubelet=1.36.x kubectl=1.36.x
sudo kubeadm upgrade node
sudo systemctl restart kubelet
kubectl uncordon worker-1
kubectl get nodes  # all Ready, all 1.36

Exam notes

Upgrading kubelet before the control plane reaches the target minor is the classic order violation. Draining the node you are standing on (the control plane via the same terminal still works, but draining all workers at once) removes all capacity: one node at a time. And re-cordon nothing: uncordon is part of the step, and a forgotten cordon shows up later as a scheduling mystery.