Back Up and Restore etcd

Why this matters

Deployments, services, secrets, persistent volume claims: all of it lives in etcd, and the API server is stateless in front of it. The CKA tests this directly because it is the one backup that decides whether an incident ends in recovery or in rebuilding from YAML you may not have. Snapshots are small, fast, and easy to automate, which makes skipping them inexcusable.

Concepts

etcdctl snapshot save talks to a running etcd member over TLS, so it needs the cluster certificates: the CA plus the apiserver-etcd-client certificate and key that kubeadm already placed under /etc/kubernetes/pki. Point etcdctl at the local member endpoint (https://127.0.0.1:2379) with ETCDCTL_API=3 and the three certificate flags, and the snapshot lands as one file. Check it with snapshot status before trusting it.

Restore is the reverse with one catch: you cannot restore over a running member's live directory. etcdctl snapshot restore writes into a fresh directory, then you restart the member (or the static-pod manifest) pointed at the new directory. For stacked etcd on kubeadm, the practical exam path is restoring to a new directory, moving it into place for the member, and restarting the etcd static pod. The API server then serves the restored state, and kubectl get across a few resource types proves it.

Automation is the difference between having backups and having restorable backups: a cron entry plus an off-node copy beats a perfect manual procedure nobody runs at 3 AM.

Worked example

On a control-plane node, certificates in hand:

export ETCDCTL_API=3
alias e='etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key'
e snapshot save /tmp/etcd-snapshot.db
e --write-out=table snapshot status /tmp/etcd-snapshot.db

The status table must show a hash plus revision. To restore: e snapshot restore /tmp/etcd-snapshot.db --data-dir /var/lib/etcd-restored, move the member data into place, restart the etcd static pod, then confirm with kubectl get pods -A and a spot check of one deployment's replica count.

Exam notes

Snapshot against the wrong endpoint or without certificates fails loudly; the fix is the three flags, not the network. Restoring while the member still runs on the old directory is the classic trap: the new data never loads. And always verify after restore by reading through the API, never by staring at etcd logs alone.