Free lesson · CKA Prep Program · Module 6: Troubleshooting

Recover Control-Plane Components and etcd

When the API itself is sick, kubectl is the patient too. This lesson works from the node upward: static manifests, component logs, restarts, and etcd restore as the last resort with the first priority.

13 min reading

Objectives

List the static-pod manifests and read each component's logs
Restart a dead apiserver, scheduler or controller-manager safely
Restore etcd from snapshot when the data is gone, not just the pod
Prove the recovery through the API with growing checks

Why this matters

Control-plane failure removes your instruments: no kubectl, no events, no drill order from earlier lessons. Recovery runs on node access plus static-pod knowledge, and the exam tests exactly this inversion because it separates operators who understand the boot chain from those who only drive the API.

Concepts

kubeadm runs the control plane as static pods from manifests in /etc/kubernetes/manifests: etcd, kube-apiserver, kube-controller-manager, kube-scheduler, one file each. Kubelet watches the directory and restarts anything whose file is present and valid: touching a manifest restarts the component, removing it stops the component. Logs live in the container runtime (crictl logs) since kubectl may be down.

Dead scheduler or controller-manager means workloads freeze but serving continues: restart via manifest touch, watch the new container start. Dead apiserver means total API blindness: check its manifest, its certificates (expiry kills apiservers silently on anniversaries), its etcd reachability. Dead etcd member in a stacked trio means quorum watch: two of three must stay healthy, and the restore drill from the etcd lesson rebuilds the lost member from snapshot.

Worked example

API blind at 03:12, node access works, drill from the metal up:

# 1. are the manifests present and valid?
ls -l /etc/kubernetes/manifests/
# 2. is kubelet running to enforce them?
systemctl status kubelet
# 3. what do the dead components say?
crictl ps -a | grep -E 'apiserver|etcd|scheduler'
crictl logs <apiserver-id> | tail -30
# "x509: certificate has expired" -> renew: kubeadm certs renew all
sudo kubeadm certs renew all
sudo touch /etc/kubernetes/manifests/kube-apiserver.yaml  # restart
# 4. etcd member down? quorum first, snapshot restore per the etcd lesson
ETCDCTL_API=3 etcdctl endpoint health --cluster  # with cert flags
# 5. growing proof: kubectl get --raw=/healthz, then nodes, then pods

Certificate expiry is the most common silent killer: calendar it, monitor it, renew with the command instead of hand-editing. Proof grows from raw health to full object reads; each level passed permanently.

Exam notes

Deleting a static manifest stops the component: never clean that directory by enthusiasm. Restarting apiserver drops in-flight kubectl connections: expected, reconnect. And etcd restore without quorum awareness can split-brain a trio: one member at a time, health-checked between each.

Quick check

A 4-question self-check. Your answers stay on your device, are never stored or sent. Only a passing score (for example 3 of 4) is saved, and only to unlock lesson progress: at least 3 correct is required before a lesson counts as complete. It never counts toward any certification assessment.

Lesson feedback

No published feedback yet.

Exercise

A fictional stacked cluster loses one etcd member plus the scheduler simultaneously. Write the recovery order with the proving command after each step, and state which component you fix first with why.

Pass criteria

Order is manifests, kubelet, logs, scheduler restart, etcd quorum plus restore, growing API proof; scheduler-first (or justified order) with the serving-versus-state reasoning.