Recover Nodes, Network Faults and Full Disks

Why this matters

Node pressure kills indiscriminately: DiskPressure evicts, MemoryPressure starves, PIDPressure chokes new processes. Full disks also block image pulls, which disguises storage failure as registry failure. The exam clusters these because they share one proving ground: node conditions plus exec-level tests.

Concepts

kubectl describe node Conditions plus Allocated resources tell the story: Ready False with KubeletNotReady means the agent is down; DiskPressure or MemoryPressure True names the shortage; PIDPressure True caps process creation. kubectl top node quantifies live use when metrics-server runs. Full inodes (millions of tiny files, runaway logs) fill df -i while df -h looks fine: check both, always.

Clearance order: copy evidence first (the logs you need for the post-mortem), then prune images (crictl rmi --prune or kubelet garbage collection), rotate logs, and only then delete data. Never clear before copying: destroyed evidence ends investigations.

Network splits by layer: pod-to-pod on one node (CNI plus kubelet), pod-to-pod across nodes (fabric plus kube-proxy), pod-to-outside (egress plus DNS). Same-node success with cross-node failure indicts the fabric; all-pods-one-node failure indicts the node; single-pod failure indicts the pod. Two commands (same-node curl, cross-node curl) place every fault.

Worked example

Node worker-2 NotReady, pods mysterious, drill in order:

kubectl describe node worker-2 | grep -A8 Conditions
# DiskPressure True, out of inodes on /var/log
kubectl debug node/worker-2 -it --image=busybox -- df -i /host/var/log
# 100% inodes: one container logging megabytes of tiny lines
# evidence first, then clearance
kubectl logs -l app=noisy --since=2h > /tmp/noisy.log
# rotate plus prune, then watch the condition clear
kubectl uncordon worker-2   # after Ready True returns
kubectl get pods -o wide | grep worker-2   # workloads land again

Cordon during repair keeps the scheduler from re-filling a sick node; uncordon only after conditions clear, verified, not hoped.

Exam notes

Inode exhaustion hides behind healthy-looking disk percentages: always pair df -h with df -i. Kubelet stopped means exec and logs also fail for that node's pods: use node-level debug or SSH instead. And cordon is not drain: cordoned nodes keep running pods, which is the safe repair posture.