Debug with Exec, Logs, Describe and Events

Why this matters

Debugging without an order revisits the same layers and misses the telling one. The four commands cover four viewpoints that never overlap: inside the container, the app's output, the object's spec plus status, and the cluster's event stream. The CKAD troubleshooting tasks are all four-command problems wearing different symptoms.

Concepts

kubectl exec -it <pod> -- <cmd> runs inside the container's namespaces: test DNS with nslookup, read env with printenv, check files the app claims are missing, curl the dependency the app cannot reach. If the image lacks a shell, ephemeral debug containers (kubectl debug) attach tooling without rebuilding anything.

kubectl logs (plus previous, plus fleet flags from the last lesson) is the testimony step. kubectl describe contrasts intent with reality: requested versus actual resources, mounted versus missing volumes, probe definitions versus failure counts, node assignment versus node conditions. Events close the loop with timestamps: pulls, scheduling decisions, kills, evictions.

Escalation follows the stack. Pod sick but node fine means app or pod spec. All pods on one node sick means the node (conditions, pressure, kubelet). Everything sick means the control plane or network fabric. Each level has one proving command before moving up.

Worked example

Checkout pods failing, drill in order, no skipping:

kubectl exec deploy/checkout -- printenv | grep -i db_     # config present?
kubectl exec deploy/checkout -- nslookup ledger            # DNS from inside?
kubectl logs -l app=checkout --since=15m | grep -iE 'error|exception|timeout'
kubectl describe pod -l app=checkout | grep -A5 Events
kubectl get events --sort-by=.lastTimestamp | tail -20
# app cause? fix the app. platform cause? node next:
kubectl describe node <node> | grep -A10 Conditions

The drill ends where evidence concentrates: env missing means config, NXDOMAIN means DNS, shared exception means code, FailedMount means volumes, node pressure means the node. Never fix a layer the evidence has not indicted.

Exam notes

Multi-container pods need -c on exec and logs or you interrogate the wrong container. kubectl debug with an ephemeral image rescues distroless containers that ship no shell. And describe output is long: pipe to grep for Events, Conditions and the failing probe instead of reading top to bottom.