Volumes That Survive the Pod: PV, PVC and StorageClass

Why this matters

A database runs in a pod with its data on the container filesystem. The node reboots for patching, the pod reschedules elsewhere, and the data is gone: it lived on the old node's overlay disk, which nobody was asked to keep. The team learns the container rule the expensive way: local disk is scratch space. Everything worth keeping lives on a volume whose lifetime exceeds the pod, claimed explicitly and bound before the application ever writes.

Concepts

Three objects share the job. A PersistentVolume is a concrete piece of storage (a local path, a disk, a share) with capacity and access modes. A PersistentVolumeClaim is a request: this much space, this access mode, optionally this class. A StorageClass names a provisioner that creates volumes on demand, so claims bind without a human pre-creating disks. The pod mounts the bound claim at a path; when the pod dies elsewhere, the volume reattaches and the data returns. EmptyDir and container disks are the counterexamples: tied to the pod or node, gone on reschedule.

Pending claims have two classic causes. No provisioner for the requested class means nothing will ever bind it: the class name is misspelled, or no driver serves that class in this cluster. No capacity or no matching volume means the request exceeds what exists. The claim events name the cause; provisioner logs confirm it. Fix the class reference or the size, never paper over it with a hostPath that ties the pod to one node and lies about portability.

The honesty limit stays printed on every local lab: KinD and k3d ship a local-path provisioner that writes to one node's disk. It proves the claim-bind-mount machinery and the backup story, and it proves nothing about replication, failover or production durability. A lab that survives pod deletion demonstrates the mechanism; claiming it demonstrates disaster recovery is fiction. Lesson M07's backup thinking applies at full force: untested restore is not restore.

Worked example

A demo app writes a counter row to a claimed volume. The learner deletes the pod, watches the replacement mount the same claim, and reads the row back. Then a deliberately misspelled StorageClass stages a Pending claim; the events name the missing provisioner, the fix is one corrected word, and binding follows within seconds.

Common wrong move

Storing state on hostPath or container disk because the claim stayed Pending. That trades a legible binding error for silent data loss on the next reschedule. Fix the claim; never bypass it. The L29 lab makes the Pending diagnosis routine so the bypass never tempts.