Config and Secrets Without Leaking Them

Why this matters

A database password lives in a values file, committed two years ago, copied into every environment since. Rotation means finding every copy, and the git history keeps a copy rotation cannot touch. When the password leaks through a screenshot in a demo, the incident is not one password but every system that ever shared it. Secrets in the repo are not a risk, they are a breach with a delayed announcement. The only safe count of secret bytes in version control is zero.

Concepts

ConfigMaps carry non-sensitive configuration: feature flags, endpoint names, tuning knobs. They live in the repo as manifests or chart values because reading them harms nothing. Secrets carry credentials, tokens and keys; in etcd they sit base64-wrapped (encoded, not encrypted, unless encryption at rest is enabled), mounted as files or environment variables by reference. The pod spec names the Secret object; the bytes arrive from the cluster, never from the commit.

The lifecycle has four motions. Create the secret outside the repo (the cluster CLI, an operator, a vault agent). Mount it by reference in the workload. Roll it on schedule: new value alongside the old where the application supports overlap, restart or reload to pick it up, verify before revoking the old. Revoke means disabling the old credential at the source, not just deleting the object; a deleted Secret whose password still works at the database revoked nothing.

Leak response is fixed choreography, decided before it is needed: revoke the credential at the source first, rotate to a fresh value through the normal lifecycle, then clean the exposure (purge history where possible, accepting that public history cannot be un-read). Debugging order inverts: stop the bleeding before studying the wound.

Worked example

The demo chart references a database Secret that does not exist in the repo: the values file holds only the secret name, the manifest mounts by reference, and the learner creates the Secret locally with a placeholder value. Rotation is rehearsed: second value added, pods reloaded, old value revoked, traffic verified throughout. The repo never sees either value.

Common wrong move

Encrypting secrets into the repo and calling the problem solved without a key story. Encrypted blobs still need keys, rotation, revocation and audit; tools that do this well exist, but adopting the blob without the machinery is decoration. Start with references and a practiced lifecycle; add machinery when the scale demands it, not before.