Rotate secrets without an outage: narrow users first
Updated
Secret rotation is safe when every reader is known and narrow: inventory readers, add the new secret, remove the old.
Secret rotation causes outages for one reason: someone changes the secret before finding every reader. The safe order is inventory, dual-run, retire. Skip the inventory and the rotation becomes a treasure hunt at midnight.
The safe order
Map every reader. List each process, job and human that touches the secret, with where it reads from. Narrow service users make this list short; wide shared files make it endless.
Run both secrets. Issue the new secret while the old still works. Move readers one by one, verifying each. The overlap window is the safety net.
Retire and verify. Disable the old secret, watch for failures, then delete. A rotation that ends with both secrets alive is unfinished.
Worked example: a fictional fixture secret
The context below is fictional. Fictional service ParcelTrack (fictional) reads a fixture token from a file shared by two services and one nightly job. Past rotations broke the nightly job twice.
This time the team narrows first: each service gets its own service user and path, the job is listed as a reader. New token issued, services moved and verified one by one, old token disabled after a quiet night, failures zero. The next rotation reuses the same reader list.
Checklist: a rotation without drama
- Every reader named with its read path.
- New secret live while old still works.
- Readers moved one by one with verification.
- Old secret disabled, watched, then deleted.
Related reading
- Hands on: Narrow Service User and Rotate Fixture Secret.
- The permissions companion: Permissions broke my service.
Straight answers
Frequently asked questions
What breaks most rotations?
An unknown reader. A cron job, a debug script or a second service nobody listed still uses the old secret.
How long should both secrets work?
Long enough to cover the slowest reader refresh plus one verification round. Hours, not seconds.
Where should secrets live?
One managed place with access logs, never in code, chat or screenshots. Narrow readers, not wide files.