Idempotent Scripts

Why this matters

A setup script adds the same firewall rule, appends the same line to the same config, and creates the same user every time it runs. The first run is perfect; the third produces duplicates that break parsing, tripled rules that slow every packet, and a user-creation error that fails the deploy. Reruns are not accidents. Pipelines retry, humans re-click, configuration tools reconcile. A script that is unsafe to rerun is unsafe to use.

Concepts

An operation is idempotent when running it twice changes nothing the second time. The pattern is check-then-change with the check reading real state, not a flag file the script wrote itself: grep -q the line before appending it, test -d before mkdir -p (which is already idempotent), query the user list before useradd. Flag files lie after manual changes; state checks do not.

Convergence beats steps. Instead of do A then B, declare the end state and move toward it: the line is present exactly once, the service is running, the package is installed. Package managers and mkdir -p already think this way; your script should too. When full convergence is too much, make each step independently rerunnable and order them so a rerun after any failure resumes instead of duplicating.

Separate checking from changing so --check or --dry-run modes stay truthful. Structure every action as a function that first reports what it would do, then does it only when not in dry-run. A dry run that cannot show its plan is decoration; a plan that differs from the real run is worse than none. Test reruns the way you test first runs: run, run again, diff the system state, assert empty diff.

Worked example

Appending a config line, made safe:

#!/usr/bin/env bash set -euo pipefail line="max_connections = 200" file="/tmp/dlab-svc.conf" dry_run="${1:-}" if grep -qxF "$line" "$file" 2>/dev/null; then echo "present: nothing to do" elif [ "$dry_run" = "--check" ]; then echo "would append: $line" else printf '%s\n' "$line" >> "$file" echo "appended: $line" fi

Expected reading: grep -qxF matches the exact full line (F for fixed string, x for whole line, so max_connections = 20 does not match 200), missing files count as absent rather than erroring, --check reports without writing, and a second run prints present and changes nothing. Lab L08 scales this pattern to a full setup script with tests proving the second run is a no-op.

The common wrong move

Guarding with a marker file the script creates itself. It makes the second run a no-op even when someone deleted the actual change, edited it by hand, or the first run half-failed after writing the marker. Markers record that the script ran, not that the state holds. Check the state, never the marker.

Lab and next step

Lab L08 requires a setup script whose second run changes nothing, proven by state diffs, plus a --check mode. Next, lesson 3 handles the data scripts consume: JSON and APIs.