Timeouts, Retries and Tests

Why this matters

A deploy waits on a health endpoint with no timeout. The endpoint hangs, the deploy hangs, the pipeline slot holds for six hours, and three teams queue behind it. Nobody failed; everybody waited. Unbounded waits convert one slow dependency into a frozen organization. Timeouts are not pessimism; they are the mechanism that turns hangs into errors, and errors can be handled.

Concepts

Every external call needs three numbers: how long to wait for a connection, how long to wait for the answer, and how long the whole operation may take. timeout(1) enforces the total from outside; curl --connect-timeout and --max-time enforce the parts; Python's timeout arguments do the same in code. Pick numbers from the caller's budget, not the dependency's optimism: a gate that must finish in two minutes cannot grant one call a five-minute timeout. When the timeout fires, log which bound tripped; undifferentiated timeouts are undebuggable.

Retry only what is safe to repeat: reads, idempotent writes, and operations with idempotency keys. Never retry non-idempotent actions (charge, send, create-without-key) without a mechanism that makes the second attempt harmless. Back off exponentially with jitter so ten callers do not hammer a recovering service in lockstep, cap attempts at a small number like three, and budget the total: three attempts with 30-second timeouts is a 90-second worst case the caller must afford. A retry policy without a budget is a hope.

Test the failure path with fixtures, not staging. A local fake endpoint that sleeps, returns 429, drops connections and serves malformed JSON lets the test suite prove every branch deterministically. Assert the observable contract: exit codes, retry counts, elapsed time under the budget, and the log line the on-call engineer will grep. Happy-path tests prove the script works; failure-path tests prove it is safe, and only the second kind matters at 3 AM.

Worked example

Fetch with bounds and a budgeted retry:

#!/usr/bin/env bash set -euo pipefail url="http://localhost:18090/ready" for attempt in 1 2 3; do if curl -sf --connect-timeout 3 --max-time 10 "$url" | jq -e '.ready == true' >/dev/null; then echo "ready (attempt $attempt)"; exit 0 fi sleep $((attempt * 2)) done echo "not ready after 3 attempts" >&2; exit 1

Expected reading: connect and total timeouts bound each attempt (3s/10s), jq -e gates on content rather than HTTP success alone, sleeps of 2 then 4 seconds separate attempts, and the worst case is bounded near 40 seconds plus overhead. Three attempts is a policy: enough for transient blips, few enough that a hard-down dependency surfaces in under a minute. Lab L09 builds the fake API this script deserves and demands green tests for the bad cases.

The common wrong move

Retry loops without limits: while true with a sleep, or recursion with no base case. They turn a five-minute upstream outage into hours of thundering load that delays the upstream's own recovery, then get blamed on the network instead of the loop. Every retry needs three visible constants: max attempts, backoff base, total budget. If any is missing, the loop is unbounded no matter what the comment says.

Lab and next step

Lab L09 provides the fake API (timeout, 429, bad JSON) and requires a script plus tests covering every bad input; real API keys never appear. From here the path continues to M04, where automation meets collaboration: the same discipline applied to shared history in Git.