JSON and API Handling

Why this matters

A deploy gate greps "status": "healthy" from an API response. The API adds a nested object, the grep still matches a substring in an error message, and an unhealthy release ships with a green gate. Text tools cannot read structure; they match shapes that recur by coincidence. Parsing JSON as JSON turns the same check into a field lookup with a default, and missing fields become explicit failures instead of accidental passes.

Concepts

jq is a filter language for JSON, not a prettier grep. .items[0].name walks structure; .items[]? tolerates absence; // supplies defaults (.replicas // 1); -e sets the exit code from the last output, which makes jq a real gate: jq -e '.ready == true' fails the script when the field is false or missing. select() filters arrays without loops, and -r emits raw strings so tokens never carry stray quotes into the next command. Quote the jq program in single quotes and pass shell values with --arg, never by interpolating variables into the filter, which reintroduces every injection lesson 1 taught.

Choose the tool by complexity. One field, one default, one check: Bash plus jq stays readable. Branching on nested shapes, retries with backoff, or building request bodies: Python with json and urllib or requests, where exceptions replace exit-code archaeology. The boundary is about three jq pipes: beyond that the shell version is write-only code that nobody will debug at 3 AM. Both must validate shape before content: check the version field, the expected top-level keys, then the values, in that order, and fail closed on anything unexpected.

Nulls and type changes are the production cases. A field that was a string becomes a list; a count becomes a string; an object gains a wrapper. Code that assumes one shape crashes or, worse, silently takes the wrong branch. Defaults handle nulls, explicit type checks (type == "array") handle changes, and a schema version field, when the API offers one, gets asserted first so new API generations fail loudly instead of parsing wrong.

Worked example

Gate on a rollout status endpoint, parsed properly:

status=$(curl -sf http://localhost:18090/rollout) || exit 1 ready=$(printf '%s' "$status" | jq -er '.ready // false') || exit 1 version=$(printf '%s' "$status" | jq -er '.version // "unknown"') [ "$ready" = "true" ] || { echo "not ready: version=$version"; exit 1; }

Expected reading: curl -f fails the script on HTTP errors before parsing starts; -e makes jq fail on missing or false fields; // false turns absent into an explicit not-ready instead of an empty string that might match something; the version rides along for the log line that the on-call engineer reads first. A grepped version of this gate passes on {"ready": "not yet"}, which is exactly the incident this lesson prevents.

The common wrong move

Parsing JSON with grep, sed or cut in production gates. It works on today's fixture and fails on the first pretty-printed response, reordered key, or nested match, and the failure mode is a false pass, never a loud error. Text tools read lines; JSON has no lines. The one exception is log triage by humans, where grep is exploration, not automation.

Lab and next step

Lab L09 serves timeouts, 429s and malformed JSON from a local fake API and requires handled cases for each, with tests. Next, lesson 4 bounds the waiting: timeouts, retries and the tests that prove both.