API failures in the wild: timeouts, 429s and bad JSON
Updated
Three API failure shapes every operator meets: slow answers, rate limits and malformed bodies, each with its own response.
APIs fail in three shapes, and each punishes a different laziness. Timeouts punish missing budgets. 429s punish blind retries. Malformed bodies punish trusting the status code. Learn the three responses once and every integration gets calmer.
Budget, backoff, validate
Timeouts get budgets. Every outbound call carries a deadline derived from what the caller can afford. A timeout converts a freeze into an error you can handle, retry or report.
429s get backoff with jitter. The server asks for slowness; answer with exponential waits plus randomness so a fleet does not retry in lockstep. Cap total retries: a budget, not a loop.
Bodies get validated. Check shape before use: required fields, types, error envelopes. Log the offending body (minus secrets) so the next failure is diagnosed in minutes.
Worked example: a fictional status page integration
The context below is fictional. Fictional monitor ParcelWatch (fictional) polls a carrier API with no timeout, instant retries and blind JSON parsing. One slow carrier evening freezes all checks; then a 429 storm extends itself; then a changed field name crashes the parser at 2 AM.
The rewrite sets a ten second budget, backoff with jitter capped at three tries, and shape validation with the raw body in the error log. Next carrier incident: bounded errors, calm retries, one log line naming the changed field.
Decision table: API failure responses
| Signal | Response | Never |
|---|---|---|
| Slow answer | Timeout budget, then handle | Wait forever |
| 429 | Backoff with jitter, capped | Retry hot in a loop |
| 200 with wrong shape | Validate, log body, alert | Trust status 200 blindly |
Related reading
- Hands on: Fake API Timeout, 429 and Bad JSON.
- DevOps Foundations Module 2 covers HTTP clients.
Straight answers
Frequently asked questions
Retry on 429?
Yes, with backoff and jitter, and only within a retry budget. Retrying a rate limit at full speed is how you extend your own ban.
What timeout should I set?
A budget from the caller side: how long the user waits, minus your own margin. No timeout means one slow dependency freezes everything.
Validate JSON responses?
Yes, at the boundary. A 200 with an unexpected shape breaks more code than a clean 500.