HTTP and Reverse Proxies

Why this matters

Users see 502 Bad Gateway and the proxy team gets paged, but the proxy is healthy: it is reporting, faithfully, that something behind it failed. Every minute spent restarting the proxy is a minute the real cause, an app that takes 40 seconds to answer through a 30-second timeout, keeps timing out. Proxy errors are witness statements. Read them as such and walk upstream.

Concepts

HTTP gives you three diagnostic gifts: the method, the status and the path, and all three are logged by default. The 5xx family splits cleanly. A 502 means the proxy got an invalid response: connection refused, reset, or garbage bytes, which usually means nothing listened or the app died mid-reply. A 503 means the upstream is reachable but has no capacity: queue full, pool exhausted, or an explicit maintenance signal. A 504 means the upstream took longer than the proxy's timeout: alive, but too slow. Same user-visible outage, three different teams, three different fixes. The status code is the routing slip.

A reverse proxy terminates the client connection and opens its own to the upstream, which creates two independent failure domains with two independent logs. The proxy log records what the client saw and how long the upstream took; the upstream log records what the app did. When they disagree in time (the proxy logged 30.0 seconds, the app logged 41), the gap is the diagnosis: timeout, not crash. Always pull both with matched timestamps, exactly as lesson 4 of M01 demands, and never declare the app healthy from the proxy log alone.

Configuration errors live in three places: the listen and server_name that decide which block answers, the proxy_pass that names the upstream, and the timeouts and buffer sizes that decide how patient the proxy is. A proxy_pass to the wrong port is lesson 2's ghost in new clothes; upstream timeouts set below the app's slowest legitimate response manufacture 504s on the busiest day of the quarter. Read the active config, not the one in version control: the reloaded file and the running process diverge more often than anyone admits.

Worked example

Checkout returns 502 since 14:05. Both sides, in order:

$ tail -5 /var/log/proxy/access.log | grep 14:05 10.0.0.8 "POST /checkout HTTP/1.1" 502 154 "-" 0.003 $ tail -20 /var/log/app/server.log | grep -iE "error|listen|bind" | head -3 14:04:58 FATAL: bind 127.0.0.1:3000: address already in use

Expected reading: the proxy answered in 3 milliseconds with 502, which rules out slowness and points at refused or broken upstream. The app log shows a bind conflict one minute before symptoms: a second copy of the app, probably from a deploy that started a new instance without stopping the old, holds the port while the intended instance exits. Fix: stop the duplicate, confirm one listener with ss -tlnp, watch 200s return. The proxy needed nothing, and touching it would have hidden the duplicate for another day. Lab L05 replays a 502/503 against a proxy-to-upstream pair until the three-code split is automatic.

The common wrong move

Raising proxy timeouts to make 504s stop. It works the way painkillers work on a fracture: dashboards go green while users wait 90 seconds per page and upstream queues grow until the whole tier falls together. Timeouts encode a contract about acceptable latency; breaking the contract at the proxy hides the breach from the only people who can fix it. Slow responses get fixed in the app or the contract gets renegotiated openly, with the new number written down and alerted on.

Lab and next step

Lab L05 hands you a proxy and an upstream with one broken link and requires the both-sides read before the fix. Next, lesson 4 secures the channel itself: TLS chains, hostnames and the expiry that pages teams every year.