API Retries and Idempotency for Customer Integrations: The Exact Patterns

The short answer

Networks retry; systems duplicate; customers notice double charges. Two mechanisms make integrations trustworthy: bounded retries (so transient failures heal) and idempotency (so retries can't corrupt). Everything else - DLQs, reconciliation - is support structure for when bounded isn't enough.

Retries that work

for attempt in 1..5:
    try: return call()
    except Retryable:
        if attempt == 5: dead_letter(e); return
        sleep(min(cap, base * 2**attempt) + jitter())

Rules:

• Retry only what's retryable: timeouts, 429s (honor Retry-After), 5xx. Never 4xx validation errors - that's a bug in your payload.
• Jitter is mandatory. Synchronized retries thundering-herd the recovering upstream.
• Budget the total: cap attempts and wall-clock time; a job that retries forever is an outage generator.
• Idempotency key on every write (header or field) so the upstream can dedupe.

Idempotency that works

1. Natural key first: if the domain has one (order ID + version), use it with a DB unique constraint. Constraints beat cleverness.
2. Key = deterministic hash of the business intent when no natural key exists (hash of normalized payload, not timestamp!).
3. Store outcomes, not just attempts: a processed_keys table returning the first result makes replays harmless.
4. Consumers must tolerate replays: at-least-once delivery means your handler runs twice; design for the second run being a no-op.

The failure table your write-up needs

That table is the artifact separating "I used a retry library" from "I designed for failure" - and it's exactly what the ERP portfolio project builds.

Continue: Production checklist · ERP integration project