FDE Foundations · Module 5: Data & Integration
Webhooks and Event Ingestion
Webhook ingestion rule: verify, acknowledge fast, process later. Everything else (retries, dedup, replay) follows from treating delivery as at-least-once.
11 min reading
Objectives
- Receive webhooks with verification and fast 2xx acks
- Process asynchronously from a durable queue
- Design replay and deduplication for at-least-once delivery
Receive safely
Verify the signature on every delivery (HMAC with a shared secret per the provider's scheme) before trusting the payload. Acknowledge with a 2xx quickly, before doing real work; slow handlers cause provider-side timeouts, which cause duplicate deliveries, which cause your duplicate problem.
Process asynchronously
Enqueue the event and return. A worker processes the queue with the idempotency and retry patterns from m04: classify errors, backoff, dead-letter the permanent ones. This split keeps ingestion resilient to your own deploys and to downstream outages.
At-least-once means dedup
Providers deliver at-least-once. Deduplicate on the event ID with a storage-backed check, not an in-memory set: the point of dedup is surviving restarts. Log duplicate deliveries at debug level so you can prove dedup works.
Replay is a requirement
Design a replay path from day one: pick a time range or a dead-letter batch, re-drive events through the same idempotent pipeline. When an upstream bug or your own bad deploy corrupts a day of data, replay is the difference between an hour of work and a week of apology.
Quick check
An optional 2-3 question self-check. Answers never leave your device, are not stored, and never count toward any assessment.
Exercise
Design a fictional webhook receiver for a payment provider: signature verification step, ack behavior, queue and worker split, dedup key, and the replay procedure for a bad 6-hour window.
Pass criteria
Verification named with mechanism, ack before processing, durable queue with worker, event-ID dedup with storage, and a replay procedure that is safe because processing is idempotent.