Building a small ETL pipeline for a customer demo
Updated
A demo pipeline that survives questions: fixtures with dirt, quarantine with reasons, and a report the owner can read.
A demo pipeline has one audience: the owner deciding whether your checks catch their dirt. Build for that reading.
The short answer
Two dirty fixtures in, four validation checks in the middle, one report out with quarantine counts and reasons. Demo the failures, not just the flow.
The four checks
| Check | Catches | Quarantine reason |
|---|---|---|
| Unique key present | Missing ids | Missing identifier, source named |
| Duplicate detection | Double-counted rows | Duplicate of an accepted row |
| Range and format | Impossible dates, bad codes | Out-of-range value with the rule |
| Referential join | Orphan rows | No matching parent record |
Worked example: the distributor demo
A fictional distributor (fictional) sends an order feed with duplicate event ids and a weekly spreadsheet with shifting headers. The demo loads both, quarantines duplicates with the accepted row named, flags header drift explicitly, and ends on a one-page report: accepted, quarantined, and the next fix owned by whom. The owner asks about the quarantine table for ten minutes and approves the pilot. The practice tasks carry the same shape with deliberately broken rows; the data quality post extends the check list.
Checklist: demo readiness
- Fixtures contain every failure you claim to catch.
- Every quarantined row names its reason and source.
- The report fits one page with counts first.
- Replay from fixtures reproduces the same numbers.
Related reading
Straight answers
Frequently asked questions
How small should a demo pipeline be?
Three stages: extract from two fixtures, validate with four checks, load into one honest report. Anything bigger hides the story.
Should demo data be clean?
No. Dirty fixtures with duplicates, missing fields and timezone traps prove the checks work. Clean data proves nothing.
What does the owner actually look at?
The error report: how many rows failed, why, and what happens to them. Counts first, architecture second.