FDE Foundations · Module 6: AI Decisions
Cost and Latency Budgets
AI features live or die on unit economics: cost per document, latency per call, monthly ceiling. Budget them at design time and instrument them from day one.
10 min reading
Objectives
- Estimate per-unit cost and monthly spend before building
- Set a latency budget per step and measure it
- Reduce cost with caching, routing, and batching
Do the arithmetic
Estimate calls per day times tokens (or seconds) per call times price per unit. A feature processing 5,000 documents daily at a few cents each is a four-figure monthly line item: someone must approve that number before build, and the brief should carry it. Include retry overhead and the review-queue labor in the same calculation.
Latency budgets
Set a budget per step, not just end to end: retrieval 200 ms, model call 2 s, post-processing 50 ms, and a page-level target that fits the user's workflow. Measure each in logs from the first deploy. Averaged latency hides the pain; track p95, because the slow tail is what operators experience.
Levers, in order of preference
Cache identical or near-identical requests (many documents repeat). Route by difficulty: cheap path first, expensive model only when needed. Batch small calls where the provider allows it. Shorten prompts by retrieving less. Compression tricks that degrade quality last.
The ceiling
Configure a hard monthly spend cap with alerting at 50 and 80 percent. A runaway loop or an abusive input should cost you an alert, not an invoice dispute.
Quick check
An optional 2-3 question self-check. Answers never leave your device, are not stored, and never count toward any assessment.
Exercise
Build the cost model for a fictional extraction feature: 8,000 docs per day, your assumed tokens and price, monthly estimate, and three named levers with expected savings. Add a p95 latency budget for three steps.
Pass criteria
Arithmetic shown and monthly figure stated, three levers with plausible relative savings, and per-step p95 budgets that sum to a user-acceptable total.