Case Study: Rescuing an AI Service Drowning in Latency and Cost
Updated
A fictional document-summarization feature reaches production and meets its enemies: p95 latency of 14 seconds and a monthly invoice nobody predicted - measurement, tiering, caching and an honest fallback turn it around.
Fictional training case, not a real engagement.
Situation
"Meridian Legal" (fictional) ships contract summarization. Demo was fast (10-page contracts); production includes 300-page monsters. p95 = 14s (budget: 5s interactive); cost projection 4× the approved budget.
The delivery arc
- Instrument first: per-workflow latency percentiles, tokens in/out, model version tags - within a week, the top cost driver is visible (long documents sent whole, premium model everywhere).
- Tiering: classify by length + question type; short/simple → small model; complex → large model with escalation on low confidence. 71% of docs never need the premium tier.
- Caching: content-hash keyed summaries for unchanged contracts - 22% hit rate in month one.
- Chunked map-reduce summaries for long docs: extract per-section, reduce to brief; p95 drops to 4.1s.
- Honest fallback: when the model API degrades, queue for batch and tell the user - silently degrading quality was worse than a wait.
- The report: monthly cost per summarized contract, trendline, and the volume assumption - finance became the project's ally.
What learners should extract
- Measure before optimizing (latency/cost method).
- Model tiering + caching are the first two levers, not prompt tweaking.
- A wait-with-explanation beats a silent quality drop (rollback thinking).
Practice version
Our cost/latency practice task gives you a call-log fixture and asks for the diagnosis + a tiering plan.