AI Proof of Concept to Production: The 24-Point Checklist

The gap in one line

POCs answer "can the model do this sometimes?" Production asks "does the system do this reliably, affordably, and safely - and can we tell when it doesn't?" The checklist below is how you answer the second question.

Evaluation (1-6)

• [ ] Golden test set (50+ real cases, including adversarial)
• [ ] Groundedness/citation checks for retrieval features
• [ ] Baseline comparison (simple non-AI approach) documented
• [ ] Acceptance thresholds agreed with the customer before launch
• [ ] Regression suite wired into CI for prompt/model changes
• [ ] Human review sampling process for ongoing quality

Safety and guardrails (7-12)

• [ ] Input validation and injection probes tested
• [ ] Output schema enforced (structured outputs where actions follow)
• [ ] Human approval gate for consequential actions
• [ ] PII handling defined (masking before prompts, retention rules)
• [ ] Refusal behavior for out-of-scope requests
• [ ] Abuse/rate limits per user and per key

Operations (13-18)

• [ ] Latency budget defined and measured (p50/p95)
• [ ] Fallback path when the model/API is down (queue, degrade, retry)
• [ ] Versioned prompts and model pins - changes are deployments
• [ ] Dashboards: error rate, latency, quality samples, cost
• [ ] Alerts on symptoms (failure rate, freshness), not vibes
• [ ] Runbook including "model misbehaving" incident class

Cost (19-24)

• [ ] Per-request cost measured, monthly projection signed off
• [ ] Caching where content repeats (hash-keyed)
• [ ] Model tiering: cheap first, escalate on confidence
• [ ] Token budgets per workflow; hard cap with graceful degrade
• [ ] Customer-visible cost report (monthly)
• [ ] Kill switch: one flag disables the AI path without redeploy

If fewer than 20 of 24 are true, you have a promising POC - label it that way to the customer. The success metrics guide covers the agreement part; RAG evaluation covers retrieval specifics.

Continue: Production delivery · Rollback runbook