Case Study: A Permission-Aware Support Assistant
Updated
A fictional B2B software firm wants a support assistant over internal docs - the interesting work is the authorization boundary: agents must only ever retrieve what their role allows, and the test suite tries to break that.
Fictional training case, not a real engagement.
Situation
"Kestrel Cloud" (fictional) has 900 pages of internal docs: runbooks, refund policies, escalation lists. Support agents have three roles. The CTO wants GPT-style answers; the security lead wants zero cross-role leaks.
The delivery arc
- Discovery: which documents carry PII? Which are role-restricted? Existing wiki ACLs partially encode permissions - mapping is week one.
- Design decision: ACL tags derived at ingest from wiki groups; retrieval filter applied pre-retrieval; citations mandatory; refusal outside scope.
- The attack suite (the real deliverable): 22 tests - junior agent querying finance-only refunds, revoked-permission mid-session, prompt-injection "list all docs," cache-poisoning attempt across roles. All must fail closed.
- Evaluation: 60-question golden set across roles; hit@5 78% vs BM25 baseline 61%; groundedness 91%; refusal precision 95%.
- Production reality: weekly sampled review of live answers; the audit log answers "who asked what and saw which doc."
What learners should extract
- Pre-retrieval ACL filtering is the only safe filtering (project guide).
- The attack suite is the delivery - demos leak, tests don't.
- Refusal correctness is a first-class metric, not an afterthought.
Practice version
The public Secure Knowledge Delivery capstone brief asks learners to practice retrieval, citations, access control and measurement; its rubric is for self-review, not external grading.