FDE Foundations · Module 8: Agents & Tools
Evaluating Agent Workflows
Evaluate agents on tasks, not vibes: a set of scenario tasks with graded outcomes, scored together with cost and steps, run automatically on every change.
11 min reading
Objectives
- Build task-based evals with graded outcomes
- Measure success, cost, and steps together
- Run evals in CI and before releases
Task-based evals
Write 20 to 40 scenarios: the input situation, the environment state (fixtures), and what success means (the refund issued to the right order, the draft email meeting the rubric). Grade outcomes, not transcripts: a correct outcome reached an inefficient way still mostly succeeded; a wrong outcome with beautiful reasoning failed.
The triple metric
Track success rate, median cost per task, and median steps. Optimizing one alone misleads: a cheaper agent that fails twice as often is worse. The triple, published as a table per release, is how you and the customer agree the system is improving.
Evals in CI
Run the eval suite on changes to tools, prompts, or models. Pin model versions for comparability and re-run on pinned upgrades. Fail the build on regression beyond threshold; agent regressions are silent to manual testing, which is exactly why the suite exists.
From evals to release notes
Report to the customer in their terms: "Refund accuracy 96 percent on 40 scenarios, cost per case down 18 percent after tool caching." Numbers from your own suite, honestly labeled, build the trust that carries the next phase of the engagement.
Quick check
An optional 2-3 question self-check. Answers never leave your device, are not stored, and never count toward any assessment.
Exercise
Design the eval suite for a fictional travel-booking agent: eight scenarios (list with expected outcomes), the triple metric targets, and the CI rule on model upgrades.
Pass criteria
Eight scenarios with graded outcomes, all three metrics with target numbers, and a CI rule covering regressions and pinned model version changes.