FDE Foundations · Module 10: Reliability
Incident Response in the Field
Field incidents are joint operations with the customer. The loop is: stabilize first, communicate on a clock, repair properly, and write the postmortem that fixes the system, not the person.
11 min reading
Objectives
- Run the incident loop: detect, mitigate, communicate, repair
- Write the customer-facing update that keeps trust
- Extract the postmortem without blame
Stabilize, then diagnose
First move is mitigation: roll back, shed load, queue the batch, switch to the fallback path. Diagnosis follows stabilization. The rollback plan from m09 is what makes this possible; without one, every incident becomes an improvisation on production.
Communicate on a clock
Agree the cadence at engagement start: for significant incidents, an update every 30 to 60 minutes even when the update is "still mitigating, next update 14:30". The update has three lines: current state, what we are doing, next update time. Silence is what erodes trust; being mid-fix is expected and accepted.
Repair and postmortem
After mitigation, schedule the real fix with a owner and date. The postmortem is blameless and factual: timeline, contributing factors, what worked, and action items with owners, each of which is a ticket. The customer may read it; write it so they can.
The FDE's role
You are the bridge: you know both systems and both teams. Run the joint bridge call, keep the timeline, and make sure both sides' action items land. Handled well, an incident increases the customer's trust; handled silently, it ends the engagement.
Quick check
An optional 2-3 question self-check. Answers never leave your device, are not stored, and never count toward any assessment.
Exercise
Write the incident plan for a fictional data-feed outage: first mitigation step, the 60-minute communication cadence template, and four postmortem action items with owners.
Pass criteria
Mitigation uses a rollback or fallback, cadence template has the three lines, and postmortem items are specific tickets with owners, none blaming a person.