DevOps Professional · Module 2: Reliability, capacity and resilience · Lab
Measure Recovery Under Controlled Failure Injection
55 min hands-on · Advanced
A local spread fixture (two failure domains with anti-affinity and replicated fixture data); injection limited to the lab fixture. Never shared, borrowed or production systems.
Local guide: run the steps below on your own machine in order, then check the validation list.
Objectives
- State the hypothesis before each injection
- Measure detection, failover and loss window with quoted numbers
- Write the HA honesty limit for what ran
Step 1
Hypothesize first
Write what should happen when one domain dies: detection source, failover path, expected loss window. The hypothesis precedes the kill, always.
Step 2
Kill and measure
Kill one domain, measure detection time, failover time and the data loss window from fixture evidence. Restore and quote all three numbers.
Step 3
State the limit
Write the honesty paragraph: the mechanism proven (detection, failover, recovery) and what local rehearsal proves nothing about (real zones, partitions, provider events, correlated scale failures).
How to confirm it worked
- Hypothesis written before each injection
- Detection, failover and loss quoted from evidence
- Honesty paragraph present and specific
- No injection outside the lab fixture