Start with a hypothesis and a steady state
Define what normal looks like in measurable terms: checkout success rate above 99.5 percent, p99 latency below 400 milliseconds, queue depth under a threshold. Then state the hypothesis explicitly — 'if one availability zone loses network connectivity, checkout success stays above 99.5 percent'. An experiment without a hypothesis is just an outage you caused.
Run it first in staging, but understand that staging rarely reproduces production's traffic shape. The valuable findings come from production, executed carefully.
Control the blast radius
Begin with the smallest meaningful scope: one pod, one percent of traffic, one dependency. Expand only after the smaller version behaves as predicted. Schedule experiments during staffed hours, announce them, and make sure on-call knows an experiment is running so a real incident is not misattributed.
Wire automated abort criteria into the tooling itself. If the steady-state metric breaches its threshold, the experiment halts and reverts without waiting for a human decision.
Inject the failures that actually happen
Prioritize realistic faults: dependency latency and timeouts, partial packet loss, node termination, DNS failure, expired certificates, disk pressure, and clock skew. Full instance termination is the classic demonstration but rarely the most instructive; degraded dependencies cause far more real outages than clean crashes.
Verify recovery behaviors too — retry budgets, circuit breakers, and backoff. Retry storms are one of the most common ways a small failure becomes a large one.
Automate and close the loop
Run a curated set of experiments continuously in the delivery pipeline so regressions in resilience surface like any other test failure. Every finding becomes a ticket with an owner, and every fix earns a permanent experiment that guards it. Resilience that is not continuously verified decays with the next refactor.
Key takeaways
- Define steady state numerically and write the hypothesis before injecting anything.
- Start with the smallest blast radius and expand only on predicted behavior.
- Automate abort criteria so experiments self-revert on threshold breach.
- Prioritize degraded-dependency faults over clean instance termination.
- Turn every finding into a fix plus a permanent regression experiment.
Need a pod that already works this way?
DevGrid Staffing assembles managed DevOps, platform, and SRE pods with the compliance and delivery practices described here built in from week one.