Every system has failure modes you have not met yet. Chaos engineering is the discipline of meeting them on your schedule, in daylight, with a rollback ready, instead of at 3am during a real outage.
It is an experiment, not vandalism
Define steady state as a measurable SLI (for example, checkout success rate)
Form a hypothesis: killing one replica should not move it
Limit the blast radius: one service, off-peak, in staging first
Inject the fault, measure, and abort instantly if steady state breaks
A real experiment
Start small, kill a pod and confirm nothing users see changes. Chaos Mesh makes the fault declarative:
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata: { name: kill-one-api }
spec:
action: pod-kill
mode: one
selector:
namespaces: [payments]
labelSelectors: { app: api }
duration: 30sGraduate to nastier, more realistic faults once the easy ones are boring, latency is often more revealing than outright death:
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata: { name: db-latency }
spec:
action: delay
mode: all
selector: { namespaces: [payments], labelSelectors: { app: postgres } }
delay: { latency: "200ms", jitter: "50ms" }
duration: 2mMake it a habit: game days
Run scheduled game days with the whole on-call rotation
Test the humans and the runbooks, not just the machines
Every surprise becomes an action item with an owner
Automate the experiments that keep passing, into CI
You do not know a system is resilient until you have watched it recover. Everything else is a hope, not a guarantee.
Resilience is a property you verify, not a box you check. Break things deliberately, learn cheaply, and turn every discovered weakness into a fix before your users find it for you.
