Chaos Engineering: Break Things on Purpose With Chaos Mesh

I stood in a room once while a failover was executed inside a single datacenter. Everything about it was supposed to be routine. The failover stack was there, it had been built, it had been signed off, and it was sitting in the rack doing exactly nothing until the moment it was needed. Then we needed it, and it did not work. There were setup issues on the failover stack, and the failover failed.

The problem was that the stack simply never had been exercised. It had been configured correctly once, in the sense that someone wrote the configuration and someone else approved it, and from that moment on it existed as a line item in a design document rather than as a thing that worked. The first time anyone found out it was misconfigured was the worst possible time to find out.

That is the disease this article is about. Once something is approved, we assume it works in future, and attention goes back to delivering new features from business demand. Chaos engineering is the discipline of refusing that assumption. Not breaking things for fun, but forming a precise prediction about how your system behaves under a specific failure, then causing that failure on purpose to find out whether the prediction holds.

The Controlled Demolition

When a demolition crew brings down a tower block, the interesting part is not the explosion. It is the weeks before it. The engineer surveys the structure, models the collapse, and writes down exactly how the building will fall, which way it will lean, where the debris will land. Then they rope off an exclusion zone sized to the charge they are actually using, secure the permits, and detonate. If the building falls the way the model said, the engineering was sound. If it falls sideways into the neighbouring street, everyone learns something expensive.

A chaos experiment is the same discipline, one structure at a time. Your steady-state hypothesis is the engineer's prediction: written before anything is broken, specific enough to be wrong. Your blast radius is the exclusion zone, deliberately sized to the charge. Staged expansion is the single low-floor charge you set before anyone goes near the load-bearing core. And the sign-off is the permit, because nobody detonates anything without one.

The value is in the prediction, not in the bang. An experiment that breaks something and teaches you nothing is vandalism with a runbook.

The analogy breaks in one place. A demolition destroys the building permanently and happens once, as a spectacle, with a crowd watching. A chaos experiment is reversible by design and is supposed to be dull. If your chaos practice is one dramatic Game Day a year that everyone remembers, you have rebuilt the same trap in a more exciting costume. Real practice is small demolitions constantly, on structures that get rebuilt in seconds.

What Chaos Experiments Actually Look For

The failures worth injecting are boring and well documented, which is what makes them worth injecting. An availability zone disappears and you find out whether your quorum survives it. You inject latency and discover retry amplification, where three layers of well-meaning retry policy turn one slow call into a self-inflicted load test. You kill a task mid-work and learn whether the work was idempotent or merely assumed to be. You watch connection pools exhaust themselves under a fault that should have been contained by the bulkheads you put in place, which is exactly the sort of isolation claim an experiment exists to falsify.

My favourite class is DNS. Someone sets a 60 second TTL, everyone plans around 60 seconds, and then a resolver or an SDK somewhere caches for far longer. You will never find that in a design review. You will find it the first time you actually move a record and watch traffic keep arriving at the old endpoint.

The tooling has matured around this. Netflix's Simian Army, the thing everyone still cites, is retired and no longer actively maintained, its functionality split into separate services. AWS Fault Injection Service covers the cloud side, and as of December 2024 it injects network faults directly into ECS tasks on Fargate, so container network chaos no longer needs a custom sidecar. On Kubernetes the space has settled into two CNCF incubating projects rather than one winner, Chaos Mesh and LitmusChaos, with Gremlin as the commercial incumbent and Steadybit the newer entrant.

Getting Your Hands Dirty With Chaos Mesh

We are going to run a real experiment on your laptop. No cloud account, no spend, nothing that can page anyone.

Step 1: Get the Tools

You need a container runtime and three binaries. Docker is the default and best-supported runtime; kind also auto-detects Podman and nerdctl, though both run rootless and need extra setup first. If Docker Desktop is running, that part is done.

The three tools are kind, which runs a Kubernetes cluster inside containers, kubectl to talk to it, and helm to install Chaos Mesh. Package managers are the path of least resistance and, unlike the raw binary downloads, carry no version number to go stale.

# macOS and Linux
brew install kind kubectl helm

# Windows
choco install kind kubernetes-cli kubernetes-helm

Note the Windows package names: kubectl ships as kubernetes-cli and helm as kubernetes-helm. For every other platform, the kind quick-start and the Kubernetes install tools page stay current in a way a blog post cannot.

Confirm all three respond before going further.

kind version
kubectl version --client
helm version

Each prints its own version and exits.

Step 2: Build Something Worth Breaking

Create a local cluster and a target workload. Two replicas of nginx behind a service is enough structure to have an interesting failure mode.

kind create cluster --name chaos-lab
kubectl create deployment web --image=nginx --replicas=2
kubectl expose deployment web --port=80
kubectl rollout status deployment/web

Expect deployment "web" successfully rolled out. The cluster takes a minute or two to come up the first time, since kind pulls its node image.

Step 3: Install Chaos Mesh

Chaos Mesh installs as a controller plus CRDs. It needs to be told which container runtime socket to talk to, and kind runs stock containerd, so the containerd path applies as-is. The namespace is not created for you, which is the single most common first-run failure.

Setting MSYS_NO_PATHCONV=1 for the command turns the translation off. It is harmless on macOS and Linux, where the variable means nothing, so the command below is safe to run anywhere.

helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update
kubectl create ns chaos-mesh
MSYS_NO_PATHCONV=1 helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-mesh \
  --version 2.8.4 \
  --set chaosDaemon.runtime=containerd \
  --set chaosDaemon.socketPath=/run/containerd/containerd.sock
kubectl get pods -n chaos-mesh

Wait until every pod reads Running. The daemonset pod is the one that does the actual injecting, so if it is stuck, nothing below will work.

Step 4: Write the Hypothesis Down Before You Touch Anything

This is the step that separates the experiment from the outage, and it is the one that gets skipped. Write the prediction first, in a file, in the repo, where someone can hold you to it afterwards.

# Experiment: single pod termination, web deployment

Steady state: requests to the `web` service from inside the cluster return
HTTP 200 on every request.
Hypothesis: killing one of two web pods causes zero failed requests, because
the remaining replica absorbs the traffic and the deployment replaces the
dead pod within 30 seconds.
Blast radius: namespace `default`, label `app=web`, exactly one pod.
Abort condition: any non-200 response, or replacement taking over 60s.
Measured from: a curl pod inside the cluster, not a port-forward, which
would be measuring the tunnel rather than the service.

Now open a port-forward in a second terminal and confirm the steady state actually holds before you break it. An experiment run against a system that was already broken proves nothing.

kubectl port-forward svc/web 8080:80

while true; do curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/; sleep 1; done

You should see an unbroken column of 200. A port-forward is fine for this pre-flight check, since nothing is being broken yet. Stop it once you have confirmed the steady state, because the moment you start injecting faults it becomes the wrong instrument, for reasons that are about to become obvious.

Step 5: Set the Smallest Charge You Can

This is the low-floor charge, not the load-bearing core. mode: one picks a single pod. The selector is scoped to one namespace and one label.

# pod-kill.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: web-pod-kill
  namespace: default
spec:
  action: pod-kill
  mode: one            # exactly one matching pod, not all of them
  gracePeriod: 0
  selector:
    namespaces:
      - default        # the exclusion zone
    labelSelectors:
      app: web

kubectl apply -f pod-kill.yaml

Expect podchaos.chaos-mesh.org/web-pod-kill created.

Note the absence of a duration field. A pod kill is a one-shot fault: it happens once, immediately, and there is nothing to keep running or to recover from. Chaos Mesh will accept a duration here and then ignore it, which is worse than rejecting it, because you would walk away believing you had bounded something you had not.

Step 6: Observe Against the Prediction, Not Against Your Feelings

Watch both terminals. The curl loop is your hypothesis under test, and the pod watch shows you the replacement.

kubectl get pods -l app=web -w

kubectl describe podchaos web-pod-kill -n default

The describe output carries a Container Records section whose Id names the pod that was injected, which is the evidence that the experiment did what you asked rather than something adjacent.

Now, the part that will mislead you if nobody warns you first. Your curl column almost certainly collapsed into errors, and the service was fine the whole time. kubectl port-forward tunnels to one specific pod, not to the service, so when chaos killed that pod the tunnel died with it and never reconnected. You will see error: lost connection to pod in the terminal running the forward. Measure from inside the cluster instead, where the service load-balances the way it would in production:

kubectl run curltest --image=curlimages/curl --restart=Never --rm -i -- \
  sh -c 'for i in $(seq 1 10); do curl -s -o /dev/null -w "%{http_code} " http://web.default.svc.cluster.local/; sleep 1; done'

An unbroken row of 200 there means the hypothesis held and this particular claim about your system is true today. If a 502 slipped through, you have found something real, and it cost you a laptop instead of a customer.

This is the most common way a first chaos experiment lies to you. The hypothesis was correct and the instrument was broken, and those two look identical from the outside. Every failed experiment has two candidate explanations, the system or the measurement, and ruling out the second is what makes the first trustworthy.

To run the kill again, delete the resource first. Re-applying an unchanged manifest reports unchanged and does nothing, because a pod kill is one-shot and Chaos Mesh deliberately pins it so it cannot re-fire:

kubectl delete -f pod-kill.yaml && kubectl apply -f pod-kill.yaml

Now repeat with a fault that is harder to survive. Network latency finds retry storms that pod kills never will, and because it is a continuous fault rather than a one-shot, duration is both valid and load-bearing here.

# network-delay.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: web-latency
  namespace: default
spec:
  action: delay
  mode: one
  duration: 60s
  selector:
    namespaces:
      - default
    labelSelectors:
      app: web
  delay:
    latency: "500ms"
    jitter: "100ms"

kubectl apply -f network-delay.yaml

Measure it the same way, asking for timings rather than status codes:

kubectl run curllat --image=curlimages/curl --restart=Never --rm -i -- \
  sh -c 'for i in $(seq 1 8); do curl -s -o /dev/null -w "%{time_total}s " http://web.default.svc.cluster.local/; done'

Expect the timings to alternate between roughly a second and roughly a millisecond, because mode: one delays a single pod while the service keeps round-robining across both. That is the experiment working, not a flaky result. Note also that the round trip lands near twice the configured latency, since the delay applies in each direction.

The fault lifts itself after 60 seconds whether or not you are still watching, which is what makes it safe to walk away from. kubectl get networkchaos web-latency -o jsonpath='{.status.conditions}' will show AllRecovered flip to True, and that is the contrast with the pod kill above: here duration genuinely bounds the fault.

Step 7: Tear It Down

kubectl delete -f pod-kill.yaml -f network-delay.yaml --ignore-not-found
helm uninstall chaos-mesh -n chaos-mesh
kind delete cluster --name chaos-lab

Everything you created is gone. That reversibility is the whole point, and it is where the demolition analogy stops applying: you can run this again tomorrow.

Sizing the Charge

The question that actually stops teams is not which tool. It is how big to make the exclusion zone and who signs the permit.

Stage	Blast radius	Who approves	What you learn
Local cluster	Your laptop	Nobody	Whether the experiment is well formed
CI or ephemeral env	One namespace, synthetic traffic	Team lead	Whether the hypothesis is falsifiable
Staging	Full stack, no real users	Team plus platform owner	Whether the tooling and observability work
Production, off-peak	One zone, one service, small traffic slice	Named on-call and service owner	Whether the thing is actually true
Production, business hours	Expanding slice, automated abort	Same, with abort authority delegated	Whether it stays true under load

Standing approval matters more than one-off approval. A one-off approval is exactly the failure mode from the top of this article: it happens once, someone signs it, and then the attention goes back to feature delivery. A standing approval with named abort authority is what turns chaos engineering into a habit rather than an event.

One disambiguation, because two unrelated things share an acronym. The EU's Digital Operational Resilience Act, Regulation 2022/2554, has nothing to do with the DevOps Research and Assessment metrics of the same name. EU DORA became fully applicable on 17 January 2025 and mandates threat-led penetration testing under Articles 26 and 27, at least every three years for significant financial entities. It does not mandate chaos engineering by name. The conflation runs in one direction, towards claiming a regulator requires resilience experiments. The regulator requires adversarial security testing. Chaos engineering is a reasonable thing to do alongside it, on its own merits, and pretending it is compulsory will not survive contact with anyone who has read the text.

The Thing That Nobody Wants to Fund

Here is the honest ending, and I am not going to tidy it up. It is always hard to sell preventing issues to management and sponsors. This includes disaster recovery. Testing disaster recovery is intensive work, it requires a lot of work and validation, and its entire success condition is that nothing interesting happens.

The principles worth carrying out of this:

• Write the hypothesis before the fault. No prediction, no experiment, just an outage you caused yourself.
• Size the exclusion zone to the charge. Start at one pod in one namespace and earn your way outward.
• Get standing approval, not one-off approval. One-off approval is how the failover stack got built and then forgotten.
• Make it boring. A single annual Game Day is a spectacle. Repetition is the practice.
• Rehearsed failure is the only kind you get to learn from. The other kind arrives without a prediction to check against.

The failover stack in that datacenter was not a technical failure. It was an organisational one: something built, approved, and then never asked to prove itself again, right up until the day it had to. Every system you have signed off is currently making the same silent promise.

Which of those promises have you actually tested? And when the next failover runs, will it be a rehearsal, or the first time anyone finds out?