Roughly half our fault injection experiments found a real failure.

That number should bother you, because every one of those systems had already passed design review, load testing, and a disaster recovery exercise. We ran about 34 experiment categories, built over 100 templates, and executed more than 350 experiments against a platform that clears around $4.5 trillion a day across multi-region Kubernetes and Kafka on AWS and our own data centres, under a 99.99% availability target and Regulation SCI.

A roughly 50% hit rate against a system that had passed every conventional gate is the single most useful data point I have about the limits of conventional gates.

Resilience reviews measure documents, not systems

Here is what I think the industry gets wrong about resilience.

We verify it by asking people to describe what would happen. An architect writes that the service tolerates the loss of an availability zone. A reviewer reads it, finds the reasoning sound, and signs. Both parties are now confident about a behaviour neither has observed.

The reasoning usually is sound. The failures we found almost never came from a wrong design. They came from the gap between a design and its configuration — a probe timeout nobody revisited, a disruption budget written when the replica count was different, a client library default from three upgrades ago.

You cannot review your way to that. You have to break something.

The blast radius problem comes before the chaos problem

The hard part of fault injection in a regulated environment is not the faults. It is proving to a risk committee that the tooling which deliberately breaks production cannot break more than you said it would.

So the security model was the first design, not a later hardening pass.

A single orchestrator account. All experiment definitions live in one account that owns no workloads. Nothing to accidentally target locally, one audit surface, one place where the catalogue of what we are permitted to break is version-controlled.

Scoped cross-account roles. The orchestrator assumes a role in each target account. That role’s permissions are the narrow set of actions a given experiment category needs, and nothing else.

Tag-based targeting exclusively. No resource ARNs in templates. An experiment declares the tags it acts on, and IAM independently refuses the action unless the resource carries them.

The IAM condition doing the real work:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "FaultInjectOnlyTaggedTargets",
      "Effect": "Allow",
      "Action": ["ec2:StopInstances", "ec2:RebootInstances"],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/fis-eligible": "true",
          "aws:ResourceTag/fis-scope": "${aws:PrincipalTag/experiment-scope}"
        }
      }
    }
  ]
}

The second condition key is the one I would point a reviewer at. Matching fis-eligible alone means every opted-in resource in the account is reachable by every experiment. Binding fis-scope on the resource to a principal tag on the assumed role means an experiment scoped to one service physically cannot act on another, regardless of what the template says.

Defence in depth here is not ceremony. The template is a statement of intent; IAM is the thing that holds when the intent is wrong.

Tag propagation quietly widens your blast radius

We learned the value of that second condition the direct way.

An early network experiment targeted a tag applied at the Auto Scaling group level. ASG tags propagate to launched instances by default, and the same tag had been applied to a second ASG months earlier by someone standardising their Terraform. The experiment matched both. We injected packet loss into a service that was not in the experiment plan, during that service’s business hours.

Nothing failed over incorrectly, which was luck rather than design. The on-call engineer for that service spent 40 minutes investigating a degradation whose cause was sitting in a different team’s experiment window, and found out what had happened from us rather than from his own telemetry.

Two things came out of it. The IAM scope binding above, so a mis-scoped tag cannot translate into a mis-scoped action. And a pre-flight resolver that enumerates the exact resources a template will touch and requires a human to confirm the count before execution.

Tag-based targeting is the right primitive. It is also a query, and a query against a tag space your whole organisation can write to will eventually return something you did not expect.

Reaching on-prem without opening an inbound path

Half our platform is not in AWS. Resilience testing that stops at the cloud boundary tests the easy half, and the interesting failures in a hybrid system live exactly at that seam.

SSM hybrid activations let us treat on-premises and DMZ hosts as managed instances. The agent establishes an outbound HTTPS connection and polls. No inbound firewall rule, no listener on a DMZ host, no new path through the perimeter — which is what made the conversation with the network security team short.

Registering a DMZ host, with the constraint that matters:

# Activation is short-lived and bounded by instance count. A long-lived,
# high-count activation code is a credential; treat it like one.
aws ssm create-activation \
  --default-instance-name "dmz-fis-target" \
  --iam-role "SSMHybridFaultInjectionRole" \
  --registration-limit 1 \
  --expiration-date "$(date -u -d '+2 hours' +%Y-%m-%dT%H:%M:%S)" \
  --tags "Key=fis-eligible,Value=true" "Key=fis-scope,Value=edge-tier"

--registration-limit 1 and the two-hour expiry are there because our first pass used a generous limit and a 30-day window, which is what the examples show. That activation code is a bearer credential that can enrol an arbitrary host into your managed fleet. Ours sat in a pipeline variable, reusable, for a month.

Nothing came of it. It was still the worst finding of the programme, and it was self-inflicted — an artifact of copying an example without asking what the parameters were protecting.

The five failure classes that actually taught us something

Across 34 categories, the results clustered into five groups. Memory and CPU constraints, network latency and packet loss, node disconnection and recovery, EBS volume failure, and cascading failure.

The first four are table stakes. The fifth is why you do this.

Memory and CPU pressure mostly finds your probe configuration

Memory constraint experiments rarely killed the thing we were pressuring. They killed its neighbours.

Under memory pressure the kubelet evicts by QoS class, and we found pods we considered important running as Burstable because someone had set requests without limits in a Helm values file. The experiment was scoped at one workload; the eviction landed on another in the same node pool.

CPU constraint found the same bug from a different direction. Throttle a service enough and its liveness probe times out, so Kubernetes restarts a container that was slow rather than broken. Several of our probe timeouts were tighter than the service’s own p99 under load, which means we had configured Kubernetes to amplify latency into outage.

A liveness probe is a statement that a slow process is better killed than waited for. Most of ours were set by copying an example, and most should have been readiness probes.

Latency injection breaks consumer groups, not producers

Our most interesting result came from injecting a few hundred milliseconds of latency on a Kafka path.

Producers absorbed it. Consumers did not — the added latency pushed processing past the configured max.poll.interval.ms, the broker considered those consumers dead, and the group rebalanced. Rebalancing paused consumption, which deepened the lag, which triggered the next rebalance. The system had found a stable oscillation and would not have recovered on its own.

No individual component was misconfigured. The timeout was a default, the latency was realistic for a degraded link, and the interaction between them had never been exercised. That is a failure mode that exists only in composition, and it is unreachable by component-level testing.

Volume failure is where recovery time hides

EBS failure experiments against stateful workloads produced our longest recovery, around six hours.

The attach/detach cycle did not complete cleanly, the volume stayed in a transitional state, and the pod’s rescheduling blocked on a six-minute multi-attach timeout that then retried against a volume still held by a node the scheduler believed was gone. Our runbook for the scenario was accurate and useless, because it assumed the API would return the volume state the runbook described.

Recovery procedures that have never been executed are hypotheses. We now rate a runbook’s maturity by when it was last run against an injected fault, not by when it was last reviewed.

Half the passing experiments were not testing anything

The finding that changed how I think about chaos engineering is an embarrassing one.

When we audited the templates after the first large batch, a meaningful share of experiments that had “passed” had never established that the fault landed. No steady-state hypothesis, no stop condition, no assertion — just an action, a wait, and a green result. An experiment that silently failed to inject anything looked exactly like a system that tolerated the fault perfectly.

We had been generating confidence, not evidence.

The gate we should have had from the beginning:

{
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:us-east-1:111122223333:alarm:clearing-tier-slo-burn"
    }
  ],
  "experimentOptions": {
    "accountTargeting": "single-account",
    "emptyTargetResolutionMode": "fail"
  },
  "logConfiguration": {
    "cloudWatchLogsConfiguration": {
      "logGroupArn": "arn:aws:logs:us-east-1:111122223333:log-group:/fis/experiments:*"
    },
    "logSchemaVersion": 2
  }
}

emptyTargetResolutionMode: fail is the setting I wish I had found on day one. The default is to proceed when target resolution returns nothing, which is how an experiment against a renamed tag reports success while doing nothing at all.

The stop condition matters for a different reason. A stop condition is the only part of an experiment that proves you were watching, and a template without one is not an experiment. It is an outage with paperwork.

What I’d do differently

I would not have built 100 templates before building the scoreboard.

We optimised for coverage — categories enumerated, templates authored, a satisfying number to report. Then we audited and discovered a portion of that coverage was decorative. Ten templates with mandatory hypotheses and automated verdicts would have taught us more in the first quarter than 100 templates did, because the thing you are building is not a library of faults. It is a feedback loop, and an unverified experiment poisons the loop while inflating the metric.

Concretely: no template enters the catalogue without a declared steady-state hypothesis, an automated pass/fail assertion, and a stop condition. Volume second.

I would also have run the pre-flight target resolver before the first experiment rather than after the one that strayed. It took about a week to build. It would have prevented a 40-minute investigation by an engineer who had no idea his service was in anyone’s test plan, and I had enough information to predict that failure before it happened.

Every passing experiment is a claim about your observability

The real output of 350 experiments was not a list of fixed bugs. It was a calibration.

We now know which of our alerts fire on injected faults and which stay silent. We know which runbooks survive execution. We know, for a set of specific failures, the difference between what the design documents claim and what the system does — and that difference was large enough, often enough, that I no longer trust a resilience assertion that has not been executed.

If you take one thing from this: treat an experiment with no stop condition and no hypothesis as a failed experiment, not a passing one. Half of ours were, and we were counting them as wins.