Roughly half our fault injection experiments found a real failure. That number should bother you, because every one of those systems had already passed design review, load testing, and a disaster recovery exercise. We ran about 34 experiment categories, built over 100 templates, and executed more than 350 experiments against a platform that clears aroundContinue reading “What 350 Fault Injection Experiments Taught Us About Kubernetes Resilience”
Monthly Archives: May 2026
Hosted Control Planes in Production: Rebuilding Ephemeral Kubernetes Environments with Kamaji
The ticket said the test platform was broken. It wasn’t. Every worker node across four environments had been reclaimed inside roughly ninety seconds, mid-test-run, because I had pinned one instance family in one availability zone and handed AWS exactly one capacity pool to take back. That is the kind of mistake you only make once,Continue reading “Hosted Control Planes in Production: Rebuilding Ephemeral Kubernetes Environments with Kamaji”
