The ticket said the test platform was broken. It wasn’t. Every worker node across four environments had been reclaimed inside roughly ninety seconds, mid-test-run, because I had pinned one instance family in one availability zone and handed AWS exactly one capacity pool to take back.
That is the kind of mistake you only make once, and it is not the most interesting one I made on this project.
Here is the claim I want to argue for: a full dedicated cluster per tenant is the wrong default for ephemeral Kubernetes environments, and most platform teams reach for it anyway because it is the shape they already know. We reached for it too. Then I threw the design out.
Context: a clearing platform that couldn’t get itself a test environment
Our platform settles around $4.5 trillion on an ordinary day, across multi-region Kubernetes and Kafka spanning AWS and our own data centres, against a 99.99% availability target, under SEC and CFTC oversight with Regulation SCI obligations on top.
And an engineer who needed an isolated Kubernetes environment to validate a change waited 22 to 45 days for it.
The spread mattered more than the average. You cannot plan a release around a number that vague, so people stopped trying. They tested in shared environments, which meant they tested against someone else’s half-finished change.
Twenty-two days was eleven queues, not one
A new environment meant a network request, a subnet allocation, a firewall rule set, a load balancer, certificates, an IAM boundary, the cluster itself, a secrets bootstrap, an identity integration, monitoring onboarding, and a change record covering all of it. Every step had an owner. Every owner had a backlog. The critical path was almost entirely queue time, not work time.
Our GitOps modernization programme dealt with that. Harness out, Argo CD and GitHub Actions in, the eleven queues collapsed into declarative pipelines. Provisioning went from 22–45 days to under four hours.
Which solved the wrong problem, or rather only half of it.
A four-hour environment that lives nine months is not an ephemeral environment. It is a permanent environment with a fast birth. It drifts, accumulates, needs patching, appears in the compliance inventory, and eventually someone has to answer what it is for. We had dozens, each carrying a full control plane — etcd members, API servers, schedulers, controller managers — sized for high availability on a cluster whose entire purpose was to be destroyed.
The vendor design priced an empty cluster like a production one
The proposal we inherited was internally consistent, which is the dangerous kind of wrong. Each tenant gets a complete cluster, built by a pipeline, torn down by a pipeline. Clean isolation. Nothing exotic.
Two numbers killed it.
Control plane overhead set the floor cost of an empty environment at roughly what a small production cluster costs — in our case about three to four dedicated nodes before a single workload pod scheduled. At the concurrency a platform team actually needs, that floor is most of the bill.
And time-to-ready has a hard physical minimum: node provisioning, bootstrap, CNI, CoreDNS, operators. We measured the vendor path at roughly 25 to 35 minutes on a good run. For an environment meant to live through one afternoon of integration testing, startup was a double-digit percentage of its entire lifespan.
I scrapped it and resubmitted the architecture request.
A control plane is a workload, so schedule it like one
The idea predates the tooling. A Kubernetes control plane is a few processes and a datastore. Nothing requires it to occupy dedicated machines.
Kamaji runs tenant control planes as ordinary Deployments on a management cluster — each with its own API server, its own certificate authority, and either its own etcd or a slice of a shared datastore. Worker nodes join normally, because from a kubelet’s perspective nothing unusual is happening.
The payoff is not primarily cost. It is that every piece of machinery you already operate now applies to control planes: the autoscaler, pod security standards, network policy, the monitoring pipeline, backup tooling. A tenant control plane becomes an object your team can roll and observe with primitives it already understands. The savings fall out of that, not the reverse.
The tenant control plane shape we settled on:
apiVersion: kamaji.clastix.io/v1alpha1kind: TenantControlPlanemetadata: name: eph-7f3a namespace: tenant-eph-7f3a labels: env-class: ephemeral ttl-class: 8hspec: dataStore: etcd-pool-b controlPlane: deployment: replicas: 2 resources: apiServer: requests: { cpu: "250m", memory: "512Mi" } limits: { cpu: "1", memory: "1Gi" } service: serviceType: LoadBalancer kubernetes: version: v1.31.4 kubelet: cgroupfs: systemd addons: coreDNS: {} konnectivity: server: port: 8132 networkProfile: port: 6443 certSANs: - eph-7f3a.ephemeral.internal
dataStore is the field that decides whether this design survives contact with real tenants. One line, and we got it wrong first.
konnectivity is the other one to look at. Our worker nodes sit in tenant VPCs with no inbound path to the management cluster. Konnectivity inverts the direction so the API server reaches kubelets through a tunnel the nodes open outbound, which let us keep the network model we had already argued through security review. Architecture that avoids reopening a settled review is worth real money.
Shared etcd turns one team’s reconcile bug into a tenancy finding
We started with a single etcd cluster behind every tenant control plane. The documentation supports it. The savings are real. It was wrong for us.
A team’s test suite installed a controller that created and deleted custom resources in a tight loop — a reconcile bug on a branch, which is exactly what a test environment exists to catch. The write churn pushed etcd backend commit latency high enough that unrelated tenant control planes started dropping their own watch connections. Three environments degraded because of a fourth tenant’s bug.
In most shops that is a noisy-neighbour incident you write up and move on from. In ours it is a tenancy-isolation finding, and that is a far more expensive conversation.
We moved to pooled datastores — a handful of etcd clusters, tenants distributed by label, hard cap on tenants per pool. We also stopped asking the datastore to protect us on its own and pushed enforcement up into the API server.
The flow schema now applied to every tenant control plane:
apiVersion: flowcontrol.apiserver.k8s.io/v1kind: FlowSchemametadata: name: tenant-workload-limitspec: priorityLevelConfiguration: name: tenant-workload matchingPrecedence: 900 distinguisherMethod: type: ByUser rules: - subjects: - kind: Group group: name: system:serviceaccounts resourceRules: - verbs: ["create","update","patch","delete"] apiGroups: ["*"] resources: ["*"]
distinguisherMethod: ByUser is the whole point of that manifest. Without it, one service account’s write storm consumes the entire priority level’s concurrency share and you have rebuilt the problem one layer up. With it, the storm is bounded to the identity producing it.
API Priority and Fairness is widely treated as a knob you tune after something hurts. On multi-tenant control planes it is a design input.
Spot diversification is the feature, not a tuning knob
Ephemeral environments have a load profile almost comically suited to spot capacity: short-lived, restartable, no trading day depending on them. Each tenant cluster gets its own spot node group, created with the environment and reclaimed with it.
The node pool, trimmed to what matters:
apiVersion: karpenter.sh/v1kind: NodePoolmetadata: name: eph-7f3a-workersspec: template: metadata: labels: tenant: eph-7f3a spec: requirements: - key: karpenter.sh/capacity-type operator: In values: ["spot"] - key: node.kubernetes.io/instance-type operator: In values: ["m6i.large","m6a.large","m5.large","m5a.large","m6i.xlarge","m6a.xlarge"] - key: topology.kubernetes.io/zone operator: In values: ["us-east-1a","us-east-1b","us-east-1c"] nodeClassRef: name: eph-worker expireAfter: 8h disruption: consolidationPolicy: WhenEmptyOrUnderutilized budgets: - nodes: "20%"
Those six instance types across three zones exist because of the ninety-second eviction I opened with. Breadth of instance types is not an optimisation you add later for a few percent of savings. It is the mechanism that makes spot survivable at all, and a single-family single-AZ pool is a correlated-failure design wearing a cost-savings costume.
Deleting the control plane deletes the thing that runs your finalizers
Every environment carries a TTL and a controller reaps expired ones. That worked on the first attempt, which should have made me suspicious.
Deleting a tenant control plane deletes its API server. The API server runs finalizers. So when the reaper removed a control plane while workloads still held PersistentVolumeClaims and Services of type LoadBalancer, the finalizers never ran, and the EBS volumes, load balancers, and network interfaces underneath stayed behind with nothing pointing at them.
We found out through a quota error about five weeks in, after accumulating a few hundred orphaned volumes. Not catastrophic. Embarrassing, and precisely the class of untracked infrastructure a Reg SCI inventory review exists to surface.
The fix is ordering, and the interesting part is the failure branch:
func (r *Reaper) Teardown(ctx context.Context, tcp *kamajiv1.TenantControlPlane) error { tc, err := r.tenantClient(ctx, tcp) if err != nil { return fmt.Errorf("tenant client: %w", err) } if err := r.cordonAll(ctx, tc); err != nil { return err } // Delete workloads and let the TENANT api server run its own // finalizers. Version one skipped straight past this. if err := r.deleteWorkloads(ctx, tc); err != nil { return err } // Block on cloud-backed objects actually disappearing. if err := r.waitForFinalizers(ctx, tc, 15*time.Minute); err != nil { // A stuck finalizer means orphaned infrastructure. Hand it to a // human who owns it rather than retrying into the void. r.events.Warn(tcp, "FinalizerTimeout", err.Error()) return r.quarantine(ctx, tcp) } return r.client.Delete(ctx, tcp)}
quarantine is the line I would defend hardest. Version one returned an error and let the controller retry, so a stuck finalizer either blocked the work queue indefinitely or got dropped on the floor depending on backoff. Quarantining parks the environment in a state a person owns, with its cloud resources still tagged and attributable.
Reapers that fail tidily are worse than reapers that fail loudly. Tidy failure is how you end up with a few hundred volumes nobody can explain.
Ephemeral infrastructure still owes you an audit trail
The cost I did not budget for: our audit obligations do not care that a cluster was temporary.
API server audit logs are evidence. If the control plane is a Deployment that gets deleted, its audit trail goes with it, and you have destroyed the record of who did what in an environment that touched real schemas and realistic test data. We added a sidecar streaming audit events to the central store continuously, and made teardown verify the final event was acknowledged before proceeding.
This generalises well past Kubernetes. Making infrastructure disposable is a decision about the infrastructure. It is not automatically a decision about its records, and conflating the two is how teams discover a retention gap during an examination rather than a design review.
What I’d do differently
Pooled datastores from day one. I identified shared etcd as a risk, wrote it down as a risk, and shipped it anyway because the savings were attractive and the failure felt theoretical. It produced an isolation incident that was entirely foreseeable, and the migration cost more than building it correctly would have. Documenting a risk is not mitigating it — a lesson I apparently needed to learn in production.
And I would write the reaper before the provisioner. We spent early effort on provisioning speed because that was the visible complaint, which is how most platform teams sequence this work. Deletion is where the real complexity lives: ordering, finalizers, orphaned state, audit evidence.
Build the delete path first
“Ephemeral” does enormous load-bearing work in architecture documents, and in nearly every one I have reviewed it means “we intend to delete this.”
Intent is not a mechanism. TTLs, reapers, finalizer ordering, and audit export are the mechanism, and they are most of the engineering. If your ephemeral environment design does not have a controller whose only job is destruction, you have designed fast provisioning and called it something else.
