The question that reorganised our entire supply chain programme took about eleven seconds to ask.

Not “do you sign your images.” Not “do you scan for vulnerabilities.” It was: for the container that was running in production on that date, show me the commit it came from, show me the build that produced it, and show me that the build could not have been altered by the person who requested it.

That last clause is the design problem. Everything before it is tooling.

I own platform architecture for a Kubernetes-based clearing platform that settles on the order of $4.5 trillion a day across three regions, with on-premises data centres alongside AWS. We hold a 99.99% availability target, under SEC and CFTC oversight, with Regulation SCI behind every change. An outage is not an incident report. It is a market event.

Under that constraint, supply chain security stops being a security feature. It is an evidence-production system that also happens to block bad builds.

SLSA levels are a list of people you have to trust, not a maturity score

Teams read the SLSA ladder as a score and then optimise the score. Read it instead as an answer to one question: who has to be honest for your provenance to be true?

Level 1 means provenance exists. Anyone can forge it.

Level 2 means the build service signed it. Now you trust the build service’s word, and anyone who can modify a pipeline can change what the build service says.

Level 3 means the provenance is non-falsifiable. The build runs in an isolated, ephemeral environment and the signing identity belongs to the build platform, not to anything a pipeline author controls. The person who wrote the pipeline cannot make the provenance lie.

Only that last jump matters in an audit, because Level 2 is “the build system says so” and Level 3 is “the build system could not have said otherwise.” If a developer can inject a step that writes arbitrary provenance, you have a self-signed claim with extra ceremony.

A trimmed predicate, so the shape is concrete:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [{
    "name": "registry.internal/clearing/matching-svc",
    "digest": { "sha256": "9f2c...e81a" }
  }],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://internal/build/container/v2",
      "externalParameters": {
        "repository": "git+https://internal/scm/clearing/matching-svc",
        "ref": "refs/heads/main"
      },
      "resolvedDependencies": [
        { "uri": "git+https://internal/scm/clearing/matching-svc",
          "digest": { "gitCommit": "4c1b9e0" } }
      ]
    },
    "runDetails": {
      "builder": { "id": "https://internal/build/runners/hardened-pool" },
      "metadata": { "invocationId": "b-2026-08-14-00921" }
    }
  }
}

Three fields carry the audit. builder.id is how you argue isolation. resolvedDependencies is how you answer “built from the commit that was reviewed.” The subject digest is what ties the statement to something that actually ran.

The digest. Not the tag. A tag is a mutable pointer, and a mutable pointer in an evidence chain is a defect.

Keyless signing converted our key custody problem into an availability problem

We went keyless. The build workload presents an OIDC token, a short-lived certificate gets issued against that identity, the signature and certificate land in a transparency log, and the certificate expires in minutes.

This was not a novelty choice. Our identity model everywhere else is short-lived certificates and mutual TLS between workloads on enterprise PKI. We had spent years removing long-lived static credentials. Introducing a signing key with a five-year life and a custody procedure would have been a regression dressed as a control.

What we underweighted: verification at admission now depends on reaching a transparency log.

The afternoon new pods stopped scheduling

One cluster runs a deliberately tight egress allow-list. A routine change to that list dropped the Sigstore endpoints. Verification was set to fail closed, which is correct.

Nothing happened for hours. Running pods kept running — admission had already cleared them.

Then a node group scaled out during a busy window, and every pod landing on the new nodes failed admission because verification needed a log lookup it could no longer make. The platform did exactly what we had configured. We had never tested the configuration.

What changed afterwards:

• Verification material travels with the artifact. Signature, certificate and inclusion proof ship as a bundle, so verification succeeds without a live call.
• The trust root is mirrored in-region, with its own availability target and its own page.
• The transparency log is on the tier-one dependency map. Anything that can stop your platform scheduling belongs there.

Fail-closed is the right default. Fail-closed on an uncharacterised external dependency is a deferred outage with good intentions.

A signature proves somebody signed; only an identity constraint proves you signed

We run Gatekeeper and Kyverno together, which draws the occasional complaint about redundancy. The split is deliberate. Gatekeeper takes the constraints that need Rego and need to reason across resources. Kyverno takes image verification, attestation checks and mutation, where its primitives sit closer to the problem and the resulting policy is still readable six months later.

The part of the policy that does the work:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-build-provenance
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-signature-and-slsa-provenance
      match:
        any:
          - resources: { kinds: ["Pod"] }
      verifyImages:
        - imageReferences: ["registry.internal/clearing/*"]
          mutateDigest: true
          required: true
          attestors:
            - entries:
                - keyless:
                    subject: "https://internal/build/runners/hardened-pool/*"
                    issuer: "https://oidc.internal/build"
          attestations:
            - type: https://slsa.dev/provenance/v1
              conditions:
                - all:
                    - key: "{{ predicate.buildDefinition.externalParameters.ref }}"
                      operator: Equals
                      value: "refs/heads/main"

subject and issuer are the entire point of that block. Without them you have built a system where anyone holding a valid Sigstore identity can sign your production images, and your policy will approve it.

This is where most implementations I have reviewed quietly enforce nothing. Policy passes. Dashboard is green. Control is decorative.

mutateDigest: true matters for a different reason: it rewrites tags to digests at admission, closing the gap between what you verified and what the kubelet pulls.

Generate the SBOM from the artifact, not from the recipe

An SBOM is not a control. It is an inventory you query under time pressure at an hour you did not pick. The only question it has to answer is which running things contain this component at this version, and how fast you can say so. Hours is bad. Days is a disclosure event.

We got this wrong twice.

We generated from the build context instead of the built image. Scanning the source tree and the manifest produced a tidy list that omitted everything the base image contributed. A base image refresh pulled in an undeclared package, and our inventory insisted it was not present. The recipe tells you intent. The artifact tells you what shipped.

We attached attestations to the wrong digest. Multi-architecture builds produce an image index plus per-platform children, each with its own digest. Our pipeline attached SBOM and provenance to the index. Verification passed in the pipeline and failed in the cluster, on one architecture only, which made it look intermittent for far longer than it deserved.

That one surfaced in a drill rather than an incident, which is the only comfortable sentence in this section.

Generate by digest, attach to every child:

syft "registry.internal/clearing/matching-svc@${DIGEST}" \
  -o spdx-json > sbom.spdx.json
cosign attest --yes \
  --predicate sbom.spdx.json \
  --type spdxjson \
  "registry.internal/clearing/matching-svc@${DIGEST}"

Control counts are not an achievement; aggregation is

Supply chain controls do not stand alone, and running them as a standalone programme is how you end up with a correctly signed image in an account that should not exist.

Ours sit inside four other layers. Service control policies deny whole classes of action at the account boundary regardless of local IAM. Sentinel policies fail infrastructure changes at plan time rather than apply time. AWS Config watches continuously for drift, because something always drifts. Admission handles signature, provenance and configuration.

Together those aggregate 2,041 active controls into Security Hub.

I quote that number carefully, because control counts get presented as outcomes and they are not. Two thousand controls across five consoles with no shared severity model is worse than two hundred in one place. The aggregation is the work. The count is a byproduct.

The break-glass path is the control a regulator is actually testing

Strip the paperwork away and the asks reduce to three. An inventory of what is running and what it was built from. Evidence for one specific artifact: commit, build, approver, signature, verification decision. And a demonstration that the control cannot be bypassed.

The third is the difficult one, and it is almost never about cryptography.

You have a break-glass path. Everyone does. On a platform with a market-facing clock, somebody will eventually need to deploy something non-conforming at 03:00 to stop a settlement cycle slipping. The only question is whether that path was designed or improvised.

Build the exception register before you turn enforcement on

We did it the other way round, and that is the decision I would reverse first.

For about a quarter, our emergency path was a human with cluster-admin and a chat thread. Nothing broke. But every use of it produced evidence reconstructed afterwards from memory and timestamps, rather than captured at the moment of the decision. Reconstructed evidence is the weakest thing you can bring to an audit, and it is weakest precisely where the stakes were highest.

Make the exception a first-class object before the first enforcing policy ships: time-bound, signed, named approver, read by the policy engine, expiring on its own, emitting an event when used. You get identical operational flexibility and evidence that was recorded rather than remembered.

The smaller correction: our first enforcing policy went cluster-wide in audit mode, then flipped to enforce. Audit mode told us which workloads were violating. It said nothing about workloads that did not exist yet — scale-out paths, DaemonSet upgrades, vendor operator images nobody had inventoried. Scope the first enforcement to one namespace containing a workload that scales, and leave it there longer than feels necessary.

What surprised me most about this work is how clerical it turned out to be. Signing is easy. Verification is easy. Producing a defensible chain of custody, on demand, for an artifact that ran four months ago is a records-management problem wearing a cryptography costume. Design for the question you will be asked rather than the demo you want to give.