The question that matters is never “do you have policies.”

It is closer to this: on 3 March, a workload ran in this cluster with a mutable image tag. Name the control that should have prevented it. Show me that the control was in effect on that date. Show me every exception that was open at the time, who approved each one, and when it expires.

Three artifacts. An admission controller produces the first one for free. It produces the second and third only if you built for them, and almost nobody does, because the demos are all about blocking a bad pod.

I run policy as code at admission across a multi-region Kubernetes platform sitting behind roughly $4.5 trillion of daily settlement under SEC and CFTC oversight and Regulation SCI. The enforcement was the easy part. The evidence took longer than the enforcement, cost more engineering hours, and is the only part the regulator ever looked at.

Blocking is the cheap half

An OPA/Gatekeeper constraint that rejects a privileged container is a weekend of work. Rolling it out without breaking three teams is a month. Proving on demand that it was enforcing continuously for eighteen months, across every namespace, with a complete exception register, is a system.

That system is the actual product. The policy engine is a component inside it.

Most policy-as-code programmes I review have inverted this. They have four hundred policies and no durable record of which ones were enforcing last quarter. They can tell you about today. Audit is rarely about today.

One layer is not a control

Our governed cloud foundation runs three enforcement layers, and I would not drop any of them:

• Preventive, as service control policies. The action is unavailable, so there is nothing to remediate and nothing to argue about.
• Detective, as AWS Config rules feeding Security Hub, with GuardDuty and IAM Access Analyzer alongside. We aggregate 2,041 active controls drawn from the AWS Foundational Security Best Practices and CIS Benchmarks.
• Pipeline, as HashiCorp Sentinel policies evaluated against the Terraform plan before anything reaches an account.

The pipeline layer is the one teams underinvest in, and it is the one that changes behaviour. A denial at plan time arrives while the engineer is still holding the context. A detective finding arrives three days later as a ticket, addressed to whoever owns the account this quarter.

A Sentinel policy that refuses untagged or unencrypted storage at plan time:

import "tfplan/v2" as tfplan
required_tags = ["data-classification", "cost-center", "control-owner"]
buckets = filter tfplan.resource_changes as _, rc {
	rc.type is "aws_s3_bucket" and
		rc.mode is "managed" and
		rc.change.actions contains "create"
}
main = rule {
	all buckets as _, b {
		all required_tags as t {
			keys(b.change.after.tags) contains t
		}
	}
}

control-owner is the tag that earns its place. It is not a cost attribute, it is the name that goes next to a finding, and making it mandatory at plan time removed an entire category of orphaned remediation work.

Account Factory for Terraform, because controls should speak the language your engineers already review

We deployed AWS Control Tower through Account Factory for Terraform rather than the CloudFormation path.

That decision got pushback, and the reasoning held up. The organization’s infrastructure standard was Terraform. Governance delivered as CloudFormation would have meant two IaC dialects, two review processes, two definitions of drift, and a control plane that no engineer on the platform team could read fluently. Controls that your own engineers cannot review are controls you are trusting rather than operating.

Account provisioning moved from three to four months down to days. The headline number people quote is the speed; the part that mattered more was that break-glass roles replaced manual credential rotation across more than 50 accounts, and compliance reporting and tagging became generated output rather than a quarterly spreadsheet exercise.

Gatekeeper versus Kyverno is a tooling question with a boring answer

Run both. The argument is a waste of a design review.

Reach for Rego and Gatekeeper when the decision needs state beyond the object under review: referential constraints, uniqueness across a namespace, arithmetic over a set of existing resources. Rego is a query language and that is what it is good at.

A constraint template that enforces image provenance by registry, which is deliberately dull:

package k8srequiredregistry
violation[{"msg": msg}] {
	container := input.review.object.spec.containers[_]
	not startswith(container.image, input.parameters.registry)
	msg := sprintf("image %v is not from the approved registry %v", [container.image, input.parameters.registry])
}

Reach for Kyverno when you need to mutate, generate or verify. Mutation and resource generation in Rego is possible and unpleasant. Kyverno policies are YAML, which means a compliance analyst can read one and confirm it says what the control document claims it says. That property is worth more at audit time than elegance.

The cost of running both is two policy languages in the platform. The benefit is that neither one is being stretched past what it was built for, and I have seen far more damage from a heroic Rego mutation webhook than from maintaining two engines.

Provenance nothing verifies at admission is a file

We built a signed supply chain from commit to runtime: SLSA provenance generated in the build, Sigstore signatures, an SBOM per artifact.

For a while, all of that was paperwork.

The attestations existed. The build emitted them. Nothing refused to run an image that lacked them, so in control terms we had documentation, not enforcement. The gap between “we generate provenance” and “unprovenanced images cannot start” is the entire control.

The Kyverno policy that closes it:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-slsa-provenance
  annotations:
    control.internal/id: "SC-SUP-004"
    control.internal/owner: "platform-security"
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: require-signed-provenance
      match:
        any:
          - resources:
              kinds: ["Pod"]
      verifyImages:
        - imageReferences:
            - "registry.internal/*"
          attestations:
            - type: https://slsa.dev/provenance/v1
              attestors:
                - entries:
                    - keyless:
                        issuer: https://token.actions.githubusercontent.com
                        subject: "https://github.com/org/*"
              conditions:
                - all:
                    - key: "{{ buildDefinition.externalParameters.workflow.ref }}"
                      operator: Equals
                      value: "refs/heads/main"

Two details carry the audit weight. The control.internal/id annotation maps this policy to a control in the register, so the evidence export can be generated rather than assembled by hand. And the condition on the build ref means a signed image built from a feature branch is still refused, which is the failure mode people miss when they treat signature verification as a boolean.

Audit wants three artifacts and your dashboard supplies none of them

The control, expressed once and mapped

Every policy needs a control ID and one named owner. We wrote policies for eighteen months and mapped them to controls afterwards, and the mapping exercise was worse than writing the policies. Several policies turned out to enforce overlapping fragments of three different controls, which is impossible to explain in a room.

State over time, not state now

A policy engine reports the present. Audit asks about a date. You need policy version, enforcement mode and effective date range, retained for as long as your record-keeping obligation runs.

Extracting enforcement state as evidence rather than as a dashboard:

kubectl get constraints -o json \
  | jq -r '.items[]
      | [ .kind,
          .metadata.name,
          (.spec.enforcementAction // "deny"),
          (.metadata.annotations["control.internal/id"] // "UNMAPPED"),
          (.status.totalViolations // 0) ] | @tsv' \
  | tee "evidence/constraints-$(date -u +%Y-%m-%dT%H:%M:%SZ).tsv"

Run it on a schedule, sign it, retain it. Unglamorous, and it answered the examiner’s question in minutes rather than the week the first request took us.

The exception register

Exceptions have to be objects with an owner, a justification and an expiry. A namespace in an exclusion list is not an exception. It is an invisible exemption, and I have the scar to prove it.

A constraint sat in dry run for months and we counted violations as blocks

This is the failure I bring up whenever a team tells me their coverage is good.

A constraint governing a meaningful control ran at enforcementAction: dryrun for months after rollout. The dashboard showed violations. Everyone read violations as blocked attempts, because that is what a policy engine is assumed to mean.

When audit asked for the number of exceptions, the honest answer was that the violation count was the exception count. Every one of those workloads had been admitted.

What changed afterwards: enforcement mode is a field in every evidence record, dry-run constraints are reported in their own section with a mandatory expiry date, and a constraint that has been in dry run for more than 30 days fails a scheduled pipeline check. Dry run is a migration state. Treating it as a permanent operating mode is how programmes end up with four hundred policies and no enforcement.

The exclusion list ate a tenant and the board stayed green

The second one was quieter and worse.

During an incident we added a namespace exclusion so a platform component could be deployed. The exclusion used a prefix match. Our namespace convention meant that prefix also matched every namespace belonging to one tenant.

That tenant was exempt from several constraints for a long time. Nobody noticed, because the dashboard showed zero violations for those namespaces, and zero violations looks exactly like compliance.

Absence of violations and absence of coverage must be visually and structurally distinguishable. We now generate exclusions from a registry with expiry dates, and the evidence set includes a coverage report listing namespaces not matched by each constraint. That report is the one I would build first if I were starting over.

The operational scar, briefly

Separately: failurePolicy: Fail is the correct setting and it will bite you during a cluster upgrade. Policy pods drained, admission stalled, and the break-glass instinct in the room was to delete the webhook configuration. We now run the policy controllers on a dedicated node pool with a priority class and a pod disruption budget, and the documented break-glass is a role assumption that fires an alert, not a kubectl delete that leaves no trace.

A lab where a policy can actually fail beats a catalog of policies that never have

I also work as principal cloud security architect on a policy-as-code guardrails programme for a large international institution, pushing CIS policies through an Azure DevOps pipeline into a dedicated lab.

The lab is the deliverable. A policy that has never denied anything is an untested assertion, and a catalog of untested assertions is how compliance theatre gets built. Every policy ships with a test that proves a deny against a deliberately non-compliant resource.

A test case that fails the build if the policy stops denying:

apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: verify-slsa-provenance
policies:
  - ../policies/verify-slsa-provenance.yaml
resources:
  - resources/pod-unsigned.yaml
  - resources/pod-feature-branch-signed.yaml
results:
  - policy: verify-slsa-provenance
    rule: require-signed-provenance
    resources: [pod-unsigned, pod-feature-branch-signed]
    result: fail

Those tests became audit artifacts in their own right. Showing an examiner a pipeline that fails when a control stops working is a stronger statement than showing them the control.

Write the evidence pipeline alongside your first policy, not after your first audit

If I could change one decision, it would be this one, and it is not close.

We built mature enforcement and immature evidence. Reconstructing eighteen months of control state from git history, cluster snapshots and Slack archaeology cost more engineering time than authoring every policy we had. None of that work made the platform safer. It only made the platform explainable, which we should have been from day one.

Start with ten policies that map to controls someone will be asked about. Give each one an owner, an ID, an expiry-bearing exception path, and a signed daily evidence export. Then add the eleventh.

Policy as code is not a security tool that happens to help with compliance. It is the mechanism by which an architecture becomes auditable, and if it does not produce evidence, it is a very expensive linter.