We had migrated our agent workloads off static API keys. Federation configured, issuer registered, rules scoped, documentation updated, ticket closed.
Then someone ran ant auth status inside a pod and found the static API key was still the credential winning every request.
A leftover ANTHROPIC_API_KEY sat in the container environment, and it sits above federation in the SDK’s credential precedence chain. Federation was configured correctly and doing nothing. For weeks the security posture we had documented and the one we had deployed were not the same thing.
That gap is the subject of this post, because it recurs at every layer of agent identity — most seriously at the layer almost nobody is designing for.
Federation does not remove the trust decision, it relocates it
The pitch for Workload Identity Federation is simple enough to be misleading. Your workload presents a short-lived OIDC JWT from an identity provider you already run. The model provider validates it against trust rules you configured and returns a short-lived access token bound to a service account in your organization. No static secret to mint, store, rotate, or leak.
That is a real improvement. A credential that expires in minutes beats one that expires never, and taking secrets out of CI removes a whole class of incident.
But notice what happened to the trust decision. It did not disappear. It moved from “who holds this string” to “which claims, from which issuer, may act as which principal.” That is a policy you now own and can get wrong — and getting it wrong is quieter than leaking a key, because nothing looks broken.
Three resources, and only one of them is obvious
The trust model is a triple. A service account is the principal the minted token acts as. A federation issuer registers the OIDC provider whose signatures you accept. A federation rule is the bridge: when a JWT from this issuer carries claims that look like this, mint a token for that service account, with this scope and lifetime.
The rule is where the security lives, and the rule is the one people under-specify.
A correctly narrow rule for a Kubernetes workload:
{ "name": "agent-orchestrator-prod", "issuer_id": "fdis_...", "match": { "subject_prefix": "system:serviceaccount:agents:orchestrator", "audience": "https://api.anthropic.com" }, "target": { "type": "service_account", "service_account_id": "svac_..." }, "workspace_id": "wrkspc_...", "oauth_scope": "workspace:inference", "token_lifetime_seconds": 600}
Two fields here are load-bearing and easy to get lazy about.
subject_prefix with no trailing wildcard pins one service account in one namespace. Write system:serviceaccount:agents:* and every pod in that namespace can mint a token as this principal. Write system:serviceaccount:* and every pod in the cluster can.
audience is subtler. Omit it and the rule accepts the cluster’s default-audience service account tokens — which every pod already has projected into it, with no configuration at all. A broad subject with no audience matcher is not a weak control. It is an open door that passes review because it looks like configuration.
workspace:inference rather than workspace:developer is the other choice I would defend. An agent that only calls the model does not need Files or Skills management. The scope is a ceiling on the minted token, and ceilings are cheap.
Your cluster is already an OIDC issuer, which cuts both ways
Every Kubernetes cluster signs OIDC JWTs for its pods through projected service account tokens. The API server is the issuer. Each token’s sub is system:serviceaccount:<namespace>:<name>. You do not have to deploy anything to get a workload identity — you already have one.
The complication on-premises is reachability. Our internal clusters use issuer URLs that resolve nowhere on the public internet, so discovery mode cannot work. That leaves inline JWKS: you paste the cluster’s public keys into the issuer registration and the provider never dials out.
Which brings me to the second thing that broke.
Inline JWKS has no automatic key refresh. A cluster upgrade rotated the service account signing key, the registered JWKS went stale, and every exchange from that cluster started failing signature verification. Nobody connected a planned Kubernetes upgrade to an authentication outage in a system that had not been touched.
Recovery took two minutes once we understood it: re-fetch the keys, update the issuer. Finding it took far longer, because the failure surfaces as a 401 reading Authentication failed and nothing else.
That opacity is deliberate and correct: a distinguishable error lets an attacker map your rules by watching which failures differ. The operational cost is that your debugging signal lives in the provider’s authentication history rather than your own logs. Learn that before an incident.
The lesson I would generalise: any trust anchor you pin by value instead of by reference becomes a dependency of your upgrade process. Put it in the runbook for the upgrade, not in the runbook for the outage.
The token lifetime is not the number you configured
A detail that cost us a confusing afternoon.
The minted token’s lifetime is the lesser of the rule’s token_lifetime_seconds and twice the remaining lifetime of the JWT you presented, with a 60-second floor. The second bound exists so a derived token cannot meaningfully outlive the identity it came from.
Read that again if you build token caches. Not twice the JWT’s configured lifetime — twice what is left on it. Present a projected token late in its rotation window and you get a short access token, and code that assumed the configured TTL will keep using a credential that expired minutes ago.
The SDK handles this: a two-tier refresh, advisory at expiry minus 120 seconds and mandatory at minus 30, re-reading the token file on every exchange so rotated projected tokens are picked up transparently.
Which is the argument against writing your own exchange loop. We wrote one anyway, for a service that predated SDK support, and got the arithmetic wrong in exactly that way.
Replay protection will find your retry loop
The third failure. An assertion carrying a jti claim can be exchanged only once per issuer.
Our custom exchange code had a retry wrapper around the token request. On a transient network error it retried — with the same cached JWT. The second attempt was rejected as a replay, the retry logic saw another failure and tried again, and the whole thing degraded into a loop that could never succeed.
Single-use assertions are the right default. But they invert a reflex: retry means mint a fresh assertion and exchange again, not send that again. Every retry path around a token exchange has to re-acquire its input, and an idempotent-looking call that is not idempotent deserves a comment in the code.
SPIFFE gives you an identity that survives leaving the cluster
Projected service account tokens are fine while the issuer is the cluster. Once an agent’s identity has to mean something across clusters, across data centres, and to workloads that aren’t pods, you want SPIFFE. With SPIRE the OIDC Discovery Provider becomes the federation issuer, and identity is a SPIFFE ID attested from node and workload selectors rather than a service account name.
The delivery path is where this goes wrong. The SDK reads its identity token from a file; SPIRE hands out JWT-SVIDs over the Workload API socket. Bridge the two by writing the SVID to a volume and that volume matters:
volumes: - name: spire-agent-socket csi: driver: csi.spiffe.io readOnly: true # The JWT-SVID is a bearer credential. Memory-backed emptyDir keeps it # off node disk, out of volume snapshots, and out of node backups. - name: svid-token emptyDir: medium: Memory sizeLimit: 1Mi
medium: Memory is the whole point of that block. A default emptyDir is a directory on the node’s filesystem, so your short-lived credential lands on a disk that gets snapshotted and backed up — and it will outlive the pod in at least one of those places.
Short-lived credentials are only short-lived where they live. We found a non-memory-backed volume in a chart we had inherited, which is the kind of finding that makes you grep every chart you run.
Federation authenticates your platform, not your users
Everything above is plumbing. This is the section I would keep if I had to cut the rest.
Workload Identity Federation solves authentication between your platform and the model provider. It says nothing about authorization between your users and your agents, and those are not the same problem.
Here is the shape of it. Your agent runs as a service account. A federated token authenticates it to the provider. A user asks the agent to do something, the agent calls tools on the user’s behalf — and at the tool boundary, every request carries the agent’s identity, not the user’s.
The agent is now a deputy holding more authority than any of its callers, acting on instructions from all of them. That is the confused deputy problem, described in 1988, and agent architectures reconstruct it faithfully.
The consequences are immediate. You cannot attribute an action to a user, because the audit trail shows a service account. You cannot enforce per-user entitlements, because the tool only knows the agent. And an instruction injected into context executes with the agent’s authority rather than the authority of whoever injected it.
You need a second identity travelling with the request, and a policy decision at the tool boundary that weighs both:
package agent.toolingdefault allow := false# Both principals must be present. An agent-only request is not a# request on anyone's behalf, so refuse to treat it as one.allow if { input.agent.service_account == "svac_agent_orchestrator" input.tool in data.agents.orchestrator.allowed_tools user_permits}# Authority is the intersection, never the union. The agent cannot do# what the user cannot do, and vice versa.user_permits if { some entitlement in data.users[input.user.subject].entitlements entitlement == data.tools[input.tool].required_entitlement}# Context-derived instructions get the caller's authority, not the# agent's. Anything originating from retrieved content is read-only.deny contains "escalation_via_context" if { input.request.origin == "retrieved_content" data.tools[input.tool].mutating}
The rule that matters is user_permits. Effective authority is the intersection of what the agent may do and what the user may do. Most agent deployments I have reviewed compute the union by accident, because the agent’s credential is the only one that reaches the tool.
The standards work is converging here. The IETF AIMS effort and the agent authorization draft treat the agent as a principal distinct from both the user and the deploying organization, with delegation stated explicitly rather than inferred from whichever credential happened to be in the request. WIMSE covers the workload half. Neither is finished. Both are worth reading first.
What I’d do differently
I would build the user-authority boundary before shipping the agent, not after.
We sequenced it the other way, because federation is a completable task with a checkmark at the end and per-user delegation is an architecture problem touching every tool. So we shipped authentication, planned authorization, and ran for a while with an agent whose effective permissions were its own. Nothing was exploited. The audit gap was real, and threading a second principal through tool interfaces already in use cost multiples of designing it in.
I would also treat the migration off API keys as unfinished until a command inside the running workload proves which credential won. “Federation is configured” and “federation is in use” are different claims, and only one of them is verifiable from a config file.
An agent holding your platform’s identity is a deputy waiting to be confused
Replacing static API keys with federated short-lived tokens is worth doing, and it is the easy half. The hard half is that an agent is not a workload. A workload acts for itself; an agent acts for someone, and usually for many someones with different entitlements.
If your answer to “what may this agent do?” is a single service account scope, you have not answered the question. You have answered a different one, convincingly enough that it may be a while before anyone notices.
