Go look at the painkillers in your medicine cabinet. Somewhere on the box, in small print next to the expiry date, there is a lot number.
Nobody reads it. It does not help you take the pill. It exists for one day — the day a manufacturer discovers a bad batch of an ingredient and needs to know which boxes used it. Not every box. Those boxes. The lot number is what makes the difference between pulling one production run and pulling a product line.
Now picture that day with AI.
Your assistant has been quoting an old refund policy for three weeks. Someone notices. Fixing the document takes ten minutes. Then a lawyer asks which customers got the wrong answer, and the room goes quiet, because nobody can say. Not “it will take a while” — nobody can say, at any cost, with any amount of effort.
Most teams keep logs. Logs are sorted by time. The documents have changed since. Few systems record which version of which document each answer actually read.
So the ten-minute fix is the easy part, and everything after it is a guess.
Logs are sorted by time, and a recall asks a different question
This is the whole gap in one sentence. Logs answer “what happened at 3:02pm?” A recall asks “which outputs used this document?”
Those are different access patterns, and you cannot serve the second one from a store built for the first — not at volume, not under time pressure, not with the retrieval corpus mutating underneath you.
The liability is not theoretical. In 2024 a Canadian tribunal held an airline liable for wrong fare information its chatbot gave a customer. Companies answer for what their AI says. The practical shape of that answerability is: when you get something wrong, can you identify who you got it wrong to?
The provenance store sits on the inference path
Three pieces. A provenance store with its own retention and access rules. A stamping interceptor in the inference path, inside the request boundary rather than bolted on afterward. And an ingredient index — the inverted mapping from ingredient to outputs.
The stamp is written synchronously before the output is returned to the caller. This is the part teams resist, because it adds latency to the hot path. Write it anyway. A stamp cannot be reconstructed after the fact: the retrieval set is gone, the prompt has been rebuilt, the model alias has moved. Provenance you intended to record asynchronously is provenance you do not have.
What you can do asynchronously is everything bulky — index maintenance, forward-trace enrichment, signing, export to long-term storage. Keep the hot path to one append of a few kilobytes.
If you have done software supply chain work this will look familiar, and that is not an accident. A lot record is an in-toto style attestation for an inference: materials in, product out, with a digest on both sides. We sign lot records the same way we sign build provenance, which means nobody can quietly backdate one after an incident.
1. Stamp every output with what went into it
The lot number records what went in: model version, instructions version, the exact documents and data read, the tools called.
A lot record:
{ "lot_id": "lot_01HQJ8M2X7", "produced_at": "2026-10-09T14:21:08.442Z", "producer": "spiffe://platform.internal/ns/support/sa/answer-service", "output_digest": "sha256:1f0c...b4e2", "ingredients": { "model": { "id": "vendor-model-4", "version": "2026-07-11", "params_digest": "sha256:aa31...90f7" }, "instructions": { "ref": "prompts/support-answer", "version": "v38", "digest": "sha256:77be...c104" }, "documents": [ { "ref": "kb/refund-policy", "version": "r214", "digest": "sha256:9d2a...1c8b", "chunks": [3, 4] }, { "ref": "kb/shipping-exceptions", "version": "r88", "digest": "sha256:04ff...77aa", "chunks": [11] } ], "data": [ { "ref": "crm.account", "key_digest": "sha256:c3d1...8e42", "as_of": "2026-10-09T14:20:51Z" } ], "tools": [ { "name": "order.lookup", "call_digest": "sha256:5b70...2f19", "result_digest": "sha256:e9a4...6630" } ], "retrieval": { "index_version": "emb-2026-08-02", "query_digest": "sha256:31cc...70d5", "k": 8 } }, "derived_from": [], "signature": { "scheme": "sigstore", "bundle_ref": "prov://lots/lot_01HQJ8M2X7.sig" }}
Every ingredient is identified by digest, not by path. A path is a pointer to whatever lives there now, and the entire reason you are building this is that the thing living there now is different from what the model read.
Stamping in Python, as a context manager around the inference call:
contextmanagerdef lot(producer: str, task: str): rec = LotRecord(lot_id=new_lot_id(), producer=producer, task=task) token = _current_lot.set(rec) # contextvar, so retrieval try: # and tool layers can append yield rec finally: _current_lot.reset(token) rec.freeze() # no appends after this point provenance.put_sync(rec) # blocking, before output returns index.enqueue(rec) # inverted index, async# Retrieval and tool wrappers append without the caller thinking about it.def record_document(ref: str, version: str, digest: str, chunks: list[int]): if (rec := _current_lot.get(None)) is not None: rec.ingredients.documents.append( Document(ref=ref, version=version, digest=digest, chunks=chunks))
The contextvar matters more than it looks. If stamping requires every call site to remember to pass a record through, stamping will be incomplete within a month — the exception path, the cache hit, the retry, the fallback model. Put collection in the layers that already touch the ingredients and the call sites cannot forget.
Freeze before write. An unfrozen record invites a well-meaning engineer to append to it later, and a lot record that can change is not evidence.
2. Trace forward to where the output landed
A stamp tells you what went into an answer. A recall needs to know where that answer went: who received it, and what it became.
An email. A closed ticket. A figure in a quarterly report. A merged code change. Each of those is a different notification problem and a different correction problem, so record which one it was.
{ "lot_id": "lot_01HQJ8M2X7", "deliveries": [ { "channel": "email", "recipient_digest": "sha256:b8e1...0c73", "sent_at": "2026-10-09T14:21:12Z", "external": true }, { "channel": "ticket", "ref": "CS-44192", "state": "closed", "closed_at": "2026-10-09T14:26:03Z" } ]}
Recipients are stored as digests with a reference to where the identity actually lives. The provenance store should be able to tell you that an output reached a person without itself becoming a second copy of your customer database.
Forward trace is the half that gets dropped. Teams build stamping, feel finished, then discover during their first incident that they can list five hundred affected outputs and cannot tell who saw any of them. Affected outputs with no recipients is a list you cannot act on.
3. Index by ingredient, not by timestamp
Store the links in both directions so that the recall question takes seconds instead of a weekend of log archaeology.
The edge table is the entire trick:
CREATE TABLE lot_ingredient ( ingredient_ref text NOT NULL, -- 'kb/refund-policy' ingredient_version text NOT NULL, -- 'r214' ingredient_digest bytea NOT NULL, lot_id text NOT NULL REFERENCES lot(lot_id), kind text NOT NULL, -- document | model | instructions | tool | data PRIMARY KEY (ingredient_ref, ingredient_version, lot_id));CREATE INDEX lot_ingredient_recall ON lot_ingredient (ingredient_ref, ingredient_version) INCLUDE (lot_id, kind);
And the recall query, which should return in milliseconds:
SELECT l.lot_id, l.produced_at, d.channel, d.recipient_digest, d.refFROM lot_ingredient liJOIN lot l ON l.lot_id = li.lot_idLEFT JOIN delivery d ON d.lot_id = li.lot_idWHERE li.ingredient_ref = 'kb/refund-policy' AND li.ingredient_version IN ('r211','r212','r213','r214') AND l.produced_at >= '2026-09-18';
Nothing clever. That is the point — this is an ordinary secondary index on a question nobody thought to ask at design time. The cost of adding it later, after a year of outputs, is a backfill you cannot run because the data was never captured.
4. Recall is a workflow with states, not an email
When an ingredient turns out to be wrong, pull every output that used it, correct what you control, tell the people who relied on the rest, and record when each one is closed.
That last clause is what makes it a workflow. A recall with no closure state is a spreadsheet somebody stops updating on day three.
recall: rc-2026-10-09-refund-policyingredient: ref: kb/refund-policy bad_versions: [r211, r212, r213, r214] corrected_version: r215scope: lots: 1842 deliveries_external: 1130 deliveries_internal: 712disposition: ticket: { action: reopen_and_correct, owner: support_ops } email: { action: notify_recipient, owner: support_ops, template: corr-02 } report: { action: restate_figure, owner: controller } code: { action: open_revert_pr, owner: platform }closure: require_per_delivery: true evidence: provenance_store
Disposition by channel, because a closed ticket and a merged pull request need different humans doing different things on different clocks.
5. Rehearse the recall and time it
Food manufacturers run mock recalls. Some operate under a two-hour trace limit.
Do the same thing, and make it boring: pick a document at random, ask which outputs used it and who received them, start a timer.
The first time we ran this the answer was not a slow number. It was a missing capability — we could identify the outputs and not the recipients. No design review surfaces that. A drill surfaces it in twenty minutes.
Run it quarterly, rotate the ingredient type, and record elapsed time as a metric with a target. An untested recall path is an assumption.
Unpinned models are not ingredients you can trace
A handful of details decide whether any of this holds up.
Pin your versions. A model called “latest” is not an ingredient. If the alias moved on 11 July and your records say “latest,” every output on either side of that date is unattributable, and you will find out during the recall. Same for embedding index versions, prompt templates, and tool schemas.
Saved outputs become ingredients. An AI-written summary that gets stored and later read by another agent is now an input to that second answer. derived_from exists for this. Without the transitive closure you recall the first generation and miss the second, which is the version of this failure that embarrasses people publicly.
A lot number tells you which outputs to check, not which are wrong. This is the honest limit of the pattern and you should say it out loud before someone else does. The record tells you what was available to the model, not what it relied on. Some of those eighteen hundred outputs never mentioned the refund policy. Provenance scopes the review; it does not perform it.
Keep the record as long as the output is in use. Not as long as the conversation, not thirty days — as long as the answer still has effects. A figure in an annual report has a long tail.
Store references, not content, and protect the record like the data it points to. A provenance store full of digests and references is a map of your sensitive data. It needs the same access controls as the systems it indexes, and it generally does not get them, because it looks like metadata.
Where this pattern breaks
Mutable pointers. The most common implementation error by a wide margin. The team records kb/refund-policy without a version or digest, because the retrieval layer only ever had the path. Everything looks fine until the first recall, at which point the index tells you which outputs touched a document whose contents you can no longer determine.
Cardinality and cost. A thousand lots a second with twelve ingredients each is twelve thousand edges a second. The edge table grows faster than anything else you own. Partition by time, roll cold partitions to object storage with the index retained, and decide deliberately what you stop indexing. Decide by accident, when the bill arrives, and someone drops the ingredient dimension to save money.
Index lag during the incident that matters. Asynchronous index maintenance is correct for throughput and dangerous during a recall, because the last hour — the hour you most want — may be the hour not yet indexed. Expose index lag and make the recall tool read through to raw lots for anything newer than the current watermark.
Caches and embeddings are ingredients and nobody treats them as such. You fixed the document. The vector index still holds chunks built from the old text, the response cache still serves the old answer, and both produce fresh outputs that cite a corrected source. Fixing the document is not fixing the ingredient.
Human copy-out. Someone pastes an AI answer into a deck and the deck goes to a client. No stamp follows. You can mitigate this at the edges and you cannot eliminate it. Pretending otherwise produces false confidence in recall completeness, which is worse than a known gap.
PII creep. Over a few quarters, “store the query digest” becomes “store the query, it helps debugging.” Now the provenance store is a retention liability. Enforce the schema in CI and reject fields that carry content rather than references.
Adoption: stamp first, index second, recall third
If agents are already in production, you do not get to stop and build this properly. You get to build it in the order where each step is independently useful.
Stamp one surface. Pick the highest-exposure output path — usually the one that talks to customers. Add the interceptor, write records, index nothing. Two weeks of records will tell you whether your retrieval layer even knows the versions of what it fetched. Usually it does not, and fixing that is the real first task.
Pin versions everywhere that produces ingredients. Model aliases, prompt templates, embedding indexes, knowledge base documents. This is unglamorous platform work with no visible output, and skipping it makes every later step cosmetic.
Add the forward trace. Delivery records at the points where output leaves the system. You now have something you could not have before: the ability to answer who received an answer, without yet being able to answer which answers are affected.
Build the index and run a drill. Then the mock recall, timed, with a target. Publish the number internally. A recall time that leadership knows about is a number that gets engineering attention.
Regulators ask who received it, not what your model scored
Audit evidence is a first-class concern in the environment I work in, under SEC and CFTC oversight and Regulation SCI. The questions that arrive after an incident are not about model quality.
They are: what did the system tell customers, over what period, based on what, and who did you notify. Each of those is a provenance query. None of them is answerable from a model evaluation report or an accuracy dashboard.
There is a second-order effect worth planning for. Once you can enumerate affected outputs you are expected to act on the enumeration, and a signed record showing you identified eleven hundred affected customers and notified four hundred is a harder document to explain than one showing you notified everyone. Which is an argument for writing the disposition policy before your first recall, not during it.
Without provenance, your only choices are warn everyone or say nothing
When you cannot tell who a mistake reached, you have two options, and both are bad. Notify your entire customer base about a problem that touched four hundred people and you have manufactured an incident out of an error. Say nothing and hope, and you have made a decision you would struggle to defend.
A lot number gives you the third option: warn the people it reached. That is the whole value proposition, and it is only available if the record was written at the moment of production, because that moment is the only time the ingredients exist in one place.
The painkiller manufacturer did not print that number because it expected a recall. It printed it because recalls happen to everyone eventually, and the number is cheap in advance and impossible in hindsight.
So: if you found a wrong document in your AI’s knowledge base today, how long would it take to list every answer that used it?
