Treat Your Retrieval Index as a Data Product

“Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?”
— A question put to Charles Babbage, Passages from the Life of a Philosopher (1864)

The index behind a retrieval-augmented feature is a dataset, and it should be run like one. Give it a named owner, written freshness targets that include deletions, lineage on every chunk, access controls that match the source systems, and quality checks on the way in and on the way out. Do this before the feature reaches customers, not after the first incident.

That runs against where most of the effort goes. When a team builds a RAG feature, the attention lands on the model and the prompt: which model, how many examples, how to phrase the instructions. The index is treated as plumbing, built once by a script during the prototype. But the approach described in the original retrieval-augmented generation paper works by conditioning the model on retrieved passages, so the answer can only be as current, as correct and as appropriately scoped as what the retriever finds. The model sets the style. The index sets the ceiling.

How a good demo turns into a bad quarter

Picture a support assistant built on retrieval. The demo went well: it answered product questions from the help center and the internal knowledge base, with citations. Six months later it quotes a pricing page that changed in the spring, recommends an API endpoint that was deprecated, and, once, shows a customer a paragraph from an internal runbook.

Nobody changed the model. Nobody changed the prompt. The problem is the index, and when someone asks who owns it, the answer is a scheduled job one engineer set up during the prototype. The team that built the feature has since moved on.

The business cost is not abstract. Stale pricing creates support tickets and refund requests. A deprecated endpoint costs a customer's engineers a day. The runbook paragraph is a security incident with a disclosure question attached. Each was predictable, and each traces back to the index being nobody's product. Treating it as a data product, with the same expectations as any other, closes most of these gaps.

The case for keeping it simple, heard fairly

The objection is reasonable. An owner, freshness targets, lineage, permission filtering and test sets sound like a lot of process for what is often an experiment. Many AI features are tried and dropped within a quarter. Why build data governance around something that might not survive?

Two parts of the answer. First, the cost of most of this is close to zero at the start and high later. Storing source metadata on each chunk is a few fields when you write the ingestion job, and a full re-processing of the corpus when you try to add it afterwards. Second, the risks are not proportional to the feature's maturity. A prototype that ingests a restricted wiki space can leak it on day one. The sensible middle is not full process for every experiment. It is two non-negotiables from the first commit, covered in the stage section below, and the rest before customer launch.

An owner

Someone has to be accountable for what is in the index: which sources it includes, how often it updates, and what happens when it is wrong. That is a different job from owning the model integration or the user interface. Without a named owner, index problems get reported as "the AI is hallucinating", routed to whoever touched the prompt last, and patched with prompt instructions that cannot fix missing or stale content.

Early on, the owner is usually the team that built the feature. Once several features retrieve from overlapping sources, ownership tends to move to a data or platform team, with each source system's owner responsible for what they publish into it. The non-invasive approach to governance, meeting source owners where they already work, applies here as it does to any shared data.

Freshness targets, including deletes

For each source, decide how stale a document can be before an answer built on it is wrong. Pricing and policy pages might need hours. Product documentation might need to update with each release. An archive of resolved tickets might be fine weekly. Write the target down and measure the lag between a change at the source and the change being searchable.

Deletions deserve their own target. When a document is removed or unpublished at the source, it has to leave the index too, and incremental sync jobs often handle additions and edits but quietly miss removals. A deleted document that keeps being retrieved is a correctness problem and, if it was removed for legal or privacy reasons, a compliance one.

Lineage on every chunk

Every chunk should carry where it came from: source system, document ID and URL, the document's version or modified time, the ingestion run that wrote it, and the chunking settings and embedding model used. That metadata is cheap to store and expensive to reconstruct.

It answers the question users and support staff will ask, "why did it say that?", by tracing an answer back to a specific version of a specific document. It lets you purge everything from one source when you need to. And it makes rebuilds tractable. Changing the embedding model or chunking strategy means re-processing the corpus, which is a backfill like any other; it is only safe if you can tell which chunks came from which version. The case for lineage as the basis of trust in data carries over unchanged.

Access controls that match the source

This is where retrieval indexes do the most damage. Source systems have permissions: a restricted wiki space, a private channel, a folder shared with three people. An ingestion job that runs with a service account can read all of it and write it into one index that every user queries. The permissions have been flattened, and the assistant will retrieve restricted content for anyone whose question is close enough. OWASP lists this pattern among its vector and embedding weaknesses for LLM applications.

Prompt instructions are not an access control. The fix belongs in retrieval. Either build separate indexes per audience, which is simple and works when audiences are few and coarse, or store each chunk's access list as metadata and filter every query by the requesting user's identity before anything reaches the model. In the second case, permission changes at the source need a freshness target too: revoking someone's access should remove their ability to retrieve the content within a known time. The same principles as any data access governance apply; the index is just one more copy of the data.

When in doubt, leave a source out until its permissions can be carried through. A feature that answers fewer questions is a better outcome than one that leaks.

Quality checks in both directions

Check the data on the way in. Ingestion has its own failure modes: documents that parse to empty text, scanned PDFs with no text layer, navigation boilerplate repeated in every chunk, near-duplicate pages, and sources that silently return nothing after a credentials change. Simple checks catch most of them: document and chunk counts per source compared with the last run, empty and very short chunks, duplicate rates. A sudden drop in chunk count from one source should alert someone before users notice.

Check retrieval on the way out. Keep a set of real questions with the documents that should come back for them, and run it on every rebuild and every change to chunking, embedding or ranking. If the right document stops appearing in the top results, you know before the model starts answering confidently from the wrong one. This is the retrieval half of writing evaluations as the spec for an LLM feature. Advanced RAG techniques can improve relevance, but you cannot tell whether they did without this baseline.

What good looks like, and what bad looks like

Good: a support lead can click from any answer to the exact document version it came from. The index owner can say, per source, how fresh it is and when deletions were last verified. A change to the embedding model ships with a before-and-after retrieval score. Nobody can retrieve what they could not open in the source system.

Bad: the index is rebuilt by a cron job nobody remembers writing. Answer quality is judged by whether recent complaints have gone quiet. Restricted content is kept out by an instruction in the prompt. The plan for changing the embedding model is "rebuild everything and see".

What it costs

The work is modest compared with the model bill. Metadata fields, a permission filter, a few ingestion checks and a test set of a hundred questions are days of engineering, not quarters. Rebuilds cost embedding compute, and that cost scales with corpus size and how often you change models, so it belongs in the same cost-per-answer calculation as inference. The expensive version is the one where an incident forces all of it to be built at once, under scrutiny.

How the call changes by stage

Prototype or internal experiment. A script that builds the index is fine. Two things are worth doing even now because they cost almost nothing at the start and a lot to retrofit: store source and version metadata on every chunk, and only ingest content that everyone who can use the prototype is allowed to see.

Customer-facing launch. The feature needs the full set: a named owner, written freshness targets including deletions, permission-aware retrieval, ingestion checks and a retrieval test set that runs automatically. The index belongs in the same trust and monitoring stack as the rest of your production AI.

Several products on shared sources. Build ingestion once as a platform, with indexes owned per domain, rather than letting each feature team write its own crawler of the same wiki with its own idea of who can see what.

The question for the next launch review of any RAG feature is short: who owns the index, how fresh is it, and can it show a user anything they could not open themselves? If those get vague answers, the feature is not ready, however good the demo was.