A Metadata Schema for Enterprise RAG: Dates, Access Control, Status and Lineage

A vector search cannot tell an active vendor contract from the one it replaced. Both use the same clauses, the same defined terms and the same signature block. Their embeddings land almost on top of each other. Ask for the current payment terms, and the system returns whichever chunk phrases them closest to your question.

That is the core weakness of retrieval built on similarity alone. It finds similar text, not correct text. A smarter model will not fix it. Structured data attached to every chunk will, because it lets the system exclude the wrong documents before it ranks anything.

Why similarity breaks as the library grows

When a RAG system ingests a document, an embedding model turns each chunk of text into a list of numbers that captures its meaning. A question gets the same treatment. The database then returns the chunks whose numbers sit closest to the question's.

This works well for finding related ideas. It fails when the library holds several versions of the same thing.

Take a procedure for international wire transfers. A 2021 draft, a 2023 approved version and a 2026 update all discuss approval limits, beneficiary checks and sign-off. To an embedding model they are near-identical. A compliance officer asks, "What is the approval limit for a vendor wire?" The system may well return the 2021 figure.

In a 100-document pilot, someone curated the files and this rarely happens. Pointed at a full document drive, it happens constantly. This is one of the most common reasons a RAG demo that impressed the board never survives contact with real data.

The fix is a hard filter that runs before similarity: only approved documents, only ones in force today, only ones this user may see. Every one of those conditions needs a field to filter on.

The four field groups

1. Time: what is in force today

Every enterprise document has a lifecycle. Store effective_date and expiration_date on every chunk, plus superseded_by pointing at the replacement document.

At query time, your application injects today's date into the filter. Expired policies, old price lists and lapsed contracts drop out before ranking. A null expiration date means open-ended, not unknown. Keep those two cases distinct.

Keep the ability to query "as of" a past date. Auditors and lawyers ask what the policy was in March 2024, and the same fields answer that. The same fields also handle superseded clinical protocols and the publish workflow that retires them, where an out-of-date version is a patient-safety problem rather than an inconvenience.

2. Access: who may see it

If you load every document into one index without permissions, you have flattened your security model. An employee asking about director salary bands could retrieve confidential HR planning files.

Store allowed_groups on each chunk, copied from the source system's permissions. At query time, look up the user's groups in your identity provider and filter on them. Deny by default: a chunk with no groups is visible to no one.

Store group IDs, never individual user IDs. People change roles weekly. Group membership changes then need no update to the index at all. Law firms are the sharpest case: ethical walls carried from the DMS onto every chunk are this one field doing its job.

3. Status: is it policy or a draft

Shared drives are full of files named Q3_Pricing_Draft_v4_FINAL.docx. The filename tells you nothing. A status field restricted to a fixed list does.

We use five values: draft, approved, superseded, archived and unverified. Default queries search approved only. Authors can opt into drafts explicitly. unverified is the holding state for anything the pipeline could not classify, and it is excluded from every default query.

The status should come from the system of record wherever possible. A document management system with an approval workflow already knows the answer. Guessing from the filename is the last resort.

4. Lineage: where it came from

Long documents are split into chunks. Chunk 42 of a 100-page services agreement means little without context.

Store parent_doc_id, chunk_index, section_title and source_uri. These give you three things. Citations a reader can check ("Section 4.2, Master Services Agreement"). The ability to fetch neighboring chunks when a passage is cut mid-sentence. A link back to the original file.

Also store version, ingested_at and extractor_version. When you improve the extraction logic, these tell you exactly which chunks were processed by the old version and need reprocessing.

The schema as a decision table

An illustrative payload for one chunk:

{
  "chunk_id": "chk_8f72b9a1",
  "parent_doc_id": "doc_449102",
  "text": "All vendor wire transfers above the policy threshold require secondary approval from the VP of Finance.",
  "metadata": {
    "document_type": "policy",
    "section_title": "Financial Controls",
    "status": "approved",
    "effective_date": "2026-01-01",
    "expiration_date": null,
    "superseded_by": null,
    "allowed_groups": ["grp_finance", "grp_exec"],
    "chunk_index": 14,
    "source_uri": "dms://policies/finance/wire-transfers",
    "version": 3,
    "extractor_version": "2026.07"
  }
}

Keep payloads lean: Pinecone documents a limit of 40 KB of filterable metadata per vector.

The filtering catch most guides skip

Vector databases use an index called HNSW to search millions of vectors quickly. Think of it as a road network linking similar chunks. The search walks the roads toward your question.

Filters complicate the walk. When a filter excludes most chunks, many roads lead to forbidden places. Engines handle this differently.

• ▸Some apply the filter after the search. A very selective filter, such as one small department's access group, can then return fewer results than you asked for. pgvector's HNSW index has historically behaved this way. Its README states that filtering is applied after the index is scanned, and that a condition matching 10% of rows returns about 4 rows at the default hnsw.ef_search of 40. Version 0.8.0 added iterative index scans (hnsw.iterative_scan) to compensate, but you must turn them on and tune them.
• ▸Some apply the filter during the walk. That keeps result counts correct, but the network can fragment and recall can drop. Qdrant addresses this with payload indexes that add extra graph edges for indexed values, and its planner switches to a full scan when a filter matches too few points.

How pgvector and Qdrant handle selective filters at 10 million vectors is a decision of its own, with the memory arithmetic that goes with it.

The business consequence is quiet failure. Nothing errors. Answers just get thinner for exactly the users with the narrowest permissions.

Three controls cover it. Create a payload index on every field you filter by. Test retrieval quality with your most selective real filters, not only with an admin account that sees everything. If one tenant or department is a small, hard boundary, give it its own partition or collection rather than a filter.

The retrieval order that works

With the fields in place, retrieval runs in a fixed sequence.

1. ▸Filter. Apply status, dates and access as hard conditions.
2. ▸Vector search inside the filtered set, for meaning.
3. ▸Keyword search (BM25) inside the same set. Embeddings often miss part numbers, acronyms and clause references. Keyword search catches them.
4. ▸Fuse the two result lists, then rerank the top candidates.
5. ▸Expand using lineage fields: pull neighboring chunks and attach citations.

The order matters. Filtering last means the model has already been offered text it should never see.

What over-fetching costs you

Without filters, teams compensate by retrieving more chunks and hoping the right one is in there. You pay for every extra token, on every query.

Use your own numbers:

Annual waste = queries per day × excess tokens per query × price per million input tokens ÷ 1,000,000 × 365

As an illustration only: 5,000 queries a day, 4,000 excess tokens each, at $3 per million input tokens, comes to about $21,900 a year. Swap in your volume and your model's current price.

The larger cost is quality. Give a model the 2021 and 2026 limits side by side, and it may pick the wrong one or blend them. Published research on long contexts, including the "Lost in the Middle" study by Liu and colleagues, found models use information buried mid-context less reliably. More text in does not mean better answers out.

Backfilling metadata onto an existing index

Most teams reading this already have a populated index with thin metadata. You rarely need to start again. You do need a disciplined backfill.

1. ▸Inventory what the source systems already know. Document management systems, SharePoint libraries and contract tools hold dates, owners, approval states and permissions. Pull these first. They are the most accurate values you will get.
2. ▸Apply rules next. Folder paths, templates and filename patterns resolve a large share of document types and some statuses. Rules are cheap and auditable.
3. ▸Use model extraction for what remains. Ask a model to fill a fixed JSON schema from the document text, with a confidence value per field. Treat its output as a proposal, not a fact.
4. ▸Route low-confidence documents to people. Anything below your threshold gets status: unverified and goes to a human review queue. Unverified documents stay out of default results.
5. ▸Update payloads, not vectors. Metadata sits beside the embedding. You can attach it without re-embedding, which saves time and cost. In Qdrant this is a payload update. In pgvector it is an ordinary column update. In Elasticsearch, the Update by query API changes every document matching a query, in scroll batches of 1,000 by default.
6. ▸Build the indexes, then test before switching on. Run your evaluation set with filters off and on. Compare answers, especially for restricted users. Then make the filters the default.
7. ▸Keep it current. Schedule permission syncs and re-run extraction when documents change. A schema that was right at launch drifts within months without this.

Set one rule early: for access, fail closed. A document with unknown permissions is hidden until someone resolves it. A missing date is an inconvenience. A missing permission is a breach.

Frequently Asked Questions

Do we need to re-embed our documents to add a new schema? Usually not. The embeddings describe the text, and the text has not changed. You reprocess the source documents to extract metadata, then attach it to the existing vectors as a payload or column update. Re-embedding becomes necessary only if you also change how documents are chunked or switch embedding models.

Can a model extract the metadata automatically? Yes, with limits. Modern models fill a strict JSON schema reliably when the information is stated in the document. They struggle when it is not, such as approval status that lives only in a workflow tool. Take values from source systems first, rules second, and model extraction third, with low-confidence results sent to a person.

Why not skip metadata and use a model with a very large context window? Because you pay for every token on every query, and accuracy does not improve with volume. A large window still holds the expired policy beside the current one, and the model still has to guess. It also does nothing for access control: confidential text in the context is already exposed. Filtering decides what the model may see. Context size only decides how much it can hold.

If you already run a RAG pilot that cites the wrong version or ignores permissions, check the schema before you change the model. This week, pick ten documents that have a newer version somewhere in the drive, and ask the system about each. Count how many answers come from the old one. Our AI Audit reviews your ingestion, metadata and retrieval path for $2,500 fixed, with a written list of what to fix first.