Why AI Proofs of Concept Fail in Production: 12 Gaps Between the Demo and the Real System

The proof of concept that impressed the board behaves differently six months later. Answers take longer. Some of them are wrong in ways nobody can explain. Someone proposes a re-architecture "before we scale further."

The pattern is common enough to have numbers behind it. In July 2024 Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025. Its reasons: poor data quality, inadequate risk controls, escalating costs and unclear business value. MIT's 2025 study found that 95% of generative AI pilots show no P&L impact, with the source. Our statistics round-up sets out what that figure does and does not measure.

The reason is structural, not a matter of talent. A proof of concept is built to demonstrate a concept: clean data, happy-path questions, one presenter, controlled conditions. A production system faces arbitrary input, concurrent users, data that does not stay clean, and months of change after launch. The twelve gaps below sit between those two sets of requirements. Check them in the order given, because the early ones hide the later ones. A system that buckles under load never runs long enough for anyone to notice that its retrieval is weak.

How to read the twelve

Each gap has the same three parts: what it looks like from the outside, what it costs, and the fix. None of the fixes is exotic. Each is standard engineering that a demo had no reason to include. Three of the twelve have their own deep-dives on this site, so those entries are short and point you onward.

The twelve gaps

1. Synchronous document ingestion in the query path

What it looks like. A user uploads a document and the system processes it before answering. With three users it feels instant. With twenty uploading at once, every other user's question waits behind the processing.

What it costs. Abandonment, at exactly the moment a department-wide rollout starts. A slow internal tool never fails loudly. People stop opening it.

The fix. Move ingestion off the query path. An upload drops a job on a queue, a worker processes it in the background, and the query path never touches ingestion compute.

2. No semantic cache on repeated queries

What it looks like. Every question goes to the model, even when most of them are rewordings of the same few dozen questions.

What it costs. A model bill that scales with headcount rather than with the number of distinct things people need to know. Latency on the questions that should have been instant.

The fix. Two layers. An exact-match cache keyed on the normalized question. Then a semantic cache that returns a stored answer when a new question embeds very close to one already answered. Set the similarity threshold high and log every cache hit, so you can audit when a stale answer was served.

3. No document quality control before ingestion

What it looks like. Scanned PDFs, HTML with broken encoding and tables stored as merged cells go straight into the chunker.

What it costs. Embeddings of garbled text retrieve passages that look relevant and are not. The answer reads well and cites a page that says something else. In a regulated setting that is a liability, not a quality issue.

The fix. A conversion stage that turns every input into clean, structured text before chunking, with a rejection path for files that fail. Docling, an open-source document converter, turns PDFs, Office files and tables into Markdown with the layout preserved. Budget this layer first. It moves retrieval quality more than any model upgrade.

4. Fixed chunking strategy regardless of document type

What it looks like. One chunk size and one overlap for everything. A 90-page contract gets the same treatment as a two-paragraph FAQ.

What it costs. Structured documents lose their structure. A clause is split from its heading, a table from its caption, a definition from the term it defines. The retriever returns half of what the model needs.

The fix. Route by document type. Documents with headings get chunks that respect section boundaries. Prose gets recursive splitting. Tables are extracted on their own and stored whole. Where small chunks retrieve well but the model needs more context, store the small chunk for search and hand the model its parent section. Multimodal RAG for complex documents covers the harder cases: scans, charts and mixed layouts.

5. Vector search only, no BM25 hybrid

What it looks like. Pure semantic retrieval. It handles paraphrase well and misses exact strings.

What it costs. Contract numbers, product codes, clause references and people's names are exactly what business users search for. A query for "clause 4.3.2(b)" can miss the clause because the embedding treats the numbering as incidental.

The fix. Run keyword search (BM25) and vector search side by side, then merge the two ranked lists. Reciprocal Rank Fusion, the merge method from Cormack, Clarke and Buettcher (2009), scores each document by its rank position in each list. The two scoring systems never need calibrating against each other. Gate keyword hits on a minimum semantic similarity. Exact-string matches with no relevance will otherwise dilute the top results.

6. No reranker after initial retrieval

What it looks like. The top chunks from search go straight to the model. Nothing checks whether they answer the question.

What it costs. Vector similarity measures closeness in embedding space, not usefulness for this question. The top result is often not the best passage, and the model writes a confident answer from the wrong source.

The fix. Retrieve a wide set, then rerank with a cross-encoder. That is a small model that reads the question and a passage together and scores how well the passage answers it. Keep the top few for the model. The ms-marco-MiniLM-L6-v2 model card reports about 1,800 passages per second on a V100 GPU, so reranking 20 candidates costs milliseconds. Measure the latency on your own hardware before and after.

7. Generic system prompt without constrained generation

What it looks like. The model is told to "answer based on the context" and nothing else. When the context does not contain the answer, it improvises.

What it costs. Fluent answers that cite policies which do not exist, quote figures that were never in the documents, or describe procedures nobody wrote. These are the failures users remember.

The fix. Give the prompt an explicit no-answer rule with a fixed phrase to use, and forbid inference beyond the retrieved text. Then back it with code: a check after generation that verifies each claim traces to a retrieved passage. Any claim that does not is removed or flagged. Prompt wording alone is not a control. Why prompt guardrails fail in production explains what the output check does instead.

8. No observability layer

The system runs and produces answers, and nobody can see retrieval quality, latency distribution or failure modes. Retrieval degrades silently as documents change, and you learn about it from complaints. Log every request as a trace: the question, the chunks retrieved with their scores, the prompt version, the answer, its latency and its cost. Score a weekly sample of those traces with the same judge you use before release. Our evaluation framework sets out the metrics and the release gate. The traces are what feed it after launch.

9. No metadata schema, or a schema nobody enforces

Every document is ingested with minimal metadata, and every query searches the whole corpus. The system returns the 2019 policy when the 2025 version exists, and nothing stops a confidential file surfacing for the wrong user. The fix is a schema designed before ingestion and enforced at ingestion. The four metadata field groups cover dates, access, status and lineage, and the query filters on them before similarity is ever computed.

10. Multi-tenant deployments without proper isolation

What it looks like. Several clients or departments share one vector index, separated only by a metadata tag that a query might omit.

What it costs. A single cross-tenant leak ends contracts and creates legal exposure. The probability can be low and the consequence is still unacceptable.

The fix. Hard isolation: a separate namespace or collection per tenant that no query can cross. Not a filter the application has to remember to apply. Separate caches per tenant. For the most sensitive deployments, separate model instances. Test it with a cross-tenant query that must return nothing.

11. Query transformation missing or too simple

What it looks like. The user's words go straight to the retriever. Short, ambiguous or multi-part questions retrieve poorly.

What it costs. "What about the Q2 numbers?" with no conversation context returns nothing useful. Questions that need two documents return one. Users learn where the system is weak and stop asking it hard things.

The fix. Rewrite the query before retrieval. For conversational systems, fold in the recent history. For multi-part questions, split them into sub-questions and retrieve for each. For abstract questions, HyDE (Gao et al., 2022) has the model draft a hypothetical answer and embeds that instead of the question. The draft reads more like the passage you are looking for. Measure each transformation on your test set. None of them helps every corpus.

12. No evaluation framework before launch

The system is tested by asking it a few questions it answers well, then deployed. Unknown failure modes surface when users find them, and the evaluation gets built afterward, under pressure. Build it first: a 100-question test set and the metrics that gate releases, written by the people who will use the system. Score it with an open-source library such as RAGAS, and check the judge against human grading at least once. Nothing ships if the scores drop.

The pattern behind all twelve

None of these is a hard problem. Each has a known solution that a demo had no reason to include. Demos run on clean data under controlled conditions, and these problems only appear outside them. Teams that reach reliable production fastest treat the twelve as a checklist at the start, not a diagnosis at the end. The build costs more than the demo did, and the cost estimate post shows how to budget for it. Retrofitting after a failed launch costs more still, in money and in the trust of the users who tried the first version.

If you are buying the build rather than making it, the same twelve items are your due-diligence list. The three questions to ask an AI engineering partner are a buyer's version of them.

Frequently asked questions

Can these be fixed incrementally, or does it need a re-architecture? Most can be added to a running system. Seven sit beside the current pipeline: the queue (1), cache (2), hybrid search (5), reranker (6), observability (8), query rewriting (11) and evaluation suite (12). Four usually mean re-ingesting the corpus through the fixed pipeline: document cleaning (3), chunking (4), prompt constraints with an output check (7) and metadata (9). That takes planning, not a rebuild. Tenant isolation (10) depends on how the index was laid out. A phased fix is almost always cheaper than starting again.

Is it cheaper to fix the existing system or replace it? Fix it, in most cases. A system that already has your documents ingested, your integrations built and your users trained is worth keeping. Replace it only when the core architecture cannot accept the fixes, which is rare, or when the original team left nothing that can be maintained.

Before anyone approves a re-architecture, run two tests this week. First, take 50 real queries from the people who use the system, not from the people who built it. Score each answer right, wrong or incomplete. A high share of wrong or incomplete answers points at gaps 3 to 7. Second, put 10 users on the system at the same time and watch latency. If it degrades, you have gap 1 and probably gap 2. Those two afternoons tell you which of the twelve you have, and that list is the plan. If you want the measurement done independently, an AI Audit runs it on your documents for $2,500 fixed.