RAG in production: the pipeline nobody shows you in the demo
Retrieval-augmented generation is the most-shipped and least-understood AI pattern in the enterprise. Here is what breaks between the demo and month six.
Every enterprise AI project eventually meets RAG. Retrieval-augmented generation is the workhorse pattern behind almost every internal knowledge assistant, support copilot, and document summariser shipped in the last two years. It is also the pattern that quietly breaks the most projects, because the demo path and the production path look nothing alike.
Databricks' 2025 State of Data + AI report found that 60% of enterprise GenAI applications in production use retrieval augmentation. The same report notes that data quality and retrieval precision, not model quality, are the top-cited reasons those applications underperform expectations. This is the piece we wish we had when we shipped our first serious RAG system.
What actually sits inside a production RAG pipeline
Ingestion. Documents arrive from ten sources, in ten formats, with three permission models. Somebody has to normalise them, extract the text, preserve the structure, and mint stable IDs. This step alone is usually 30% of the project and 0% of the demo.
Chunking. The naive approach splits on token count. The production approach splits on semantic units, with metadata that lets you reassemble context at retrieval time. Bad chunking is the number one cause of confidently wrong answers.
Embedding and indexing. The vector store is the least interesting part of the system, and the part vendors talk about the most. Pick one that supports metadata filtering and hybrid search, and move on.
Retrieval. Hybrid retrieval, keyword plus vector, wins in almost every real corpus. Pure semantic search sounds elegant and misses the exact phrase the user typed.
Re-ranking. A small cross-encoder or an LLM re-ranker takes the top 50 and returns the top five. This one step usually moves precision at rank one by 10 to 20 points and is worth the extra latency in almost every enterprise application.
Generation with citations. The model answers and cites the chunks it used. No citations, no answer. This is a policy, not a feature.
Evals. A held-out set of real questions with graded answers, run on every deploy. If your RAG system does not have this, you do not have a RAG system, you have a demo.
The failure modes that only appear at scale
Permission bleed. A chunk from a document the user should not see leaks into the answer. Fix at ingestion by tagging chunks with the source ACL, and at retrieval by enforcing the filter before, not after, ranking.
Freshness drift. The index is rebuilt weekly, the source is edited daily, and answers slowly go stale. Fix with incremental indexing and a per-document 'last synced' surface on the citation.
Chunk hallucination. The model confidently references a chunk that does not exist because the citation was generated, not retrieved. Fix by only allowing citations from a whitelist of retrieved IDs, enforced in code.
Prompt bloat. Every new failure mode adds three lines to the system prompt until latency and cost double. Fix with a scheduled prompt review and a rule that new failure modes must be handled by evals or retrieval changes first, prompt changes last.
The numbers to watch
Retrieval precision at rank five. If the right chunk is not in the top five, the model cannot answer, no matter how good it is.
Answer groundedness. What percentage of factual claims in the answer can be traced back to a retrieved chunk. Anything under 90% is a support ticket waiting to happen.
Deflection rate. What percentage of user sessions end without a human handoff. This is the business metric that pays for the project.
Cost per resolved query. Model cost plus retrieval cost divided by resolved queries. Track weekly.
When RAG is the wrong answer
If the answer changes based on live data (inventory, prices, account balances), RAG is not what you want. You want function calling against the source of truth, with the model as the interface layer, not the memory layer.
If the answer requires reasoning over long chains of documents rather than lookup, RAG will disappoint. You want a summarisation pipeline that pre-computes structured notes, and RAG on top of those notes.
If the corpus is small enough to fit in the context window at a reasonable cost, skip the vector store entirely. Long context has quietly killed a lot of the case for retrieval on small corpora.
The handover that keeps RAG healthy
A named owner for the corpus. Not the model, not the pipeline, the corpus. The person who decides what gets ingested, what gets deprecated, and what the answer to a policy question should be.
A weekly retrieval quality report. Ten sample questions, human graded, published to the team. This is the artefact that keeps drift honest.
A published escalation path. When the assistant is wrong, the user knows exactly where to send the correction and how quickly it will be reflected. This is what turns a chatbot into a system people trust.
For the broader context on where a RAG pipeline sits in an operating system, see our piece on one connected AI system beats twenty disconnected tools.
Founder, TeknonOS · Physician-operator writing on AI systems for real businesses. If any of this rings true for your business, connect on LinkedIn or book a call and we will walk through it with you.
Follow on LinkedInOne operating system beats ten disconnected tools
Why automations fail when nobody owns the control plane between departments, and what to build instead.
AI agents vs workflows: when to use which in 2026
Autonomous agents are the headline. Deterministic workflows still ship most of the value. Here is how to pick without wasting a quarter.