RAG in production: what actually breaks
Retrieval-augmented generation looks trivial in a notebook. In production, chunking, hybrid retrieval, and evaluation are what actually break — here's what we've learned shipping RAG systems that hold up under real traffic.
Every RAG demo works. Feed a language model a handful of hand-picked documents, ask a friendly question, and the answer arrives clean and confident. The moment a real corpus, real users, and a real SLA arrive, the seams show. This is a field guide to the five failure modes we see most often when a retrieval-augmented generation proof-of-concept moves from a notebook into a system a business actually relies on — and the fixes that hold up under load.
Chunking: why bigger chunks aren't the fix
Chunking is where most RAG pipelines quietly break before anyone notices. Fixed-size splits — 500 tokens, 1000 tokens, whatever number felt safe in testing — look clean on a whiteboard but destroy the semantic boundaries that make a chunk useful as retrieval evidence. A policy clause gets cut in half. A function signature loses its docstring. A table's header ends up three chunks away from its rows.
The instinct when retrieval quality drops is to make chunks bigger, on the theory that more context can't hurt. It usually backfires: bigger chunks dilute the embedding, pulling in irrelevant sentences that drag the vector away from the query's actual intent. The fix that has worked consistently for us is a chunker that respects the shape of the source document — markdown headings, function boundaries, policy clause numbering, table structure — rather than a blind token count. Structure-aware chunking costs more engineering time upfront and pays for itself the first time a support answer needs to cite the right clause instead of a plausible-sounding neighbor.
Retrieval: hybrid search beats a single embedding
Cosine similarity over a single dense embedding rarely wins on its own, no matter how good the embedding model's benchmark scores look. Dense retrieval is excellent at capturing semantic similarity and terrible at exact terms — SKU numbers, error codes, proper nouns, acronyms specific to a client's business. Users ask about exactly those things.
Hybrid search — BM25 keyword matching combined with dense vector retrieval, merged with reciprocal rank fusion — closes that gap. Layer a lightweight cross-encoder re-rank on top of the merged candidate set and, in our experience, the answerable-question rate roughly doubles compared to dense-only retrieval. The re-rank step is cheap relative to generation cost and is the single highest-leverage change most teams can make to an underperforming RAG system.
Prompts: turning chunks into evidence, not decoration
Systems that pipe raw retrieved chunks straight into a prompt template hallucinate under pressure, especially when the retrieved context is thin or slightly off-topic. The model has no signal to distinguish 'this is grounding evidence' from 'this is just more text in my context window,' so it fills gaps with its own training data instead of admitting uncertainty.
Structuring the context changes that. Wrap each chunk with explicit metadata — source, title, timestamp, an excerpt identifier — and the model starts treating retrieved passages as evidence to be cited rather than raw material to be blended into a fluent-sounding paragraph. Pair that with an explicit instruction to answer only from the provided sources, and to say so when the sources don't cover the question, and hallucination rates drop measurably without touching the underlying model.
Evaluation: ship the evaluator before the feature
Without a golden set of question-answer pairs that grows alongside real user queries, every model swap, every chunking change, and every prompt tweak is a coin flip dressed up as an engineering decision. Teams that skip evaluation end up debugging RAG quality by vibes — someone notices an answer 'feels off' in a demo, and a week gets spent chasing a regression nobody can reproduce.
The teams that avoid this build a small evaluation harness before they ship the first version of the feature: a set of representative questions, expected answers or expected source citations, and a scoring pass that runs on every meaningful change. It does not need to be sophisticated. It needs to exist, and it needs to run automatically, so regressions surface in CI instead of in a client's inbox.
Cost: retrieval is a budget line, not just an accuracy lever
Long-context models are seductive. Stuffing forty thousand tokens of context into a single call feels like it sidesteps the entire retrieval problem, and for prototypes, it does. In production, at real query volume, that same approach turns retrieval quality into a line item on the monthly bill. Good retrieval is not only an accuracy improvement — it is what keeps the cost per query sane once traffic stops being a demo number and starts being a real workload.
Getting RAG right in production is less about picking the fanciest embedding model and more about treating retrieval as its own discipline: structure-aware chunking, hybrid search with re-ranking, evidence-structured prompting, continuous evaluation, and cost-aware context sizing. Get those five right, and the rest of the system — the part that actually talks to users — tends to take care of itself.