Building Production-Grade RAG Systems: The Architecture Decisions That Actually Matter

Every RAG tutorial follows the same shape: load a PDF, chunk it, embed it, drop it in a vector store, retrieve top-k, stuff it into a prompt. It works great in a demo with twelve pages of clean text. It falls apart within a month of real usage — not because the model is weak, but because the retrieval pipeline around it was never designed as a system.

Chunking is not a solved problem

Fixed-size chunking with a sliding window is the default in almost every framework, and it is the first thing that breaks. A 500-token chunk boundary that splits a table in half, or separates a clause from the condition that governs it, produces retrieval that looks correct and is quietly wrong.

• Semantic chunking (splitting on structural boundaries — headings, list items, table rows) outperforms fixed windows on anything structured: contracts, invoices, technical docs.
• Overlap should be a function of document type, not a global constant. Narrative text tolerates 10-15% overlap; structured data often needs none, and instead needs metadata (page, section, parent heading) attached to every chunk.
• Chunk size is a latency and cost decision as much as a retrieval-quality one — larger chunks mean fewer retrieved items fit in context, and more tokens billed per query.

Why pure vector search fails in production

Dense retrieval is excellent at semantic similarity and bad at exact match. Ask it to find "invoice INV-2291" and it will happily return five semantically related invoices that are not INV-2291, because embeddings do not privilege exact tokens. Any production RAG system that handles IDs, SKUs, ticket numbers, or names needs a lexical layer, not just a vector one.

Hybrid retrieval: BM25 + vectors + reranking

The pattern that actually holds up is hybrid search — combine a lexical scorer like BM25 with dense vector similarity, merge the candidate sets, and pass the merged set through a cross-encoder reranker before it ever reaches the LLM. The reranker is doing the expensive, accurate comparison on a small candidate set instead of the whole corpus, which keeps latency bounded.

candidates = bm25_search(query, k=50) + vector_search(query, k=50)
candidates = dedupe(candidates)
ranked = reranker.score(query, candidates)  # cross-encoder
top_k = ranked[:8]
context = build_context(top_k, max_tokens=3000)

Evaluation is the part everyone skips

Teams ship RAG systems with no regression test suite, then get surprised when a prompt tweak silently degrades retrieval quality for an entire category of queries. A minimal eval harness needs retrieval-level metrics (precision@k, recall@k against a labeled query set) separate from generation-level metrics (an LLM-as-judge grading faithfulness and completeness). Conflating the two hides which layer actually broke.

Cost and latency are architecture decisions, not afterthoughts

Every additional reranking pass, every larger context window, every synchronous embedding call on the request path is a latency and a cost line item. The teams that get this right treat retrieval depth as a tunable parameter validated against the eval harness, not a constant picked once and forgotten.

The takeaway

RAG that works in production is built by people who treat it as an information retrieval system with an LLM at the end, not a prompt-engineering exercise with a database attached. Chunking strategy, hybrid retrieval, reranking, and continuous evaluation are the actual engineering — the prompt is the easy 10%.

Christian Chukwuka

Founder & AI Systems Engineer