Why "Just Add RAG" Is Usually the Wrong Answer to a Hallucination Problem

A model states something false with total confidence, and the fix everyone reaches for first is retrieval: give it real documents to ground its answer in, and the hallucination goes away. Sometimes that's exactly right. Often it isn't, because "the model made something up" is a symptom with more than one possible cause, and retrieval only fixes one of them.

The failure mode RAG actually fixes: a genuine knowledge gap

If a model is asked about something outside its training data or that changed after its training cutoff — your company's internal pricing, a policy that shipped last week, a customer's specific account history — it was never going to answer correctly, because the information was never available to it in the first place. This is the case retrieval augmentation was built for, and it works well here: give the model the actual document, and a "no knowledge" failure becomes an "answer from what's provided" success.

The failure mode RAG doesn't fix: weak grounding

A model can be handed exactly the right retrieved passage and still answer incorrectly, because it weighs its own parametric knowledge — what it learned during training — against the provided context, and sometimes the parametric knowledge wins even when it's wrong or outdated for this specific case. This shows up as a model contradicting the very document it was just given, and it's a genuinely different failure than not having the document at all. Adding more retrieval doesn't address it, because the model already had the right context and used it wrong.

The failure mode RAG doesn't fix: weak instruction-following

Related but distinct: a model that isn't actually constrained to answer only from the provided context will happily blend retrieved information with things it half-remembers, producing an answer that sounds grounded and isn't entirely. This is a prompt and system-design problem — explicit instructions to only use provided context, refuse when the context doesn't contain the answer, and cite which passage supports each claim — not a retrieval-coverage problem. Retrieving better documents doesn't fix a model that wasn't going to stick to the documents anyway.

A diagnostic that actually tells you which one you have

Before assuming a hallucination is a knowledge-gap problem, run the failing query with the correct, relevant context manually placed in the prompt — bypass retrieval entirely and hand the model exactly the passage it needs. If it still gets the answer wrong, retrieval was never the bottleneck; the issue is grounding or instruction-following, and improving retrieval quality won't move the needle. If it gets the answer right with the correct context manually provided, the problem really is retrieval failing to surface that context in the first place — chunking, embedding quality, or ranking — and that's worth fixing directly.

Manually inject correct context into the prompt, bypass retrieval:
  - Still wrong  -> grounding / instruction-following problem, not retrieval
  - Now correct  -> retrieval problem (chunking, embedding, ranking)

The takeaway

"Add RAG" is the right fix for exactly one of three hallucination causes, and it's tempting to reach for because it's the most well-known lever. Running the manual-context diagnostic before investing in retrieval improvements — better chunking, reranking, a bigger index — takes twenty minutes and tells you whether that investment will actually fix the symptom you're looking at, or whether the real fix is a grounding-focused prompt and a stricter instruction to stay inside the provided context.

Christian Chukwuka

Founder & AI Systems Engineer