For a wrong RAG answer, first check whether a sufficient supporting source exists, whether retrieval returned it, and whether it survived into the final context. Only then investigate how generation used that evidence. A fluent answer or a high groundedness score does not prove the context answered the requested question.
Consider this fictional support report: A customer asked about the refund window for enterprise annual plans, and the assistant answered with the refund window for monthly self-serve. Fluent, formatted, cited, wrong. Somebody on the thread writes "the model is hallucinating again" and the next two days go into prompt edits.
Prompt changes may miss the problem if the system was given the wrong evidence. Inspect retrieval, context assembly, and generation before attributing the failure to any one layer. This post provides that diagnostic path.
The Complaint: "It Makes Things Up"
"Hallucination" is a useful word for a specific thing: the model asserting something that is not supported by anything in its context. It has become an unhelpful word because it now gets applied to every wrong answer, including the large category where the model faithfully reported a passage that was retrieved and simply was not the right passage.
Those are different bugs with different owners, different fixes, and different costs. Conflating them is expensive because prompt engineering is cheap to attempt and therefore attracts effort that a retrieval fix would have absorbed better.
This article makes no claim about a universal proportion of retrieval failures. Measure your own corpus and question distribution. Inspect the supplied evidence before choosing a fix: that check can identify a missing passage or show that the context needs a different investigation.
The RAG Pipeline As Five Places A Defect Can Live
Draw the pipeline honestly and the failure surface becomes obvious.
| Stage | What happens | What breaks here |
|---|---|---|
| 1. Ingestion and chunking | Documents are parsed, split, and embedded or indexed | The answer is split across a boundary, or lost in bad parsing, or never ingested |
| 2. Query construction | The user turn becomes a retrieval query, possibly rewritten or filtered | Pronouns unresolved, conversational context dropped, wrong or missing filters |
| 3. Retrieval | The index returns candidates | Nothing relevant in the top k; vocabulary mismatch; wrong tenant or version matched |
| 4. Context assembly | Candidates are truncated, ordered, and packed into the prompt | The right chunk is retrieved but dropped by the token budget or buried mid-context |
| 5. Generation | The model reads the context and answers | The model ignores or misreads a passage that was present and sufficient |
The stages separate responsibilities; their count is not evidence about how frequently each one fails. Inspect the actual intermediate outputs to diagnose a complaint.
Authorization must constrain which candidates a request can access, before restricted material reaches an external reranker or generation stage. Retrieval implementations may enforce that boundary in different places; verify the actual contract. It is never something ranking, reranking, prompting, or caching can be trusted to handle, and a "the assistant told me about another customer's contract" incident is a permissions bug, not a relevance bug.
What a fluent wrong answer can hide
Fluency is not evidence of grounding. A capable model handed three passages about monthly self-serve refunds and asked about enterprise annual refunds may produce a confident, well-structured answer about monthly self-serve refunds, because that is the most plausible completion of the material in front of it. The supplied context does not establish the enterprise policy, even if the response sounds confident.
Look for these possible signs, then confirm them in the trace:
- The answer is on-topic but about the wrong entity, plan, version, region, or date.
- The citations, if you show them, are real documents that genuinely say what the answer said.
- Rephrasing the question sometimes fixes it; inspect the trace to determine whether the change affected retrieval, assembly, or generation.
- The answer degrades on specific, narrow, or recent questions and looks fine on broad ones. Check those question types separately rather than assuming an aggregate result describes each one.
Three distinct failures hide behind the same screenshot, and they need to be separated before you can fix anything:
A. Nothing relevant was retrieved. The passage that contains the answer was not in the candidate set at all. A reranker cannot introduce evidence missing from its input. A generator may answer from prior knowledge, but that is not evidence that retrieval succeeded or that a current internal answer is supported.
B. Relevant was retrieved but never reached the model. For a hypothetical example, the chunk came back at rank 14, the context budget took the top 8, and it was truncated away. It existed in retrieval logs but not in the final prompt.
C. Supporting material reached the context, but the answer is wrong. Inspect whether the passage was sufficient, contradicted, truncated internally, or poorly positioned before blaming generation. If sufficient evidence was available and the answer still did not use it correctly, investigate the prompt, ordering, output constraints, and model behavior. Presence in context does not establish that a model read or used it.
Use these as an initial triage, retaining an unresolved state when the supporting source or trace is incomplete. They locate the next investigation rather than proving a single root cause.
The First Check: Was The Right Passage Ever In The Context Window?
This is unglamorous and it settles the question.
Log, per request: the final retrieval query string after any rewriting, every filter applied, the retrieved chunk IDs with their scores and ranks, which of those chunks survived truncation into the final prompt, and their order in the prompt. Use access-controlled trace references rather than copying raw questions, documents, or customer details into general logs. Retain only what is necessary under your data policy, and reproduce with non-sensitive examples where possible.
Then, for a failing example, do this by hand:
Use the recorded identifiers to compare the retrieved candidates with the final prompt’s evidence list in an approved debugging environment. Read the authorized supporting passage and check whether it actually answers the question.
Two booleans at the bottom, and they classify the failure:
| retrieved | in prompt | Diagnosis | Where to work |
|---|---|---|---|
| false | false | Failure A: retrieval miss | Chunking, query construction, filters, retrieval strategy |
| true | false | Failure B: assembly loss | Context budget, truncation policy, reranking, ordering |
| true | true | Generation or evidence-use candidate | Inspect sufficiency, contradictions, prompt, ordering, and model behavior |
If you cannot identify the chunk that should have answered the question, that is its own finding. Either the content does not exist (a documentation problem, not an engineering one) or your chunker mangled it during ingestion, which you verify by searching the chunk store directly for a distinctive phrase from the source document.
Start with a small authorized sample of failing examples and trace them before changing the prompt. The time required depends on trace quality and access to the source. A passage being present is not by itself proof that the context was sufficient or unambiguous.
Five failure classes to investigate
These are diagnostic categories, not a measured frequency ranking.
1. Chunking destroyed the answer
A chunk boundary landing mid-explanation can remove the evidence needed to answer a question. Split a table from its header and the numbers lose their meaning. Split a procedure from its "for enterprise plans only" qualifier and the chunk now confidently answers the wrong question. Chunk too small and no single chunk carries enough context to match the query; chunk too large and the relevant sentence gets diluted by paragraphs about something else, which weakens both lexical and vector matching.
Confirming check: pull the chunks for the source document by hand and read them in order. If the evidence requires several chunks, check that they can be retrieved and assembled together. If parsing lost the needed text entirely, fix ingestion before testing a scoring change. Compare representative questions before and after changing boundaries; keep the source snapshot and judgment rules fixed.
2. Missing or wrong metadata filters
The corpus contains the right answer for enterprise plans, the right answer for self-serve, and versions of both for three product releases. Similarity alone may not enforce the requested plan, version, or date. Investigate structured constraints: attach plan, version, region, product, and effective date as indexed fields, and constrain the query.
Confirming check: take a failing query, apply the filter you believe should have been there, and re-run retrieval. If the right chunk now appears, you have found it. This isolates a structured-constraint issue from a similarity issue; evaluate the effect on other query types before changing the serving configuration.
3. Query and document vocabulary mismatch
Users write "can I get my money back", the corpus says "refund eligibility". Users paste an error code, the corpus describes the error in prose. Lexical retrieval fails on the first case; dense retrieval frequently fails on the second, because exact identifiers are precisely where embedding similarity is weakest. A conversational turn like "what about for the annual one?" is a third variant, where the query sent to the index is missing the subject entirely.
Confirming check: manually rewrite the query into the corpus's own vocabulary and retrieve again. If that fixes it, you have a query-construction problem, and the options are query rewriting, expansion, hybrid retrieval, or better document-side text.
4. Truncation and ordering in context assembly
Your retriever returns 50, your budget fits 8, and nothing between the two is measured. A truncation policy can select what reaches the model even when the retriever returns the right material. Ordering matters too: what you put where in a long context changes what the model attends to, and stuffing more context is not a free improvement.
Confirming check: compare the retrieved-ID list against the in-prompt-ID list across your failing examples. Any case where the right chunk appears in one and not the other is a stage-4 bug with a stage-3 appearance.
5. The model ignored a sufficient passage
This is another plausible failure class, without a claimed frequency relative to the others. It shows up as an answer that contradicts a passage sitting plainly in the context, or an answer that drifts to parametric knowledge when the context was thin but adequate. Confirming check: hand the exact prompt, unchanged, to the model in isolation and see if it reproduces the error. If it does, you own a generation problem and prompt work is finally the right tool.
What To Measure Before You Touch A Prompt
Hand-tracing finds the defect. Measurement is what stops it recurring and tells you whether your fix worked.
The measurement that matters is retrieval recall against known-answer chunks: for a set of real questions, record which chunk actually contains the answer, then measure how often that chunk appears in the candidate set at your retrieval k, and how often it survives into the final prompt. Those rates describe the evidence available for a source-grounded answer; sufficient evidence and correct use still need separate checks. Build the set from real task types, identify supporting passages independently of one retriever, and include answerable and unanswerable questions. Review ambiguous judgments and keep held-out questions for checking a change.
This is also why generation-quality scores can look healthy while retrieval is broken. Faithfulness or groundedness scoring asks "is this answer supported by the provided context?" A confidently wrong answer drawn cleanly from a wrongly-retrieved passage can be faithful to its context even when it fails the requested task. An automated faithfulness score may therefore be favorable without proving correctness. Answer-similarity scoring against reference answers is closer to the truth but blends both layers into one number, so a retrieval regression and a generation regression look identical. If your evaluation cannot separate "the context was wrong" from "the model misused the context", it cannot tell you where to work. Measure the layers separately, always.
Nothing about this makes RAG solved. No architecture eliminates ungrounded answers; retrieval quality sets the ceiling, generation behaviour sets how close you get to it, and both need their own guardrails, their own monitoring, and an honest abstain path for when retrieval comes back empty.
In Plain English
The model can inspect only the external evidence the system actually supplies, alongside information encoded in its parameters. When a retrieval-augmented answer is wrong, the first question is not "why did the model make that up" but "what text was actually in the prompt". Log the retrieved chunk IDs, read them, and compare the evidence that was actually used with what the task required.
This matters most to engineers who own an LLM feature and are getting "it hallucinates" tickets they cannot reproduce, and to leads deciding whether to spend the next sprint on prompts, a bigger model, or the ingestion pipeline. The trace, rather than the symptom alone, should decide where the next experiment belongs.
Use this triage the moment a wrong-answer complaint arrives, before any prompt change. Skip it only when you already have per-layer measurement in place and it is telling you the context was correct.
Next concrete action: take ten real complaints, trace each one to A, B, or C using the two booleans above, and tally them. That tally is your roadmap; it may point at retrieval, assembly, generation, or missing source content.
Next step
Take a small authorized sample of failures and record four facts for each: whether a supporting source exists, whether retrieval returned it, whether it reached the final context, and whether that context was sufficient for the requested answer. Record uncertainty instead of forcing every case into one owner’s queue.
Use RAG evaluation mistakes to turn the trace into separate retrieval and answer measures. Hybrid search and the reranker guide cover candidate coverage and candidate ordering respectively. The Developer Tools category keeps this specialist material together.
Sources checked
Official research and documentation checked 2026-10-17. These sources support the named mechanisms and research context. The selection procedures and trial worksheets are editorial guidance; they are not a report of experiments run by Ranker.dev.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — The research architecture and its reported experimental setting.
- Lost in the Middle — Position-sensitive use of context in the evaluated models; this does not establish the same effect size in every model.
By Ranker.dev · Editorial standards · Suggest a correction
Ranker.dev is an independent software directory and publication. It is not affiliated with Ranker, Inc. or ranker.com.