Evaluate retrieval and generation separately, then check end-to-end correctness and refusal behavior. Include independently judged evidence, held-out questions, and questions the corpus cannot answer. A favorable groundedness score only concerns support in the supplied context; it cannot replace task correctness or retrieval coverage.
Consider a hypothetical failure: A user asked about parental leave for contractors, the assistant produced four confident paragraphs, and every one of them described the policy for full-time employees. Meanwhile the evaluation suite is green: faithfulness high, answer relevance high, no regression on the last three runs.
Both observations are correct. The suite is measuring the generator, and the answer could be well supported by the wrong document. This walks through six evaluation mistakes that produce exactly that gap, each with what the metric can see and what it structurally cannot, and ends with a layered scorecard that closes the blind spot.
The framing to hold onto: a RAG pipeline is at minimum a retrieval system and a generation system in series. A metric applied to the output of the second one cannot attribute failure to the first. An output score can conceal retrieval defects or identify an incorrect answer without explaining the layer responsible.
Mistake 1: only scoring the final answer
One possible setup is a set of questions, a set of reference answers, and a scorer that compares them. It produces one number per question and it feels complete.
What it can see: whether the final text resembled the expected text. That is genuinely useful and worth having.
What the final score alone does not establish: why a low or high score happened. A final answer can arise through several mechanisms that an answer-only score does not distinguish:
- The right chunk was retrieved and the model used it.
- The right chunk was retrieved and the model ignored it, answering from parametric knowledge that happened to agree.
- A wrong chunk was retrieved and the model answered from parametric knowledge anyway.
- Several chunks were retrieved, one of them right, and the model happened to pick it.
The fourth can also be a successful case if the evidence is sufficient and the model uses it appropriately. The second and third do not demonstrate useful reliance on retrieval, and you will discover it the day a question falls outside the model's training data, or the day the underlying policy changes and the model keeps answering from memory.
This mechanism also makes answer-only scoring optimistically biased on any corpus that overlaps public knowledge. A general-purpose model already knows a lot about parental leave in the abstract. It knows nothing about your contractor policy. Your eval set, if built from generic questions, will heavily sample the first category and under-sample the second.
The fix is not to abandon answer scoring. It is to stop treating it as a diagnosis. Score the layers separately, then read them together.
Mistake 2: measuring groundedness and calling it relevance
This one is subtle enough that it survives in mature suites for years.
Groundedness (also called faithfulness, or attribution) asks: is every claim in the answer supported by the retrieved context? It compares the answer against the context.
Correctness asks: is the answer true, and does it answer the question the user asked? It compares the answer against reality, usually via a reference answer.
These are different questions and they come apart in a specific, predictable way. Consider the contractor leave question above. Retrieval returned the full-time employee policy. The model read it carefully and produced an answer entirely supported by that document, citing it accurately, inventing nothing.
Under an ideal rubric, the answer could be fully supported by the supplied context and still be incorrect for the question. An automated scorer is not guaranteed to identify either condition perfectly.
A groundedness metric is structurally incapable of noticing this, because the wrong context is still context. Groundedness evaluates support relative to the supplied material; relevance and correctness need their own task-aware checks.
Groundedness becomes misleading when it is read as a complete measure of task correctness. It addresses support for claims relative to supplied evidence, often one of the concerns behind the word “hallucination.” A favorable score still needs calibration and task context. It does not, by itself, establish whether the answer is correct for the requested entity and scope.
Keep groundedness. It catches a real failure mode: the model drifting away from provided evidence. Just never read it as a statement about retrieval, and never let a groundedness number stand in for a correctness number on a dashboard.
Mistake 3: eval questions generated from the retrieved chunks
Building an eval set is tedious, so a natural shortcut appears: sample chunks from the index, ask a model to write a question each chunk answers, and use those pairs as ground truth. Retrieval is then evaluated on whether it finds the source chunk.
This can create two kinds of evaluation bias; inspect whether either occurs in your generated set.
Vocabulary circularity. A generated question may inherit the source chunk’s terminology. For example, it might repeat a formal policy term while a user asks an informal question. That can make matching easier than it is on real traffic. Check the generated language rather than assuming every synthetic question represents a user’s phrasing.
Answerability circularity. Every question in the set is, by construction, answerable from exactly one chunk that exists in the index. Real question distributions contain questions answerable only by combining two chunks, questions answerable only by a document nobody uploaded, questions resting on a false premise, and questions that are out of scope entirely. A set constructed only from individually answerable chunks does not automatically include these cases; add and judge them explicitly.
There is a third, quieter problem: the "correct" chunk is asserted, not judged. If an evaluation labels only the source chunk as correct while other chunks could also answer the question, it can penalize an equally good alternative. Your recall number is then partly a measure of near-duplicate density in your corpus.
Synthetic questions are not worthless. They are a reasonable way to get coverage cheaply while you build something better, and they are fine for smoke tests and regression detection. They are not a substitute for questions drawn from real user language, and their absolute values should never be reported as a measure of production quality. If you use them, label them as synthetic in the results, and keep a smaller human-sourced set alongside for calibration. Use questions from the actual task, identify supporting passages, review disputed judgments, and keep a held-out portion for checking changes.
Mistake 4: no recall metric on the retrieval layer
If you take one thing from this post: you need a number that says whether the right material was in the context window at all.
Without it, every other metric is ambiguous. Answer wrong? Could be retrieval, could be generation. Groundedness high but correctness low? Inspect the evidence, task scope, and judgments to determine whether retrieval or another layer caused the disagreement.
The metric is straightforward once you have judgments at the chunk or document level. For each eval question, you know which chunks contain the answer. Retrieval recall at k asks what fraction of those were in the top k passed to the generator.
Worked example with invented judgment data, to fix the arithmetic. Take five questions, and suppose your context window holds the top 5 chunks:
| Question | Relevant chunks in corpus | Relevant chunks in top 5 | Recall@5 |
|---|---|---|---|
| Q1 | 2 | 2 | 1.00 |
| Q2 | 1 | 1 | 1.00 |
| Q3 | 3 | 1 | 0.33 |
| Q4 | 1 | 0 | 0.00 |
| Q5 | 2 | 1 | 0.50 |
Using the exact Q3 fraction, mean recall@5 = (1 + 1 + 1/3 + 0 + 1/2) / 5 = 17/30 ≈ 0.567. The table rounds Q3 for display; the calculation uses the unrounded fraction.
These numbers are fabricated to illustrate the calculation. What matters is what the table lets you say: on Q4 none of the judged evidence reached the generator, and a prompt change cannot make that missing evidence appear in the retrieved set. On Q3, two thirds of the judged relevant chunks were missing; inspect whether the remaining chunk is sufficient, rather than inferring answer completeness from recall alone. Those are retrieval work items, and until you compute this table they are invisible.
Set k to the number of chunks you actually pass to the model, not to some larger number your retriever returns internally. The question the metric answers is "did the evidence reach the generator," and the context window is where that is decided.
One caveat that keeps this honest: recall computed against your own judgments is bounded by the completeness of those judgments. If you only judged chunks that your current system returned, your recall is measured against a pool your system defined, and it will flatter your current system. That is a judgment-coverage problem: pool results from different candidate systems, review newly surfaced passages, and report how much remains unjudged.
Mistake 5: judging with the same model that generated
Using a language model as the judge is a reasonable practice with real limits, and the limits get sharpest when the judge and the generator are the same model or the same family.
The concerns, stated plainly:
- Self-preference. A model asked to evaluate text may score its own output style favorably. Whether and how strongly this occurs depends on the models and the prompt, and the honest position is that it is a known concern rather than a quantified constant.
- Shared blind spots to check. A judge from the same family may share gaps in knowledge or reasoning. If the generator misreads a clause in a policy document, a closely related judge may repeat the error. Measure this on your examples rather than assuming a fixed relationship.
- Correlated failure under distribution shift. Both models can fail on an unfamiliar domain. A judge score is not proof of correctness, so inspect disagreement on the shifted task.
Mitigations, in rough order of cost:
- Use a different model as judge, ideally a different family. This does not eliminate the problem but breaks the tightest coupling.
- Calibrate the judge against humans on a sample. Take a subset of your eval set, have people grade it against your written guidelines, and compare. What you are looking for is systematic disagreement: a judge that is lenient on incomplete answers, or strict on differently-worded correct ones. Re-run this calibration whenever you change the judge model or the prompt.
- Give the judge a rubric, not a vibe. "Rate 1 to 5 for quality" invites the judge's priors. A rubric that names the failure modes ("does the answer state the eligibility condition; does it cite the correct policy version") produces gradings you can audit.
- Keep humans on the hard slice. Ambiguous questions and out-of-scope questions are exactly where automated judging is weakest and where the product's reputation is decided.
Do not report a correlation between your automated judge and human grading unless you measured it yourself on your own set and can state how. Numbers from elsewhere do not transfer, and reporting one you did not compute is worse than reporting nothing.
Mistake 6: eval sets that never include unanswerable questions
An evaluation set containing only answerable questions cannot establish behavior when the corpus lacks an answer. Include those cases to expose the following failure modes.
Over-retrieval. A top-k retriever without an effective relevance threshold can return up to k chunks even when useful evidence is absent. When nothing relevant exists, it returns the k least irrelevant things, with no signal that it did so. Vector search is particularly good at this: every query has a nearest neighbor, and cosine similarity to the nearest neighbor of an unanswerable question can look unremarkable next to a real hit. If your eval set never contains such a question, you have never measured what your system does with the resulting context.
False confidence. The generator receives plausible-looking context and produces a plausible-looking answer. Users cannot tell the difference from the output alone, which is exactly the failure that erodes trust fastest, because it is undetectable to the person least equipped to catch it.
Include at least three kinds of unanswerable question:
- Out of scope. Answerable in general, not by this corpus. ("What is the statutory minimum in Germany?" for a US-only policy set.)
- Not yet in corpus. In scope but genuinely absent. ("What is the 2027 rate?" before that document exists.)
- False premise. Presupposes something untrue. ("How do I claim the contractor housing stipend?" when no such stipend exists.)
Then define what a correct response looks like for each, in writing, before you measure. Usually it is a refusal or a scoped hedge that names what the system does not have, rather than silence. Score it as its own metric: refusal behavior, the fraction of unanswerable questions where the system correctly declined or qualified.
Track the opposite direction too. A system that refuses everything scores perfectly on refusal behavior and is useless. Over-refusal on answerable questions is the accompanying guardrail. Report the pair together so improved refusal behavior cannot hide an increase in inappropriate refusals.
A layered evaluation scorecard
Here is the structure that removes the ambiguity. Each layer answers a question the layer above cannot.
| Layer | Metric | The question it answers | What it cannot tell you |
|---|---|---|---|
| Retrieval | Recall@k (k = context size) | Did the evidence reach the generator? | Whether the evidence was usable, or whether the answer was right |
| Retrieval | Context precision / rank of first relevant chunk | Was the evidence near the top, or buried under noise? | Whether anything was missing |
| Generation | Groundedness | Did the model stay inside the provided context? | Whether the context was the right context |
| End to end | Answer correctness | Was the user's question answered truthfully? | Which layer failed |
| Boundary | Refusal behavior on unanswerable questions | Does the system decline or qualify unsupported answers? | Whether it over-refuses |
| Boundary | Over-refusal rate on answerable questions | Is the refusal behavior too aggressive? | Anything about answer quality when it does answer |
Context precision earns its place because context size is finite. Relevant material that arrives at position 9 of 10, surrounded by near-misses, is present but competing, and models do not attend uniformly across a long context. Recall says it was there; precision and rank say whether it had a fair chance. Check evidence retention after truncation separately from the earlier candidate list.
Read the scorecard as a diagnostic grid rather than six independent numbers:
- Recall low, everything else fine. Retrieval work: chunking, embeddings, hybrid retrieval, query rewriting.
- Recall high, groundedness high, correctness low. Much of the judged evidence was present and the answer appears supported by its context. Check reference judgments, evidence sufficiency, scope qualifiers, and conflicting documents; neither score proves that the context answered the requested question.
- Recall high, groundedness low. Inspect context sufficiency, contradictions, ordering, and generation behavior before attributing the failure to a prompt or model.
- Everything fine, refusal behavior poor. Your system cannot say no. Add abstention handling and a relevance threshold, and expect to trade some coverage.
- Refusal behavior perfect, over-refusal high. You over-corrected.
No single row is sufficient, and no weighted average of the rows is a good idea either. Collapsing this into one composite "RAG score" destroys exactly the attribution the layering exists to provide, and composite scores drift when their components move in opposite directions. Report the layers.
For an individual complaint, use the wrong-document diagnostic to inspect the actual evidence path.
In Plain English
A RAG system finds documents and then writes an answer from them. A suite that only grades writing does not measure every stage of that process. A favorable writing score says little about how the evidence was obtained. It does not tell you the system found the right documents, and a well-written answer built from the wrong document is a serious failure this evaluation design should be able to detect.
Six habits produce that blind spot: scoring only the final answer, treating groundedness as if it measured relevance, generating eval questions from the chunks themselves, having no recall metric on the retrieval layer, judging with the model that generated, and never asking a question the corpus cannot answer.
The correction is structural. Measure each layer with a metric that can see that layer: recall and context precision on retrieval, groundedness on generation, correctness end to end, and refusal behavior plus over-refusal at the boundary. Read them as a grid, not a score.
Do this when your eval dashboard is green and your users are not, which is the exact situation this post exists for. Skip the elaborate version if you are pre-launch with ten test questions, but even then, add the recall column. It adds a direct check on the evidence available to the generator; other measures remain necessary.
Next: take your current eval set, add a recall@k column computed at your real context size, and add ten unanswerable questions. Use those changes to test which layer needs attention, rather than treating a better aggregate answer score as an explanation.
Next step
For each judged question, record candidate retrieval coverage, evidence retained in the prompt, answer support, correctness, and appropriate abstention. Choose the measures and judgment rules before comparing configurations, and keep the source and model versions with the result. Treat unresolved judgments separately from demonstrated failures.
The wrong-document diagnostic connects that scorecard to an individual complaint. Choosing an embedding model covers a controlled selection procedure; benchmark transfer explains why another team’s aggregate result does not make the decision for you.
Sources checked
Official research and documentation checked 2026-10-17. These sources support the named mechanisms and research context. The selection procedures and trial worksheets are editorial guidance; they are not a report of experiments run by Ranker.dev.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — The research architecture and its reported experimental setting.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Reported judge limitations and biases; it is not a calibration result for your own judge.
- Lost in the Middle — Position-sensitive use of context in the evaluated models; this does not establish the same effect size in every model.
By Ranker.dev · Editorial standards · Suggest a correction
Ranker.dev is an independent software directory and publication. It is not affiliated with Ranker, Inc. or ranker.com.