Add a reranker when relevant evidence is already in the candidate pool but its order is poor, and a measured ordering gain fits your latency and cost budget. Fix missing candidates first. Measure end-to-end behavior and failure fallback under the actual workload rather than inferring it from single-pair timing.
A reranker adds a scoring stage and an operational dependency. Its benefit needs to justify the extra serving cost, waiting time, and recovery path. The decision below separates candidate coverage from ordering so the added stage addresses the actual failure.
This guide will not tell you whether to add one. It gives you three things to check first, a cost formula you fill with your own numbers, what to try before reaching for a reranker at all, and the situations where the answer is clearly no. The quality benefit is corpus-dependent, so nobody can size yours without your data. The workload can be estimated in advance; cost and latency still need measurements and the actual deployment’s billing rules.
What a reranker can and cannot fix
A reranker is a second scoring pass. Your retriever returns a candidate list; the reranker scores each candidate against the query with a more expensive model and reorders them. That is the entire mechanism, and it bounds what a reranker can do.
It can fix ordering problems. If the right document sits at rank 30 and needs to be at rank 2, a reranker is the standard tool. It reads the query and the document together and can pick up interactions a vector comparison cannot, in a cross-encoder-style reranker. Other reranking architectures have different computation patterns; this guide’s capacity discussion focuses on scoring query-document pairs.
It can fix precision at small k. If you show three results and the third is usually junk, or you pack five passages into a context window and two are noise, reordering the candidate pool is directly relevant.
It cannot add a missing candidate. If the right document is not in a fixed candidate list, the reranker never sees it. Reordering can improve recall at a smaller displayed cutoff; it cannot change which documents were supplied in that full pool. Scoring the wrong hundred documents very carefully produces a beautifully ordered list of wrong documents.
It cannot fix the corpus. If the answer is not written down anywhere, or exists only in a stale version, no scoring model invents it.
It cannot fix filtering or permissions. Enforce authorization before restricted documents reach an external scorer or generator, using the system’s established permission boundary. A ranking score or prompt instruction is not an authorization decision. If restricted documents are reaching your reranker, that is a security bug, and demoting them is not a fix.
It cannot fix an unmeasured pipeline. If you cannot measure the difference, you will be defending a permanent latency cost with a feeling.
Precondition 1: is candidate evidence coverage the real problem?
This is the first check and it disqualifies a large share of reranker projects. It requires judged examples from your own task.
For this initial check, candidate evidence coverage is the fraction of queries for which the correct document appears anywhere in the candidate list your retriever hands to the next stage. This bounds how often a reranker can place at least one of those relevant candidates in the output. If the fixed input contains supporting evidence for six questions in ten, reordering alone cannot supply evidence for the other four. This is not an upper bound on answers a generator might produce from prior knowledge, and it does not establish that one passage is enough for a multi-part question.
How to measure it:
- Take a query set with judgments. If you do not have one, identify representative questions and supporting passages using the local judgment procedure.
- For each query, retrieve your usual candidate depth.
- Ask a single binary question per query: is at least one document graded relevant present anywhere in that list?
- The share answering yes is candidate answer coverage at that depth. This is a hit-rate measure, distinct from document recall: it asks whether at least one relevant document is present, not whether all required evidence was found. Multi-document answers need an additional completeness check.
Then read the result:
- Candidate coverage is low. The problem is upstream. Fix candidate generation: analysis and tokenisation, filters, chunking, field weights, hybrid retrieval. All of those are in the cheaper-alternatives section below.
- Candidate coverage is high but the right document is often deep in the list. This is the reranker's home ground. Measure both candidate coverage and ordering at the depth your interface uses. Do not subtract metrics with different definitions to claim an achievable gain.
- Candidate coverage is high and the right document is already at rank 1 or 2 most of the time. Little to win, full cost to pay. Check whether the failing queries are a distinct class, because routing may serve you better.
Do this before anything else. Every hour spent here is an hour not spent tuning a component that was never the constraint.
Precondition 2: do you have latency headroom?
Reranking adds a serial stage. It cannot start until retrieval finishes, and generation or rendering cannot start until it finishes. Whatever it costs is added to your critical path, and its effect on tail latency must be measured under the concurrency you intend to serve.
Work out the budget explicitly:
- What is your current p95 end-to-end, and what is the ceiling you are held to? Use that difference as a planning check, not as a guarantee: stage percentiles do not generally add or subtract into a reliable end-to-end percentile.
- Where does the current p95 go? Measure whether an extra stage fits the remaining user-facing budget; the largest existing stage alone does not decide the tradeoff.
- Is this search-as-you-type, a results page, or a RAG turn where the user is already waiting seconds for tokens? These have very different tolerances; derive the budget from the actual user experience rather than assuming RAG makes additional waiting acceptable.
- Does your traffic have bursts? A stage that is comfortable at median load can queue at peak, and queueing delay is not linear. Measure whether the resulting queue delays remain inside your user-facing budget.
Also budget for the failure path before you deploy, not after. A reranker is a network hop or a GPU-bound local call, and both fail. You need a timeout shorter than your headroom, and a fallback that returns the retriever's original ordering. Without those, a slow reranker becomes an outage. Test timeout and fallback behavior under load before making the added stage load-bearing.
Precondition 3: can you measure the difference?
If you cannot measure it, do not ship it. Not as a principle, as economics: you are taking on a permanent recurring cost in exchange for an effect whose size you cannot state.
The minimum kit:
- A query set with judgments, pooled across both the reranked and un-reranked orderings, since otherwise you penalise whichever system surfaces documents the other never showed.
- A rank-sensitive metric. Reranking changes order, not membership, so recall at your candidate depth will not move at all. Evaluate with recall alone and you will measure exactly zero improvement from a working reranker.
- A noise floor, so you can tell a real difference from resampling luck.
- Latency measured at p95 and p99 under realistic concurrency, not a single-request timing on an idle box.
And a decision rule written down before you look at the numbers: how much quality gain justifies how much latency and how much spend. Deciding that afterwards is how a marginal metric move gets talked into justifying a doubled p95.
The cost structure: candidates × queries, then measured batches
Separate the amount of work from its wall-clock time. Let N be candidates per query and Q be queries per second. The required scoring workload is Q × N query-document pairs per second for a pair-scoring reranker. This is not yet a latency estimate or a vendor bill.
Batching changes the timing. Let B be pairs in one batch and t_batch be the measured wall-clock time for that batch at representative input lengths. A simple sequential-batch approximation is:
- Batch count per query:
ceil(N / B). - Approximate query service time: batch count multiplied by
t_batch. - Required pair workload per second:
Q × N.
The service-time estimate excludes waiting for a worker, network calls, serialization, and other overhead. It assumes comparable batch times; smaller final batches and variable document lengths can break that approximation. Do not substitute single-pair latency for full-batch latency while also assuming the batch runs all pairs concurrently.
With explicitly hypothetical inputs, take N = 100, B = 32, and a full-batch time of 18 ms. Four batches suggest 72 ms of service time before overhead. At N = 400, thirteen batches suggest 234 ms. These are illustrative calculations, not measured product performance. At 20 queries per second and 100 candidates, the workload is 2,000 pairs per second.
One worker processing 32 pairs per 18 ms would have an idealized capacity of about 1,778 pairs per second before other costs. That is already below the hypothetical workload. Adding workers or changing batch size needs fresh measurements; a single-request demo cannot establish stable peak-load latency.
For a hosted service, apply the actual billing units to the request volume and input sizes. For self-hosting, include hardware capacity, idle capacity, operations, and evaluation work. Per-query pair counts can scale linearly while money and tail latency do not. Choose the acceptable quality, latency, and cost tradeoff before reading the results.
Alternatives to test before another scoring stage
Inspect text analysis and literal identifiers; confirm document version and task filters; revisit chunk boundaries; and compare field weighting with the same judged set. These can change candidate quality before reranking sees it. They are experiments, not guaranteed free improvements: reindexing, added fields, and query changes have storage, latency, and maintenance consequences.
If lexical retrieval misses paraphrases or dense retrieval misses important literal matches, evaluate hybrid search. If only one identifiable query class needs reranking, investigate a bounded route for that class and measure classification mistakes too. Do not assume routing is cheaper once its maintenance and fallback are included.
Enforce authorization before restricted material enters an external scoring or generation stage. A rank score cannot establish permission. Use harmless samples and approved environments for experiments, retain the serving baseline, and plan recovery for any indexing or configuration change.
A decision table: your situation → recommended move
| Your situation | Next experiment | Reason |
|---|---|---|
| Supporting evidence is absent from candidates | Inspect ingestion, filtering, and retrieval | Reordering a fixed pool cannot introduce missing evidence |
| Evidence is present but poorly ordered | Trial a reranker against the baseline | This is an ordering problem the stage may help |
| Most queries work but one segment fails | Evaluate that segment separately | A mean can hide both improvement and harm |
| No judged questions or reliable outcome measure | Build a local evaluation first | An unmeasured gain cannot justify an ongoing cost |
| A measured gain exceeds your latency or cost budget | Sweep depth, input length, batching, or model | Each changes the quality and capacity tradeoff |
| The gain is uncertain or fallback is unsafe | Keep the baseline while investigating | Offline improvement alone does not establish production readiness |
Corpus size by itself does not decide the question. A small set can contain many confusing near-matches, and a large set can already have a satisfactory ordering. Judge the actual failure, workload, and user requirement.
When to defer a reranker
Defer it when missing candidates are the binding problem, when no local evaluation can demonstrate a useful ordering change, or when the deployment cannot stay within its measured latency and cost budget. Also defer it when there is no safe timeout, passthrough fallback, or way to disable the stage.
Do not reject or adopt a reranker solely because the product is autocomplete, RAG, or a small corpus. Those labels do not establish a budget or a relevance gain. Test the relevant workflow and make the decision on the observed result.
In plain English
A reranker reorders the shortlist your retriever already produced. For a pair-scoring model, more candidates and more queries mean more pairs to score. Batch behavior, input lengths, queueing, and the serving arrangement determine how that workload becomes waiting time and cost.
Start by confirming that useful evidence reaches the shortlist. Then compare the reranked output with the baseline on the same questions and measure the complete failure path as well as the successful request. Adopt the stage only when the measured benefit fits your constraints.
Your next step
Make a small decision record containing the baseline, candidate coverage, ordering result, measured batch timings, peak workload, cost assumptions, and fallback behavior. Keep it with the versioned configuration; GitHub is one cataloged option for software collaboration.
Read RAG evaluation mistakes when the final output is an AI answer. For the broader buying decision, use fair app comparisons rather than an unexplained overall score.
Sources checked
Official research and documentation checked 2026-10-17. These sources support the named mechanisms and research context. The selection procedures and trial worksheets are editorial guidance; they are not a report of experiments run by Ranker.dev.
- Passage Re-ranking with BERT — A query-based passage reranking approach, not a latency guarantee for another deployment.
- BEIR — Heterogeneous retrieval evaluation and transfer across tasks.
By Ranker.dev · Editorial standards · Suggest a correction
Ranker.dev is an independent software directory and publication. It is not affiliated with Ranker, Inc. or ranker.com.