Use public retrieval benchmarks to shortlist candidates, not to predict your production ranking directly. Read the task, corpus, relevance judgments, metric, model version, and evaluation conditions. Confirm the decision with local held-out queries, including cases that differ from the benchmark distribution.
In a hypothetical selection mistake, a team swaps an embedding model because the new one sits higher on a public leaderboard. The migration costs two weeks of re-embedding and a careful index cutover. Then the offline numbers on their own query set move by roughly nothing, and the support queue does not change at all.
Nobody did anything dishonest here. The leaderboard was reporting real measurements on real datasets. The mistake was treating a score computed on someone else's corpus, with someone else's judgments, as a prediction about a corpus the benchmark has never seen.
What a benchmark is evidence of
A relevance benchmark is a frozen triple: a document collection, a set of queries, and a set of relevance judgments (qrels) that say which documents count as relevant to which query, usually on a graded scale. A score is the output of a metric applied to that triple.
That gives you exactly one thing: evidence that a system ordered those documents for those queries in a way that agreed with those judgments, under whatever metric was chosen.
It is not evidence about:
- your document collection's vocabulary, length distribution, or structure
- your users' query formulations
- your definition of relevant, which is a product decision and rarely matches an annotator guideline written for a shared task
- your latency budget, index size, or serving constraints, which a headline quality score does not establish
The honest framing is that a benchmark is a task-specific test with coverage limits. It rules candidates out cheaply. It does not rule one in.
Read the task before the score
The BEIR research evaluates retrieval across heterogeneous settings. That is a useful reason to inspect multiple tasks rather than treating success on one collection as universal. It does not establish performance on a private collection that was not evaluated.
MTEB's original paper evaluates text embeddings across several task types, including retrieval. A broad embedding aggregate and a retrieval-specific result answer different questions. For a search shortlist, inspect the retrieval tasks and their individual results rather than treating an all-task embedding score as a search-quality measurement. BEIR and MTEB are useful evidence sources; neither evaluates your private workflow merely by naming a similar domain.
For a candidate benchmark, write down its document type, query source, judgment method, metric, and model-selection protocol. A short-passage question-answering task can differ from searching long internal procedures even when the subject matter overlaps. A multilingual result is informative only about the evaluated languages and tasks.
Aggregates hide variance. Find the relevant per-task results before quoting the average, and check whether the evaluation is supervised, zero-shot, or otherwise adapted to the task. Dataset size and licensing must come from that dataset’s own current documentation, not from a general comparison table.
Judgment depth and the unjudged-document problem
This is a failure mode that can make leaderboard order misleading, and it is worth understanding mechanically.
Some collections use pooling to make judgment work manageable: run a set of systems, take the top k results from each, judge the union, and treat everything unjudged as non-relevant. That is a reasonable compromise, and it has a direction of bias.
Suppose a topic has 30 truly relevant documents. The pool judged 20 of them because those were the ones the original pooled systems surfaced. Now you evaluate a newer system with a different retrieval prior. Suppose its top 10 contains five judged-relevant documents and five genuinely relevant documents that were never judged. Under a convention treating unjudged results as non-relevant, the latter five receive no relevance credit. Its measured precision@10 is 5/10 rather than 10/10.
Incomplete judgments can penalize systems that find useful documents outside the original pool, depending on how the metric treats unjudged results. Dense retrievers evaluated on pools built largely from lexical systems is the textbook version of this, and the same logic applies to any new architecture on any old collection. Some collections and metrics address this with condensed-list evaluation or explicit treatment of unjudged documents; whether a given leaderboard does is something you have to check rather than assume.
Two practical consequences:
- Small score differences on a shallowly judged collection require an uncertainty and judgment-coverage check. Do not infer a meaningful ordering from rank alone.
- Age alone does not establish pool bias. Inspect how judgments were collected, whether they were expanded, and how unjudged results are handled.
Domain, language, and document-length mismatch
Even with perfect judgments, a score transfers only as far as the distributions match. The mismatches worth checking before you trust a number:
- Vocabulary overlap. Does the benchmark corpus contain your product names, part numbers, internal acronyms, ticket IDs? Identifier-heavy corpora behave very differently from prose.
- Document length and granularity. Passage-level collections reward models tuned for passages. If you index whole documents or long chunks, you are measuring a different task.
- Query length and formulation. Well-formed natural-language questions are a different retrieval problem from two-word keyword queries or copy-pasted error strings.
- Relevance definition. Some collections judge topical aboutness. Your product may need answer-bearing, recent, in-stock, or permitted-for-this-user. A model can be excellent at aboutness and useless to you.
- Language and script. Aggregate multilingual scores can be carried by high-resource languages. Read your language's row.
- Corpus size and hardness. Corpus size and the presence of difficult near-matches can change the task and the ordering of systems. Size alone does not establish difficulty.
A quick self-check: write one sentence describing the benchmark's documents and one describing yours. If a colleague could not tell they were different domains from those two sentences, transfer is plausible. If the sentences are obviously about different worlds, treat the benchmark as a smoke test only.
Contamination: benchmarks in training data
Modern retrieval and reranking models are trained on very large web-scraped corpora, and public benchmarks live on the public web. Queries, passages, and in some cases the qrels themselves can end up in pretraining or fine-tuning data.
Why this produces gains that do not reproduce on your corpus:
- The model may have memorized query-document associations rather than learned a transferable relevance function. Memorization can inflate a benchmark score without demonstrating transferable behavior on unseen documents.
- Contamination is invisible in the score. A contaminated result and a genuinely strong result look identical on the leaderboard.
- It is asymmetric across candidates. A newer model trained on a more recent crawl may have absorbed more of a given benchmark than an older one, so the comparison is not apples to apples.
- Some training pipelines deliberately include benchmark training splits, which is legitimate and disclosed, while test-split leakage is neither. Distinguishing the two requires the model's data documentation, which is often incomplete.
What you can actually do:
- Prefer candidates whose training-data provenance is documented, and read the documentation rather than the marketing page.
- Weight benchmarks released after a model's training cutoff more heavily than older ones, while noting that cutoffs are self-reported.
- Treat your own held-out eval set, built from your own corpus, as the a task-relevant independent check, provided it is held out from tuning and its provenance is understood. This is the strongest argument for building one.
This section makes no claim about how common contamination is in any particular model family, and it does not need one: the score cannot tell you either way, and the three steps above work whether contamination is common or rare.
Reading a leaderboard responsibly
A short protocol that costs ten minutes and prevents most bad decisions:
- Check the date on everything. The leaderboard snapshot date, the model's release date, the dataset's release date. A score with no date is not a citable fact.
- Find the per-task breakdown. Ignore the composite. Locate the two or three sub-datasets nearest your domain and read only those rows.
- Check the metric and the cutoff. NDCG@10 and Recall@100 answer different questions. A model that wins one may lose the other, and which one matters depends on whether you rerank downstream. If you are shortlisting a first-stage retriever feeding a reranker, recall at your candidate depth is the relevant column.
- Check who ran it. Self-reported vendor numbers on a vendor's own harness are a different evidence class from an independent reproduction. Neither is disqualifying; the labels are just different.
- Check reproducibility. Is there a harness, a version pin, a command? If a result cannot be re-run, it is a claim, not a measurement.
- Look at the gap size, not the rank. Position 1 and position 6 may be separated by less than the collection's judgment noise.
- Note serving cost. Leaderboards rarely publish dimensionality, model size, or latency. A model that wins by a hair and costs several times more per query is not a win for you.
If you plan to quote a number in a design document, write it as: metric, cutoff, dataset, dataset version, model version, harness, who ran it, run date. If important fields are missing, state the limitation and do not use the rank alone as a selection verdict.
Using benchmarks well: shortlisting, not deciding
Here is the rule the rest of this post exists to support:
Benchmarks shortlist candidates. Your own eval set decides.
Concretely, a defensible model-selection process looks like this:
- Filter on constraints first. Licence compatibility, self-hostability, embedding dimensionality and its index-size implications, max sequence length, language coverage, latency class. This often eliminates most of the field before quality enters the picture.
- Use public benchmarks to cut the survivors to two or three. Read per-task rows near your domain, at your relevant metric and cutoff. Treat close scores as ties and break ties on cost or operational simplicity.
- Run the shortlist on your own eval set, on your own corpus, with your own judgments. This is the step that produces the decision. The guide to choose an embedding model by measuring it walks the mechanics, and the result still needs a safe production rollout.
- Include per-query win/loss review, not just the aggregate delta. A candidate that wins the mean while destroying your ten highest-traffic queries is not the candidate.
- Price the migration. Re-embedding, index rebuild, and cutover are real costs. A tie on quality plus a large migration cost means keep what you have.
A representative local evaluation answers a different question from a public benchmark. A small or biased local set can still produce a poor decision, so assess its coverage and uncertainty too. If you lack a local evaluation, define and build one before treating a public score as your deployment verdict.
Keep the dataset’s provenance with the result
Record the official dataset source, version, included artifacts, and applicable usage documentation alongside the experiment. A paper about a benchmark and access to every underlying document are different things. This site does not mirror or grant rights to benchmark data.
Do not assume that a query file, document collection, and judgment file share identical terms. Resolve permitted access and use for the actual artifact before acquiring or sharing it through your organization’s normal review process. This article makes no dataset-specific licensing conclusion.
In Plain English
A public relevance benchmark is a fixed exam that someone else wrote, graded with someone else's answer key. Doing well on it means a system passed that exam. It does not mean the system will do well on the exam your users give it every day, because your documents, your queries, and your definition of a good answer are all different.
Three issues to investigate are: the answer key is incomplete, so genuinely good results get marked wrong (shallow pooling); the exam leaked into the model's study material (contamination); and the exam is about a subject you do not teach (domain mismatch).
This matters most to whoever is choosing an embedding model, a reranker, or a retrieval architecture and is being handed leaderboard screenshots as justification. Use benchmarks early, to cut a long list of candidates down to two or three cheaply. Do not treat those results as the final decision. Build a small eval set from your own corpus and your own queries, run the survivors on it, review the per-query wins and losses, and let that decide.
Next step: if you have no local eval set, that is the gap to close before the next model comparison. If you do have one, add a line to your model-selection template recording which benchmark rows you used for shortlisting and their dates, so the reasoning is auditable six months from now. Keep the selection notes with the Developer Tools reading path.
Build the set that actually decides
Use representative questions and supporting passages from the intended task, keep a held-out portion, review disagreements, and record missing judgments. The embedding-model selection guide includes a local judgment procedure and a comparison setup. It does not require a Ranker.dev harness or dataset.
The same caution applies when choosing everyday apps: compare the actual job instead of converting a headline ranking into a universal recommendation.
Sources checked
Official research and documentation checked 2026-10-17. These sources support the named mechanisms and research context. The selection procedures and trial worksheets are editorial guidance; they are not a report of experiments run by Ranker.dev.
- MTEB — Multi-task text-embedding evaluation; distinguish retrieval results from a broad aggregate.
- BEIR — Heterogeneous retrieval evaluation and transfer across tasks.
- Sentence-BERT — Sentence embeddings and similarity-based comparison.
- Dense Passage Retrieval — Dual-encoder passage retrieval and evaluation in the paper’s stated setting.
By Ranker.dev · Editorial standards · Suggest a correction
Ranker.dev is an independent software directory and publication. It is not affiliated with Ranker, Inc. or ranker.com.