Choose an embedding model by first excluding candidates that fail deployment, language, license, or cost requirements, then comparing the survivors on held-out questions from your own task. Keep the corpus and evaluation method fixed, record retrieval quality and operational cost, and budget the re-embedding migration before choosing.
Sorting a public leaderboard and selecting the highest model you can serve is a tempting shortcut. It provides a shortlist, but it does not establish which candidate retrieves the right evidence for your documents and users.
The alternative is a bake-off on your own documents, and it is less work than it sounds. Building the local evaluation set is a substantial part of the work and is useful beyond this one comparison. What follows is the procedure, including the operational criteria a benchmark score alone does not establish and the migration bill you are quietly signing up for.
Why leaderboard rank is weak evidence for your corpus
A public leaderboard reports how a model performed on specific datasets, with specific queries, judged by specific annotators, at a specific point in time. It is evidence that the model does well on those datasets. Whether that transfers to yours depends on how similar yours is; assess that similarity rather than assuming transfer.
Domain mismatch is one reason transfer can fail. General retrieval benchmarks are built largely from web text, encyclopedic content, and natural-language questions. If your corpus is maintenance logs, clinical notes, statutes, internal wiki pages, or support tickets full of product codes and abbreviations, the vocabulary, the sentence shapes, and the notion of what makes two texts "similar" are all different. A model that separates encyclopedia articles cleanly may collapse ten of your near-identical procedure documents into indistinguishable neighbors, which is precisely the discrimination you need.
Mismatch shows up along several axes at once:
- Vocabulary. Jargon, identifiers, abbreviations, and product names that may be represented differently from the benchmark’s vocabulary.
- Query shape. Benchmarks often use well-formed natural-language questions. Your users may type three keywords, a serial number, or a pasted error message.
- Document length and structure. Short passages behave differently from long structured documents with headings, tables, and boilerplate.
- What counts as relevant. A benchmark’s relevance definition is specific to its task. In your product it might mean "applies to this customer's plan and this software version", which no general model knows about.
- Language mix. A corpus with several languages, or code-switching inside single documents, is a different problem from any monolingual benchmark.
There are more mundane reasons too. Leaderboard positions can be separated by small differences whose stability needs checking on another query sample. Some entries are self-reported. Evaluation-set contamination is a live concern for any model trained on web-scale data. And the numbers age: a score published on one date, with one model version, is not a claim about the version you will download next month. If you do cite a benchmark figure anywhere in your decision document, cite it with its date and version, or do not cite it. The broader question of what these resources are evidence of is covered in what benchmarks do and do not transfer.
Used properly, a leaderboard is a shortlisting tool. It narrows the field from hundreds to a handful. It does not make the decision.
Shortlisting candidates on non-quality criteria first
Filter on constraints before you filter on quality, because a model that fails a hard constraint is not a candidate no matter how it scores. A headline quality score does not answer these operational questions.
| Criterion | Why it decides things | The question to answer |
|---|---|---|
| Max input length | Determines your chunking strategy, and silent truncation destroys recall on long documents | How long are your chunks at the 95th percentile, and does the model accept them without truncating? |
| Dimensionality | Drives index size, memory footprint, and per-query distance computation | What does your index cost at this dimension times your document count? |
| Self-host feasibility | Governs data residency, offline operation, and whether cost scales with usage or with hardware | Can you run it on hardware you have, and are the weights actually obtainable? |
| Licensing | A license that forbids commercial use or requires attribution you cannot give is disqualifying | Have you read the actual license text for the weights, not the marketing page? |
| Versioning policy | A model that changes underneath you invalidates your index without a deploy on your side | Is the version pinnable, and is there a deprecation notice period? |
| Language coverage | English-only evidence does not establish quality in another language | Which of your languages are supported, and how were they evaluated? |
| Throughput and latency | Embedding is on the query path, and the backfill has to finish in a window you can live with | What is your measured throughput at your chunk lengths and batch sizes? |
| Data handling | Sending documents to an external service may be prohibited outright | Does anything in your corpus have a residency, confidentiality, or retention constraint? |
| Symmetry | Some models expect asymmetric query/document handling with distinct prefixes or instructions | Does the intended usage match how you will actually call it? |
Two of these deserve emphasis because they are so often reasoned about backwards.
Dimensionality is a cost, not a quality signal. More dimensions means a larger index, more memory, and more work per distance computation. It does not mean better retrieval. Smaller-dimension models regularly match or beat larger ones on specific corpora, and some models support dimension reduction with modest quality change. Treat dimensionality as a line item in the cost column and let the measurement decide quality.
Versioning policy is the criterion teams discover too late. If the endpoint you call can silently upgrade its model, your stored vectors and your query vectors can drift apart, and no code change on your side triggers it. Pin a version or accept that you have an unscheduled migration in your future.
Shortlist three to five candidates. More than that and the bake-off stops being something you finish.
Build the local judgment set
Sample questions from the actual task, including literal identifiers, natural-language questions, language differences, and cases the corpus cannot answer. Remove unnecessary private information and use an approved access-controlled location for any retained source material. Do not collect production logs just to populate a benchmark.
For each answerable question, identify supporting documents or passages independently of the candidates under comparison. Record the applicable version and a short relevance rule. Have another person inspect an overlapping sample where feasible, resolve disagreements, and keep a held-out portion that you do not use to tune settings.
Pool candidate results for review so a useful passage found only by a new model is not automatically counted as wrong. Report judged coverage and unresolved cases. A small, poorly judged set can mislead too; relevance to your task is necessary but does not replace adequate coverage.
The bake-off setup: hold everything else constant
The purpose of the bake-off is to isolate the effect of the embedding model. That only works if the model is the only thing that differs between runs. This is easy to say and easy to violate.
Hold constant across every run:
- The evaluation set. Same queries, same judgments, same k. If you do not have one yet, build it before the bake-off; use the local judgment procedure below. This is the bulk of the work; keep it versioned and review it when the corpus or user task changes.
- The chunking. Same chunk sizes, same overlap, same boundary rules, same extraction pipeline. Changing chunking between models confounds the comparison: the observed difference may come from the chunks rather than the model.
- The retriever. Same index approach where compatible, supported distance metric, same candidate depth, and same filters. Record model-required changes rather than forcing an invalid configuration for the sake of sameness.
- Any downstream stages. Turn off reranking, query rewriting, and fusion for the comparison, or hold them fixed. You are measuring the embedding model, not the pipeline.
- The corpus snapshot. Same documents. If content changes mid-bake-off, restart.
Vary only: the embedding model, and the mechanically required consequences of that choice: index dimensionality, and any prefix or instruction the model's documented usage requires.
That last point is worth care. Some models expect the query and the document to be encoded differently, or expect a task instruction prepended. Skipping that because the API worked without it means you have measured the model badly and will conclude the wrong thing. Follow each model's documented usage exactly, and note in your results what you did.
A few more mechanics:
- Include your current setup as a baseline row, even if it is lexical-only. "Better than what we run today" is the decision you are actually making, and lexical retrieval is a genuinely hard baseline on corpora full of identifiers and exact terms, which is also why hybrid search belongs in the comparison if exact matching matters to you.
- Run each configuration end to end, from raw documents to results. Do not reuse an index built with different chunking because it was already there.
- Script the whole thing so the run is reproducible and a new candidate is one config entry. You will do this again in a year.
- Record everything: model identifier and version, date, chunking parameters, index settings, eval set version, and the code commit.
Dimensionality, index size, and query latency
Measure the operational cost of each candidate at the same time as its quality, because the decision is a tradeoff and you cannot make it from a single column.
The index-size arithmetic is straightforward. With entirely hypothetical inputs:
- 2,000,000 chunks
- 768 dimensions
- 4 bytes per value at full precision
- 2,000,000 × 768 × 4 = about 6.1 GB of raw vector data
Double the dimensionality and you double that, plus whatever your index structure adds on top, plus the memory needed to serve it. Quantization can cut it substantially at some cost in precision, and whether that cost matters is again corpus-specific, so measure it as a separate configuration rather than assuming.
The numbers to record per candidate:
- Raw vector bytes and total index size on disk
- Resident memory required to serve at your query concurrency
- Query latency at your real concurrency, at the percentile you actually promise, not the median
- Embedding throughput for a full backfill of your corpus, measured yourself
- Per-query embedding latency, since the query itself must be encoded on the critical path
That last one is easy to forget and lands directly in your user-facing latency budget.
Domain, language, and document-length fit
Beyond the aggregate metric, look at where each model succeeds and fails. Segment your evaluation set before you run it, so the breakdown falls out automatically:
- By query type. Exact identifier, short keyword, natural-language question, long pasted text. Test identifiers separately; their literal distinctions can behave differently from paraphrased questions. Do not infer either retriever’s result from the query label alone.
- By language. Report per language. An aggregate over a multilingual set hides a model that is strong in one language and unusable in another.
- By document length. If long documents underperform for a candidate, suspect an interaction between that model's effective context and your chunking rather than a general quality gap.
- By content type or source. Structured reference pages, prose articles, and transcripts behave differently.
- By query frequency. Head and tail queries have different characteristics, and a model that helps the tail while leaving the head alone may be exactly what you want.
You are looking for a model whose failure pattern you can live with, or better, one whose weaknesses your other stages already cover. A dense model that is weak on identifiers is a reasonable choice if you run hybrid retrieval with a lexical component handling those. In isolation the same model would look like a poor fit. Hybrid search explains how literal and semantic matching can complement each other.
Cost of switching later: the migration you are signing up for
This belongs in the decision, not in a footnote afterwards.
Choosing an embedding model commits you to a vector space. Changing it later means re-embedding every document, building a parallel index, keeping it synchronized against live writes during a backfill that may run for days, gating the cutover on measurements, and holding a tested rollback the whole time. Do not compare query vectors from one model with document vectors from another merely because their dimensions match. Keep model versions and vector fields or collections distinct, build and validate the replacement alongside the current path, and preserve a rollback. The exact migration depends on the storage system and write workload; it is part of the selection cost.
Factor into the decision:
- The likely lifetime of the choice, given your rate of change
- Whether the model version can be pinned, or whether a migration will eventually be forced on you
- Whether you would be able to self-host if the hosted option changed terms or disappeared
- Whether you have the operational maturity today to execute a dual-index migration, and if not, how much that raises the value of picking something stable
A model that scores slightly lower on your bake-off but is pinnable, self-hostable, and unlikely to force a migration may be the better choice. That is a legitimate engineering tradeoff, and it is one the leaderboard cannot express.
Reading the results and choosing
Lay the results out as a criteria table, not a ranking. Hypothetical illustrative numbers on an invented evaluation set, to show the shape:
| Baseline (lexical) | Model A | Model B | Model C | |
|---|---|---|---|---|
| NDCG@10, all queries | 0.41 | 0.52 | 0.55 | 0.53 |
| Recall@50, all queries | 0.62 | 0.78 | 0.80 | 0.81 |
| NDCG@10, identifier queries | 0.71 | 0.44 | 0.46 | 0.68 |
| NDCG@10, non-English subset | 0.33 | 0.49 | 0.31 | 0.50 |
| Dimensions | n/a | 384 | 1024 | 768 |
| Index size (GB) | 1.2 | 3.1 | 8.2 | 6.1 |
| Query p95 (ms) | 22 | 31 | 48 | 38 |
| Self-hostable | yes | yes | no | yes |
| Version pinnable | n/a | yes | no | yes |
Read across, not down. Model B has the highest aggregate NDCG in this invented table. If non-English quality, self-hosting, or version pinning is essential, it fails those stated requirements despite that aggregate. Model C is a candidate to investigate under those constraints; Model A may be adequate if its measured quality meets the team’s threshold. None of these hypothetical rows establishes a real model’s behavior. Nothing here answers the question by itself; the table exists so the tradeoff is visible.
How to interpret the quality columns honestly:
- Small aggregate differences require an uncertainty check. Before treating a gap as real, check whether it survives on a held-out portion of your evaluation set, and how many queries actually changed. If four queries out of two hundred moved, you have measured four queries.
- Segment regressions outweigh aggregate gains when the segment matters to your product. A model that improves the mean and destroys identifier lookups is a bad choice for a parts catalog and a fine one for a policy library.
- Inspect the queries that changed most, in both directions, by hand. This is where you learn what the model is actually doing, and it takes an hour.
- Weight the operational columns explicitly. Decide in advance how much quality you would trade for half the index size, so the answer is not invented after you see the numbers.
Any effect you measure is a property of your corpus, your chunking, and your evaluation set. Report that scope whenever you share the result; another corpus requires its own evaluation.
Documenting the decision
Write it down while it is fresh. In a year, someone will ask why this model, and the honest default answer is that nobody remembers.
Record, in one page in the repository:
- The candidates considered and why each was shortlisted or eliminated
- The evaluation set used, its version, and its size
- The held-constant configuration: chunking parameters, retriever, index settings, downstream stages
- The results table, with dates and exact model versions
- The decision and the tradeoff accepted in plain language
- The known weaknesses of the chosen model and what compensates for them
- The trigger that would justify revisiting: a new evaluation set, a corpus change, a segment regression appearing in monitoring
- The pinned version and where the pin lives
That last item is not paperwork. It is the input to your next migration, and to the benchmark interpretation you will revisit when someone forwards a new leaderboard.
In Plain English
Choosing an embedding model means picking which vector space your entire corpus will live in, and public leaderboards measure that choice against documents that are not yours. One transfer risk is domain mismatch: a benchmark’s documents and questions may differ from the vocabulary, structure, and tasks in your corpus.
So use the leaderboard to shortlist three to five candidates, filter them first on constraints a benchmark score does not establish (input length, dimensionality, self-hosting, licensing, version stability, language coverage), and then run each one against your own evaluation set with chunking, retriever, and judgments held identical. Read the results by segment, not just the mean, and count the migration cost of switching later as part of the choice.
This is for teams standing up dense retrieval or a RAG pipeline for the first time, and for teams considering an upgrade. If you do not yet have an evaluation set, that is the actual first task; the evaluation effort depends on corpus size, model access, and the operational checks required.
Next: write down your three shortlist constraints and eliminate on those before you look at a single quality number.
Next step
Write down the non-negotiable constraints, a baseline, and the questions that distinguish useful retrieval from a plausible near-match. Store the configuration and evaluation notes with the rest of your engineering work; GitHub’s directory profile is one starting point for exploring a collaboration tool, not a requirement to use that product.
Use the benchmark guide for shortlisting and the RAG evaluation checklist when retrieved passages feed an answer generator. The broader software trial guide applies the same decision-first discipline outside retrieval.
Sources checked
Official research and documentation checked 2026-10-17. These sources support the named mechanisms and research context. The selection procedures and trial worksheets are editorial guidance; they are not a report of experiments run by Ranker.dev.
- Sentence-BERT — Sentence embeddings and similarity-based comparison.
- Dense Passage Retrieval — Dual-encoder passage retrieval and evaluation in the paper’s stated setting.
- BEIR — Heterogeneous retrieval evaluation and transfer across tasks.
By Ranker.dev · Editorial standards · Suggest a correction
Ranker.dev is an independent software directory and publication. It is not affiliated with Ranker, Inc. or ranker.com.