Bi-encoders vs cross-encoders: the retrieval trade-off
The difference between the model that finds candidates and the model that reranks them comes down to when the query and document meet. One scales to millions; the other is precise. Production retrieval uses both.
Two kinds of model score relevance in a retrieval system, and knowing the difference is what lets you reason about why retrieval works the way it does. The distinction is simple once you see it: it's about when the query and the document are combined.
Bi-encoder: encode separately, compare vectors
A bi-encoder encodes the query and each document independently into vectors, then measures similarity — usually cosine — between them. The query and document never meet inside the model; they're each turned into a point in space and compared afterwards. The huge advantage: you can embed every document ahead of time and store the vectors, so at query time you only encode the query and do a fast nearest-neighbour search over millions of precomputed vectors. This is what powers the retrieval (first) stage. The cost is precision — because the two texts are encoded in isolation, the model can't reason about how they relate.
Cross-encoder: encode jointly, score the pair
A cross-encoder takes the query and the document together, as a single joint input, and runs them through the model with full cross-attention between every query token and every document token — then outputs one relevance score. Because the two texts interact inside the model, it judges relevance far more accurately. But there's no shortcut: you can't precompute anything, because the score depends on the specific pair, so you must run the model once per (query, document) pair at query time. That's far too expensive to run over a whole corpus.
Retrieve, then rerank
So you use both, in stages. A bi-encoder retrieves a wide shortlist cheaply from the full corpus (fast, approximate). Then a cross-encoder reranks just that shortlist precisely (slow, accurate, but only over a bounded set). You get the bi-encoder's scale and the cross-encoder's precision — which is exactly the two-stage retrieval from the advanced-RAG post, now with the reason each stage uses a different kind of model made explicit.
A bi-encoder asks 'where do these two land in space?' and can precompute the answer for a million documents. A cross-encoder asks 'how do these two actually relate?' and has to look every time. Retrieve with one, rerank with the other.