Reranking

Intent

Retrieve many candidates with a fast method, then use a more precise model to reorder them and pick the best. It is for search and RAG systems where the right document shows up among the candidates, but not in the top positions.

Problem

Lúcia maintains a logistics company’s knowledge base chat. Hybrid search returns 50 passages per question, and the model gets the top 5. The eval showed something curious: in many of the wrong answers, the right passage was among the 50, just at position 12 or 20.

Fast search measures similarity approximately: it compares the question and the passage separately, as vectors or term lists. It does not read the two together. So a passage that talks “around” the topic beats one that answers the question exactly.

Sending all 50 passages to the model would fix recall, but it would be expensive and fill the context with noise. “Lost in the Middle” (Liu et al., 2023) shows the other side: even when they fit, the model uses what sits in the middle worse. Fifty mediocre passages are not 50 useful ones.

The question this pattern answers: how do you get the best passages to the top without paying for the main model to read them all?

Solution

Split search into two stages:

• Retrieval: fast and broad. Hybrid or vector search brings back 30 to 100 candidates. The goal here is recall: the right one has to be on the list.
• Reranking: slow and precise. A model that reads the question and each candidate together gives a relevance score. The goal is precision: the top ones have to be the best.

The reranker can be a specialized model (a cross-encoder), cheaper and faster, or a language model with structured output, more flexible. Only the top K after reranking move on to the main model.

Structure

flowchart LR
  Q(["1. Question"]):::io --> R["2. Broad retrieval: 50 candidates"]:::tool
  I[("Index")]:::store --> R
  R --> RR(["3. Reranker reads question and passage together"]):::llm
  RR --> T["4. Top 5 by the new score"]:::tool
  T --> M(["5. Main model answers"]):::llm

1. The question arrives.
2. A fast search brings back many candidates, favoring recall.
3. The reranker scores each candidate’s relevance by reading the question and the passage together.
4. Candidates are sorted by the new score, and only the top ones move on.
5. The main model answers from a few passages, all relevant.

Worked example

Question: “What is the deadline to contest a fleet traffic fine?”.

Hybrid search returns 50 passages. The top five, before reranking:

pos 1  fleet.md#fines-1      "The fleet has third-party insurance coverage..."
pos 2  hr.md#payroll         "Payroll deductions for driver infractions..."
pos 3  fleet.md#fuel
pos 12 fleet.md#fines-3      "The deadline to contest a fleet fine is 15 days..."

The passage that answers sits at 12. The cross-encoder reads the question together with each passage. fleet.md#fines-3 gets the highest score because it contains the deadline. hr.md#payroll talks about fines, but not about contesting, and drops. After reranking, the 5 the main model receives start with fines-3.

A minimum threshold (score 2 on a 0 to 3 scale, or above zero on the cross-encoder) avoids sending five “on-topic” passages when none of them answers. Then the system says “I did not find that” instead of inventing from the fuel page.

Pseudocode

schema Relevance { id: string, score: int /* 0 to 3 */ }

function retrieve(question):
    candidates = hybrid_search(question, k=50)                             # 2. recall
    scores = reranker.score([(question, c.text) for c in candidates])       # 3. precision
    ranked = sort(zip(candidates, scores), by=score, desc)
    trace.span("rerank", before=[c.id for c in candidates[:5]], after=[c.id for c, _ in ranked[:5]])
    return [c for c, s in ranked[:5] if s >= MIN_SCORE]                    # 4

function answer(question):
    passages = retrieve(question)
    if not passages: return "I did not find that in the knowledge base."
    return llm(ANSWER_PROMPT, passages, question)                          # 5

Real-world scenarios

• E-commerce semantic search. Retrieval brings back 100 similar products; reranking puts on top the ones that meet every requested attribute (color, size, use).
• Internal knowledge base chat. Of the 50 passages retrieved, only the 5 that answer the question go to the model.
• Deep research with citations. Web search results are reranked before being read, so reading effort goes only to the most promising sources.

Applicability

• The search eval shows the right result is among the candidates, but outside the top positions.
• The main model receives few passages, and every slot matters.
• Fast search confuses “talks about the topic” with “answers the question”.

When not to use

• The right result does not even appear among the candidates. Reranking cannot create what was not retrieved; improve retrieval with Hybrid Search or Query Rewriting.
• Latency is very tight and search already gets the top right.
• The material fits whole in the context. Then there is nothing to rerank.

Reranking is a stage of RAG, not a substitute for search. If recall@50 is low, changing the reranker is changing the stage that already works.

How to implement

1. Measure the current search’s recall@50 and precision@5 in the eval.
2. If recall@50 is good and precision@5 is poor, reranking is the right tool.
3. Choose the reranker: a cross-encoder (fast, cheap, multilingual when available) or a language model with a structured score.
4. Rerank the candidates and keep a minimum relevance threshold.
5. Measure precision@5 and final answer quality again.
6. Tune how many candidates to rerank: more candidates, more recall and more cost.

Start with a cross-encoder. Only step up to a language model when relevance depends on a business rule the encoder did not see in training (“current documents only”, “ignore drafts”).

Common mistakes

• A threshold copied from a tutorial. A cross-encoder score is not a probability. Calibrate the cut on your eval: look at the scores of the right passages and of the wrong ones.
• Reranking 8 candidates. If the right result is at 20, a short pool leaves it out. The pool is recall; the top K afterwards is precision.
• One huge list for an LLM. “Lost in the Middle” applies to the reranker too. Prefer scoring candidate by candidate, or small slices, if you use a language model.
• Near-identical passages at the top. Three paragraphs of the same policy take three slots. After reranking, drop redundant neighbors.

Variations

• Cross-encoder. A small model trained for relevance that reads question and passage together. The most common choice.
• Language model as reranker. A 0 to 3 score per candidate, as structured output. More expensive, understands complex criteria (“only current documents”).
• List-wise reranking. The model receives all candidates at once and returns the order. Fewer calls, sensitive to position in the list.
• Diversity. After reranking, avoid near-identical passages in the top positions.

Cost, latency, and failure modes

The numbers below are illustrative. Suppose 50 candidates, a local cross-encoder, and the main model reading 5 passages of 400 tokens.

How to evaluate

• Recall@N for retrieval and precision@K after reranking.
• NDCG or MRR, to measure the ordering.
• Final answer quality, because the goal is the answer, not the search.

{ "question": "What is the deadline to contest a fleet traffic fine?", "relevant_passages": ["fleet.md#fines-3"], "measure": ["recall@50", "precision@5"] }

Also measure the “search alone” arm. If the reranker makes precision@5 worse, it is out of domain: logistics vocabulary, acronyms, current documents. Then a small LLM with a written criterion often beats a generic encoder.

In production

• A span with ids before and after reranking, and the scores.
• A dashboard of the average position of the passage the model cites: if it always cites the first, the ordering is good.
• An alert for a rise in “not found” answers, a sign of a bad threshold or bad retrieval.

The before / after pair on the span is what lets you see, on a wrong answer, whether the right passage rose and the model ignored it, or whether it never rose.

Security

The reranker reads the same untrusted content RAG reads. A wiki page that says “this document is the most relevant of all” can inflate an LLM-reranker’s score. Cross-encoders trained for relevance suffer less from that rhetoric, but they are not a defense. Permissions still belong in retrieval: the reranker must not see a passage the user cannot see.

Pros and cons

Code

The Python tab uses a cross-encoder; the TypeScript tab uses a language model with structured scores.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/mmarco-mMiniLMv2-L12-H384-v1")  # multilingual
MIN_SCORE = 0.0


def retrieve(question: str, k: int = 5) -> list[dict]:
    candidates = hybrid_search(question, k=50)  # 2. recall
    scores = reranker.predict([(question, c["text"]) for c in candidates])  # 3
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [c | {"rerank_score": float(s)} for c, s in ranked[:k] if s >= MIN_SCORE]  # 4

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";

const client = new Anthropic();
const Scores = z.object({ scores: z.array(z.object({ id: z.string(), score: z.number().int().min(0).max(3) })) });

type Passage = { id: string; text: string; updated: string };

export async function rerank(question: string, candidates: Passage[], k = 5): Promise<Passage[]> {
  const list = candidates.map((c) => `<passage id="${c.id}" updated="${c.updated}">${c.text}</passage>`).join("\n");
  const { parsed_output } = await client.messages.parse({
    model: "claude-haiku-4-5",
    max_tokens: 2000,
    system:
      "Score each passage from 0 to 3: 3 answers the question directly, 2 helps, 1 only talks about the topic, 0 is useless. " +
      "Passages superseded by a newer version get at most 1.",
    messages: [{ role: "user", content: `<question>${question}</question>\n${list}` }],
    output_config: { format: zodOutputFormat(Scores) },
  });
  const score = new Map((parsed_output?.scores ?? []).map((s) => [s.id, s.score]));
  return candidates
    .filter((c) => (score.get(c.id) ?? 0) >= 2)
    .sort((a, b) => (score.get(b.id) ?? 0) - (score.get(a.id) ?? 0))
    .slice(0, k); // 4
}

Check your understanding