Query Rewriting

Intent

Turn the user’s question into one or more better searches before querying the index. It is for search and RAG systems where questions arrive vague, dependent on the conversation, full of acronyms, or mixing several topics.

Problem

Rodrigo runs an energy company’s internal chat. Questions rarely arrive search-ready. “And for contractors?” only makes sense after the previous question, about expense reimbursement. “What’s the MV rule when WFH?” uses two acronyms no document uses. “How does vacation work and what happens if I resign in the middle” has two topics that live in different documents.

Sending those sentences straight to the index brings bad results. The first has no subject; the second lacks the words documents use; the third finds one topic and ignores the other.

Hybrid search helps when the query is already well formed. It does not invent the subject the message hid, and it does not split two questions glued together.

The question this pattern answers: how do you convert the way people ask into the way documents are written?

Solution

Before searching, a short step rewrites the question using the available context:

• Resolve references: fold in what the conversation already said (“and for contractors?” becomes “expense reimbursement for independent contractors”).
• Expand terms: replace internal acronyms and slang with the documents’ terms, with help from a glossary.
• Decompose: split questions with several topics into independent searches.
• Generate variations: two or three different phrasings, to raise recall.

The output is structured (a list of searches, each with optional filters) and goes to search. The original question is still what the main model answers.

Structure

flowchart LR
  Q(["1. Question and history"]):::io --> RW(["2. Rewrite into searches"]):::llm
  G[("Internal glossary")]:::store --> RW
  RW --> S1["3. Search 1"]:::tool
  RW --> S2["3. Search 2"]:::tool
  S1 --> U["4. Merge and dedupe"]:::tool
  S2 --> U
  U --> M(["5. Answer the original question"]):::llm

1. The question arrives with the conversation’s latest messages.
2. A small model rewrites it into one or more searches, using the glossary.
3. The searches run in parallel.
4. Results are merged, without duplicates.
5. The main model answers the original question with the passages found.

Worked example

Recent history: “How does travel expense reimbursement work?”. New question: “and for contractors?”.

Without a rewrite, hybrid search looks up contractor and returns pages about hiring independent contractors, nothing about reimbursement. With the rewrite step and the glossary {contractor: independent contractor (PJ)} (PJ is the Brazilian label for that arrangement), the structured output is:

{
  "queries": [
    { "text": "travel expense reimbursement for independent contractors", "area": "finance" }
  ],
  "needs_clarification": false
}

Search now finds the right policy. The main model still sees the original question (“and for contractors?”) plus the passages, so it answers in the conversation’s tone, not with a loose paragraph from the policy.

Second case, two topics: “How does vacation work and what happens if I resign in the middle”. The rewrite returns two searches, statutory vacation and resignation during vacation. Without decomposing, hybrid search tends to keep only the vacation document, the one most similar to the first half.

Pseudocode

schema Rewrite { queries: list[{ text: string, filter: { area: string? } }], needs_clarification: bool }

function retrieve(question, history):
    glossary = glossary_for(question)                                  # acronyms found
    rw = llm(REWRITE_PROMPT, question, history[-6:], glossary, model="small") -> Rewrite   # 2
    if rw.needs_clarification:
        return ask_user("Do you mean X or Y?")
    parallel {                                                         # 3
        results = [hybrid_search(q.text, filters=q.filter, k=20) for q in rw.queries]
    }
    trace.span("rewrite", original=question, queries=rw.queries)
    return dedupe(flatten(results))                                    # 4

Real-world scenarios

• Internal knowledge base chat. Internal acronyms and nicknames become the documents’ official terms; follow-up questions inherit the conversation’s topic.
• E-commerce semantic search. “Running shoes for the rain under 80” becomes a search by category, attribute (waterproof), and price filter.
• Support bot with human escalation. Long, confusing messages become one or two focused searches in the help center.

Applicability

• Conversational questions that depend on what came before.
• Users’ vocabulary differs from the documents’ vocabulary.
• Questions with more than one topic.
• Implicit filters in the sentence (price, date, department) that search can apply.

When not to use

• Questions already arrive well formed, as in a catalog search by term. Rewriting only adds latency.
• Very tight latency on every query. Use a glossary in code to expand acronyms, with no model.
• The problem is ordering, not the query. See Reranking.

In Agentic RAG, rewriting stops being a fixed step: the agent decides whether to search again. This page’s pattern is the cheap, fixed step before a single search.

How to implement

1. Gather real questions that fail in search and classify why: reference, vocabulary, several topics, implicit filter.
2. Build a glossary of internal acronyms and nicknames.
3. Write the rewrite prompt with structured output, asking for few searches.
4. Pass the conversation’s latest messages, not the whole history.
5. Run the searches in parallel and merge the results.
6. Measure search recall with and without rewriting, by failure type.
7. Record the original question and the rewrites on the trace.

The glossary starts small: MV, WFH, contractor, N2, the HR system’s nickname. Every question with no result after rewriting is a candidate for a new entry, reviewed by someone who knows the domain.

Common mistakes

• Five variations for a simple question. The prompt needs a cap (3 is a start) and needs_clarification for real ambiguity.
• Too much history. Twenty old turns contaminate the new question. Six recent ones usually suffice; a “thanks” turn should not reopen the previous topic.
• Answering with the rewritten question. The user asked “and for contractors?”. The answer talks about “independent contractors” in the tone of a manual. Rewrite to search; answer the original utterance.
• An invented filter. “under 80” became max_price_cents: 80 instead of 8000. Check units in the eval.

Variations

• Glossary expansion in code. No model: known acronyms are swapped for official terms. Cheap and predictable.
• Hypothetical document. The model writes an imaginary answer and search uses its text, which looks more like the documents than the question does. That is the idea behind HyDE (Gao et al., 2022).
• Clarifying question. When the question is too ambiguous, the system asks the user instead of guessing.
• Rewriting inside the agent. In Agentic RAG, the agent itself decides and redoes the searches.

Cost, latency, and failure modes

How to evaluate

• Search recall with and without rewriting, per question type.
• Faithfulness: a person or a judge checks that the rewrite keeps the intent.
• Final answer quality, in the full eval.

{ "history": ["How does travel expense reimbursement work?"], "question": "and for contractors?", "expected_queries_contain": ["reimbursement", "independent contractor"] }

Include cases where the rewrite should refuse: “that” with no history, or two equally possible readings. needs_clarification=true is a success on those cases, not a failure.

In production

• A rewrite span with the original question, the queries, and the filters.
• A dashboard of questions with no results after rewriting, to feed the glossary.
• A sample of rewrites for faithfulness review.

If the generated query contains no term from the original and none from the glossary, treat it as an anomaly. It is the cheapest signal of meaning distortion.

Security

Conversation history can carry personal data (name, employee id, reimbursement amount). The rewrite reads that history. Mask what search does not need before sending it to the small model, and do not store raw history on the span if the trace is too widely readable.

Rewrite output goes to the index, not to the user, but an invented filter (area: hr when the question was finance) changes who can see the passage if you reuse that field for permission. Do not reuse an extracted filter as an access control.

Pros and cons

Code

Both tabs rewrite the question using recent history and a glossary, and run the generated searches in parallel.

import asyncio

import anthropic
from pydantic import BaseModel

client = anthropic.AsyncAnthropic()

GLOSSARY = {"MV": "meal voucher", "WFH": "remote work", "contractor": "independent contractor (PJ)"}


class Query(BaseModel):
    text: str
    area: str | None = None


class Rewrite(BaseModel):
    queries: list[Query]
    needs_clarification: bool


async def rewrite(question: str, history: list[str]) -> Rewrite:
    terms = {k: v for k, v in GLOSSARY.items() if k.lower() in question.lower()}
    reply = await client.messages.parse(  # 2
        model="claude-haiku-4-5", max_tokens=500,
        system="Rewrite the question into at most 3 self-contained searches for the HR knowledge base. "
               "Resolve references to the conversation, use the glossary terms, and split different topics.",
        messages=[{"role": "user", "content": f"Recent conversation: {history[-6:]}\nGlossary: {terms}\nQuestion: {question}"}],
        output_format=Rewrite,
    )
    return reply.parsed_output


async def retrieve(question: str, history: list[str]) -> list[dict]:
    rw = await rewrite(question, history)
    if rw.needs_clarification:
        return []  # the caller asks the user
    results = await asyncio.gather(*(search(q.text, area=q.area, k=20) for q in rw.queries))  # 3
    seen, merged = set(), []
    for doc in (d for r in results for d in r):  # 4
        if doc["id"] not in seen:
            seen.add(doc["id"])
            merged.append(doc)
    return merged

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";

const client = new Anthropic();

const ProductQuery = z.object({
  text: z.string(),
  category: z.enum(["shoes", "apparel", "accessory", "other"]).nullable(),
  max_price_cents: z.number().int().nullable(),
  attributes: z.array(z.string()),
});

export async function rewriteSearch(userText: string) {
  const { parsed_output } = await client.messages.parse({
    model: "claude-haiku-4-5",
    max_tokens: 300,
    system:
      "Convert the customer's search into a catalog query. Extract category, maximum price in cents, and attributes " +
      "(e.g., waterproof, cushioning). The text field must use catalog terms, no slang.",
    messages: [{ role: "user", content: userText }],
    output_config: { format: zodOutputFormat(ProductQuery) },
  });
  if (!parsed_output) return { text: userText, category: null, max_price_cents: null, attributes: [] };
  return parsed_output; // "running shoes for the rain under 80" -> { text: "running shoes", category: "shoes", max_price_cents: 8000, attributes: ["waterproof"] }
}

Check your understanding