Retrieval-Augmented Generation (RAG)

Intent

Before answering, the system retrieves the most relevant passages from your knowledge base and hands them to the model along with the question, asking it to answer only from them. RAG (retrieval-augmented generation) is for assistants that must answer about your own documents, which the model has never seen or which change often.

Problem

The HR team at Lucia’s company launched an assistant to answer employee questions. It answered confidently about vacation, expense reimbursement, and remote work. The problem: it answered from what the model “knew” in general, not from the company’s policy.

An employee asked how many remote days per week they could take. The bot said three, which is common in the market. Company policy said two. The policy had changed in March, and the model did not even know the old version.

The first attempt was pasting the whole handbook into the system prompt. It worked for a while, until the handbook reached 300 pages. Every question got slow and expensive, and the bot started mixing up the reimbursement rule with the travel rule.

This kind of error has consequences outside the engineering team. In Moffatt v. Air Canada (2024), the airline’s chatbot told a customer he could claim the bereavement fare after traveling. The policy published on the website said otherwise. British Columbia’s Civil Resolution Tribunal ruled that the company is responsible for what its bot says and ordered it to pay the difference.

The question this pattern answers: how do you get the model to answer with the right, current information without sending everything on every call?

Solution

Separate knowledge from the model. Documents live in a search index, split into chunks. On every question, the system retrieves the few most relevant chunks and builds the model’s input from them, tagged with identifiers.

The prompt asks for three things: answer only from the passages, cite where each claim came from, and say “not found” when the answer is not there. Retrieval decides what the model reads; the model decides how to answer.

Updating knowledge becomes updating the index, without touching the prompt. When the remote work policy changes again, someone reindexes the document and the next answer is already right.

The name comes from Lewis et al. (2020), who combined a retriever with a text generator. The version you will build today is simpler: an ordinary index, an off-the-shelf model, and a well-written prompt.

Structure

flowchart LR
  Q(["1. Question"]):::io --> S["2. Retrieve passages"]:::tool
  IDX[("Document index")]:::store --> S
  S --> C["3. Build context with IDs"]:::tool
  C --> M(["4. Answer with citations"]):::llm
  M --> V{"5. Cited valid passages?"}
  V -->|"yes"| OUT(["6. Answer with sources"]):::io
  V -->|"no"| N(["6. Not found"]):::io

1. The question arrives from the user.
2. Retrieval looks up the chunks closest to the question in the index, by embedding, keyword, or both.
3. Code builds the context: each passage with an ID and its source.
4. The model answers only from the passages and cites the IDs it used.
5. Code checks that the cited IDs exist in the context it sent.
6. The answer goes out with sources, or with an honest “not found”.

The index is built beforehand, in a separate process: load documents, split into chunks, generate embeddings, and store them. That process runs again whenever a document changes.

Worked example

Follow the employee’s question from start to finish. It arrives like this: “How many remote days per week can I take?”.

Retrieval returns the five closest chunks. Here are the top three, with their similarity scores:

remote-policy#2   0.82  "Starting March 2026, remote work is limited to two days per week..."
remote-policy#1   0.74  "This policy applies to full-time employees in every department..."
benefits#7        0.61  "The home office allowance is $120 per month, paid with payroll..."

Code builds the context with the IDs in brackets and sends it to the model. The answer comes back structured:

{ "text": "You can work remotely up to two days per week, a rule in effect since March 2026.", "sources": ["remote-policy#2"], "found": true }

Code checks that remote-policy#2 was among the IDs it sent. It was, so the answer goes out with a link to section 2 of the policy.

Now the token math, with illustrative numbers. Suppose 500 tokens per handbook page: 300 pages add up to 150,000 tokens on every call. With RAG, five chunks of 400 tokens add up to 2,000 tokens of context. At a thousand questions a day, that is 150 million versus 2 million input tokens.

Pseudocode

schema Answer { text: string, sources: list of string, found: bool }

function ask(question):
    # 2. retrieve only what matters
    chunks = index.search(question, k=5)
    trace.span("retrieve", chunks.ids, chunks.scores)

    # 3. context with identifiers
    context = join(f"[{c.id}] ({c.source}) {c.text}" for c in chunks)

    # 4. answer only from the passages
    answer = llm(RAG_PROMPT, { context, question }) -> Answer

    # 5. check citations in code
    if not answer.found or not answer.sources ⊆ chunks.ids:
        return "I could not find that in the documents."
    return answer

Real-world scenarios

• Internal knowledge base chat. Policies, processes, and technical docs in a single index. Citations become links to the original document, and employees check the rule at the source.
• Support bot with human escalation. The bot searches help center articles before answering. When no passage is relevant, the conversation goes to a person instead of turning into invention.
• Health assistant with guardrails. The base is restricted to content approved by professionals. “Not found” is part of safety: outside the base, the assistant advises seeing a doctor.
• Legal assistant. In Mata v. Avianca (2023), US lawyers filed a brief with court decisions invented by ChatGPT and were fined. A legal assistant with RAG only cites decisions that are in the index, and code checks every citation before the answer goes out.

Applicability

• The knowledge you need is yours: policies, manuals, contracts, tickets, code.
• Content changes and answers must follow the current version.
• The base is too large to fit, whole, in every call.
• You need to show where the answer came from.

When not to use

• The base fits comfortably in the context. Sending it all, with prompt caching, is simpler and avoids retrieval errors. See the RAG for Small Data smell and the Replace RAG with Direct Context refactoring.
• The question asks for an exact value from a system (balance, order status, a bill’s due date). That is a query, and the path is Tool Use.
• The question requires reasoning over the whole base (“how many contracts expire this year?”). Similarity search brings back some passages, not all of them. The path is a structured query, as in the Natural language to SQL scenario, or batch processing with Map-Reduce.

How to implement

1. Collect the real questions the assistant will get. They guide everything else.
2. Pick the sources and clean the documents: remove menus, footers, and duplicates.
3. Split by structure (section, article, clause), not by a fixed number of characters. Store the title, source, and effective date with each chunk.
4. Generate embeddings and store them in an index. Start simple: a database with a vector extension handles many cases.
5. Measure retrieval alone before wiring in the model: for each question, is the right passage among the top results?
6. Write the prompt with the three rules: passages only, cite IDs, admit when nothing was found.
7. Check citations in code and treat “not found” as a valid answer.
8. Build the index update process and record the index version on every trace.

Common beginner mistakes

• Cutting every 1,000 characters. The rule “up to two days per week” lands half in one chunk and half in the next. Neither answers the question alone.
• Chunks without a heading. The passage “the limit is two days” does not say which policy it belongs to. Prepend the section title, or see Contextual Chunking.
• Two versions of the same policy in the index. The 2024 and 2026 versions compete in retrieval. Tag the effective date and filter out superseded documents.
• Tuning the prompt when the error is in retrieval. If the right passage never arrived, no instruction saves the answer. Look at the retrieved IDs in the trace first.

Variations

• Hybrid Search. Combines embeddings and keywords. Helps with acronyms, codes, and proper names.
• Reranking. Retrieves many candidates and uses a model to re-sort them and keep the best.
• Query Rewriting. Rewrites the question before retrieval, useful in conversations where “and for interns?” depends on context.
• Agentic RAG. An agent decides when to search, what to search for, and whether to search again.
• Citation Grounding. Requires every claim to point at the exact passage that supports it, sentence by sentence.

Cost, latency, and failure modes

A note on k: more passages are not always better. The “Lost in the Middle” study (Liu et al., 2023) showed that models make worse use of information placed in the middle of a long context. Prefer a few good passages over many mediocre ones.

How to evaluate

Evaluate retrieval and answering separately. Most RAG problems are retrieval problems.

• Retrieval: for each question, did the right passage show up in the top k? (recall@k)
• Faithfulness: is every claim in the answer supported by a cited passage?
• Abstention: for questions with no answer in the base, does the system say “not found”?
• Usefulness: does the answer solve the asker’s question?

An eval case:

{ "question": "How many remote days per week can I take?", "expected_chunk": "remote-policy#2", "expected_answer_contains": "two days" }

Include questions with no answer in the base, such as “does the company pay for a gym?” when no policy mentions it. They measure whether the system knows how to say it does not know. To build the set, see Golden Dataset.

The split helps diagnosis. If recall@5 is 95% and the answer is wrong, the problem is the prompt or the model. If recall@5 is 60%, changing the prompt will not help.

In production

• A retrieve span with IDs, scores, and the index version.
• Daily “not found” rate. A sudden rise can mean a new topic or a broken index.
• A list of questions with low retrieval scores, to discover missing documents.
• User feedback (helpful or not) tied to the cited passages, to find bad documents.

Security

Every indexed document becomes text the model reads. If someone can put a document in the base, they can put instructions in front of the model. Greshake et al. (2023) demonstrated this indirect injection on real systems: the malicious instruction lived in retrieved content, not in the user’s message.

Three precautions:

• Control who writes to the index. A wiki any employee can edit is a way in.
• Enforce permissions at retrieval time. An intern must not receive passages from the board’s folder. Filter before retrieving, not after answering.
• Treat retrieved passages as data. Read the Instructions from Untrusted Data smell and the Prompt injection in practice fundamental.

In Brazil, the LGPD (the country’s data protection law, similar to GDPR) also applies: indexing customer tickets brings personal data into the index. Decide what goes in, how long it stays, and who can retrieve it.

Pros and cons

Code

The tabs do the same thing: retrieve passages, build the context with IDs, ask for a structured answer with sources, and check the citations. Python and TypeScript call Anthropic. LangChain swaps the model that answers. Retrieval stays yours: it decides what the model is allowed to cite.

import anthropic
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer, util

client = anthropic.Anthropic()
embedder = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")

# index built beforehand: each chunk with id, source, and text
CHUNKS = load_chunks("policies/")  # your loading and chunking function
VECTORS = embedder.encode([c["text"] for c in CHUNKS])

RAG_PROMPT = """Answer the question using only the passages provided.
List the IDs of the passages you used in sources.
If the answer is not in the passages, set found=false and do not make anything up."""


class Answer(BaseModel):
    text: str
    sources: list[str]
    found: bool


def ask(question: str, k: int = 5) -> Answer | None:
    # 2. retrieve the k closest chunks
    scores = util.cos_sim(embedder.encode(question), VECTORS)[0]
    top = [CHUNKS[i] for i in scores.argsort(descending=True)[:k].tolist()]

    # 3. context with identifiers
    context = "\n\n".join(f"[{c['id']}] ({c['source']})\n{c['text']}" for c in top)

    # 4. answer only from the passages
    answer = client.messages.parse(
        model="claude-opus-5-5",
        max_tokens=2048,
        system=RAG_PROMPT,
        messages=[{"role": "user", "content": f"<passages>\n{context}\n</passages>\n\nQuestion: {question}"}],
        output_format=Answer,
    ).parsed_output

    # 5. check citations in code
    ids = {c["id"] for c in top}
    if not answer.found or not answer.sources or not set(answer.sources) <= ids:
        return None  # 6. "not found"
    return answer

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";

const client = new Anthropic();

const RAG_PROMPT = `Answer the question using only the passages provided.
List the IDs of the passages you used in sources.
If the answer is not in the passages, set found=false and do not make anything up.`;

const Answer = z.object({ text: z.string(), sources: z.array(z.string()), found: z.boolean() });

// search queries your vector database: returns { id, source, text }[]
export async function ask(question: string, k = 5) {
  // 2. retrieve the k closest chunks
  const top = await search(question, k);

  // 3. context with identifiers
  const context = top.map((c) => `[${c.id}] (${c.source})\n${c.text}`).join("\n\n");

  // 4. answer only from the passages
  const { parsed_output: answer } = await client.messages.parse({
    model: "claude-opus-5-5",
    max_tokens: 2048,
    system: RAG_PROMPT,
    messages: [{ role: "user", content: `<passages>\n${context}\n</passages>\n\nQuestion: ${question}` }],
    output_config: { format: zodOutputFormat(Answer) },
  });

  // 5. check citations in code
  const ids = new Set(top.map((c) => c.id));
  if (!answer?.found || answer.sources.length === 0 || !answer.sources.every((s) => ids.has(s))) {
    return null; // 6. "not found"
  }
  return answer;
}

from langchain.chat_models import init_chat_model
from pydantic import BaseModel

class Answer(BaseModel):
    text: str
    sources: list[str]
    found: bool

RAG_PROMPT = """Answer the question using only the passages provided.
Cite the ids of the passages you used in sources.
If the answer is not in the passages, set found to false and do not invent one."""

# "provider:model". Retrieval is not part of that string.
answerer = init_chat_model("openai:gpt-4o-mini", temperature=0).with_structured_output(Answer)


def ask(question: str, k: int = 5) -> Answer | None:
    top = search(question, k)  # 2. embeddings, keywords, or both

    # 3. context with identifiers
    context = "\n\n".join(f"[{c['id']}] ({c['source']})\n{c['text']}" for c in top)

    # 4. answer only from the passages
    answer = answerer.invoke([
        ("system", RAG_PROMPT),
        ("human", f"<passages>\n{context}\n</passages>\n\nQuestion: {question}"),
    ])

    # 5. check citations in code
    ids = {c["id"] for c in top}
    if not answer.found or not answer.sources or any(s not in ids for s in answer.sources):
        return None  # 6. "not found"
    return answer

Framework support

Checked against the LangChain 1.x docs in October 2026.

Check your understanding