Contextual Chunking

Intent

Split documents along their structure (sections, clauses, tables) and attach to each chunk the context it came from: document, section, date, and a sentence that situates the passage. It is for RAG systems where chunks lose their meaning on their own.

Problem

Eduardo maintains search over a health insurer’s internal manual (a Brazilian private plan, overseen by ANS, the national regulator). The documents were cut every 500 tokens, ignoring structure. One of the chunks says only:

...the deadline is 21 business days from the request, and it may be
extended once with justification. In the cases covered by item 3.2...

Deadline for what? For whom? From which version of the manual? Search finds this passage for “what is the authorization deadline for elective surgery?”, but also for “what is the reimbursement deadline?”, because the passage does not say which topic it covers. The model answers confidently and sometimes gets the topic wrong.

The blind cut also separated a table from its title. The rows with deadline numbers landed in one chunk; the header with the columns (authorization, reimbursement, appeal) stayed in the previous one. The bot started citing “21” without saying which deadline.

Then there is item 3.2. It lives in another chunk. Anyone who reads only the passage above cannot tell what the item allows. In this insurer, that mixes ANS rules with the plan’s own rules.

The question this pattern answers: how do you split documents so that each chunk is understandable and findable on its own?

Solution

Two changes at indexing time:

• Split by structure: use headings, sections, clauses, list items, and tables as boundaries. A chunk never starts mid-sentence or separates a table from its header. A maximum size exists, but it is the limit, not the rule.
• Attach context: each chunk gets a header with its path (document, chapter, section), its date or version, and, optionally, a model-generated sentence that situates it (“This passage covers the authorization deadline for elective surgery on individual plans”).

The header goes into the indexed text (for search) and into the text sent to the model (for the answer and the citation).

Anthropic described this idea in Contextual Retrieval (September 2024). Before indexing, a model reads the document and the chunk and writes a sentence that situates the passage. That sentence goes into the text used for the embedding and for keyword search.

Worked example

Take the deadline passage and the document it came from. After the blind cut, the index stores only the middle. After this pattern, the same middle gets a header:

[Member Handbook | Authorizations | 4.1 Elective surgery | v7, effective 2026-03]
This passage covers the authorization deadline for elective surgery on individual plans.
The deadline is 21 business days from the request, and it may be
extended once with justification. In the cases covered by item 3.2
(urgency and emergency, see section 4.3), the deadline does not apply.

Three things changed, and each one teaches a lesson.

1. The path. “Authorizations > 4.1 Elective surgery” enters the embedding and BM25. A reimbursement question no longer competes with the same similarity.
2. The context sentence. It translates the vague passage into the question’s vocabulary (“authorization”, “elective surgery”, “individual plan”).
3. The neighboring clause. If section 4.1 fits the limit, item 3.2 travels with it. If it does not, the header still points at 4.3, and the citation does not pretend the passage is complete.

A token account, with illustrative numbers. Suppose 800 tokens of body and 80 of header. Across 2,000 chunks, the index grows by about 160,000 tokens. At query time, five chunks go from 4,000 to 4,400 tokens. The extra cost sits in ingestion (one small call per chunk, with the document cached), not in the question.

The paper “Lost in the Middle” (Liu et al., 2023) shows that models use the start and end of a context better than the middle. Putting the path at the start of the chunk helps twice: search finds the right topic, and the model reads first where the passage came from.

Structure

flowchart LR
  D(["1. Document"]):::io --> P["2. Parse the structure"]:::tool
  P --> S["3. Split by section and clause"]:::tool
  S --> C(["4. Generate a context sentence"]):::llm
  C --> H["5. Build header and passage"]:::tool
  H --> I[("6. Index")]:::store

1. The document arrives as PDF, HTML, or text.
2. A parser identifies headings, sections, lists, and tables.
3. The split follows that structure, within a maximum size.
4. A small model reads the document and the passage and writes a sentence that situates it.
5. Each chunk becomes a header (path, version, context sentence) plus the original text.
6. The full chunk is indexed for lexical and vector search.

Pseudocode

schema ChunkContext { summary: string /* one sentence */ }

function index_document(doc):
    tree = parse_structure(doc)                         # 2. headings, sections, tables
    for section in tree.leaves(max_tokens=800):         # 3. by structure
        path = " > ".join(section.ancestors_titles())
        ctx = llm(CONTEXT_PROMPT, doc.summary, section.text, model="small") -> ChunkContext   # 4
        header = f"[{doc.title} | {path} | version {doc.version}] {ctx.summary}"
        index.add(id=section.id, text=header + "\n" + section.text,
                  metadata={doc: doc.id, path: path, version: doc.version})               # 5 and 6

Real-world scenarios

• Internal knowledge base chat. Policies split by section, with the path and version on every passage. A search for “reimbursement deadline” no longer brings back the authorization deadline.
• Legal assistant. Contracts split by clause and item, with the clause number in the header, for exact citation. In Mata v. Avianca (2023), lawyers filed a brief with decisions invented by ChatGPT. An index that stores the clause number cannot invent a clause, but it only helps if search returns the right passage.
• Tax and health-plan manuals. Brazilian rules cite each other (“under art. 12”). Without the path on the chunk, the model treats the cross-reference as if it were the rule.

Applicability

• Long, structured documents (manuals, policies, contracts, regulations).
• Passages that, on their own, use vague terms (“the deadline”, “in this case”).
• Several versions or several similar documents in the same index.
• Answers that need to cite the exact section.

When not to use

• Short documents that fit in a single chunk. A title and version at the start are enough; splitting only invents artificial boundaries.
• Unstructured content (conversation transcripts). Split by topic, or by overlapping windows.
• The material fits whole in the context. See RAG for Small Data.

How to implement

1. Look at real documents and identify their structural markers (numbered headings, clauses, tables).
2. Write a parser per document type; convert PDFs to a structured format first.
3. Define the maximum size and what to do with larger sections (split by paragraph, keeping the heading).
4. Build the header with path and version.
5. Generate the context sentence with a small model, caching the whole document in the prompt.
6. Reindex and compare recall and precision in the search eval.
7. Use the header’s path in the answer’s citations.

Common traps

• Cutting by characters and calling it structural. If the boundary ignores the table title, you only changed the size.
• Generating the sentence from the chunk alone. Without the document, the model writes “this passage discusses a deadline”, as vague as the body. The call needs to see the path and a document summary.
• Leaving repealed versions in the index. A header that says “v6” does not stop search from returning v6. Filter by effective date at query time and remove the old document on reindex.
• Trusting the sentence and relaxing the parser. A wrong sentence poisons the embedding. A sample of chunks at ingestion catches that before a user does.

Variations

• Structural header only. Path and version, no generated sentence. Cheap, and it already solves much of the problem.
• Generated context sentence. Improves search for vague passages; costs one small call per chunk at indexing time.
• Hierarchical chunks. Small chunks for search, and the whole section (the parent) sent to the model when a small one is found.
• Tables as their own chunks. Each table becomes a chunk, with title and caption, as text.

Start with the structural header. Turn on the generated sentence only if the topic-confusion eval still fails.

Cost, latency, and failure modes

How to evaluate

• Search recall and precision before and after, on the same eval.
• Topic confusion: similar questions about different topics (“reimbursement deadline” and “authorization deadline”).
• Citations: the section cited in the answer is the right one.
• Table and clause boundaries: cases where the blind cut used to split a block.

{ "question": "What is the authorization deadline for elective surgery?", "relevant": ["manual-v7#authorizations-4.1"], "not_relevant": ["manual-v7#reimbursement-2.3"] }

Build twin questions on purpose. If the relevant reimbursement passage is not in the top 5 for the authorization question, the header is doing its job.

In production

• Record the path and version of the passages used in each answer.
• A reindexing routine when a document changes, removing old versions.
• A sample of chunks to check the parser and context sentences.
• An alert on the empty-path rate at ingestion: a broken parser after a new kind of PDF.

Security

An indexed chunk can carry personal data (a member’s name, a card number). Under Brazil’s LGPD, the index is one more place that data lives. Do not send documents with personal data to the context-sentence model if company policy forbids it. In those cases, stay on the structural header, built only from title and section.

Pros and cons

Relations

• Destination pattern: RAG. Contextual chunking is the ingestion step that makes search return the right passage.
• Neighbors: Hybrid Search combines keyword and embedding, which makes good use of the path in the header. Embeddings explains why a vague passage becomes a vague vector.
• When it does not apply: RAG for Small Data, if the material fits whole.

Code

Both tabs split a document by headings and build the header with the path; the Python tab also generates the context sentence with the whole document cached.

import re

import anthropic

client = anthropic.Anthropic()


def split_by_headings(markdown: str, max_chars: int = 3000) -> list[dict]:  # 2 and 3
    chunks, path = [], []
    for block in re.split(r"\n(?=#{1,4} )", markdown):
        m = re.match(r"(#{1,4}) (.+)", block)
        if m:
            level = len(m.group(1))
            path = path[: level - 1] + [m.group(2).strip()]
        for i in range(0, len(block), max_chars):  # large section: split, keeping the path
            chunks.append({"path": " > ".join(path), "text": block[i : i + max_chars]})
    return chunks


def situate(doc_text: str, chunk_text: str) -> str:  # 4
    reply = client.messages.create(
        model="claude-haiku-4-5", max_tokens=100,
        system=[{"type": "text", "text": f"<document>\n{doc_text}\n</document>",
                 "cache_control": {"type": "ephemeral"}}],  # document cached across chunks
        messages=[{"role": "user", "content": "In one sentence, say which topic and case this passage covers, "
                                              f"to help find it in a search.\n<passage>{chunk_text}</passage>"}],
    )
    return reply.content[0].text.strip()


def index_document(doc: dict) -> list[dict]:
    out = []
    for c in split_by_headings(doc["markdown"]):
        header = f"[{doc['title']} | {c['path']} | version {doc['version']}] {situate(doc['markdown'], c['text'])}"
        out.append({"id": f"{doc['id']}#{c['path']}", "text": f"{header}\n{c['text']}", "path": c["path"]})  # 5
    return out

type Chunk = { id: string; path: string; text: string };

export function chunkByHeadings(doc: { id: string; title: string; version: string; markdown: string }, maxChars = 3000): Chunk[] {
  const chunks: Chunk[] = [];
  let path: string[] = [];
  for (const block of doc.markdown.split(/\n(?=#{1,4} )/)) {
    const m = block.match(/^(#{1,4}) (.+)/);
    if (m) path = [...path.slice(0, m[1].length - 1), m[2].trim()];
    const p = path.join(" > ");
    const header = `[${doc.title} | ${p} | version ${doc.version}]`; // 5. structural header
    for (let i = 0; i < block.length; i += maxChars) {
      chunks.push({ id: `${doc.id}#${p}#${i}`, path: p, text: `${header}\n${block.slice(i, i + maxChars)}` });
    }
  }
  return chunks;
}

Check your understanding