Context Compaction

Intent

Summarize or prune the context as it grows, keeping what matters for the next steps. It is for long agents and conversations that would otherwise fill the window and get worse with every turn.

Problem

Renata’s coding agent handles two- to three-hour tasks: migrating a module, upgrading dependencies, fixing a batch of tests. At the start of a task it is precise. After 150 turns, the context holds hundreds of tool results: whole files, test output thousands of lines long, searches that led nowhere.

Near the end, the agent starts editing files it already fixed, reapplying changes that were reverted, and forgetting the constraint agreed at the start (“do not touch the public API”). On some tasks the window overflows and the run fails.

That is the Context Rot smell: the history is still there, but it is no longer useful memory. The paper “Lost in the Middle” (Liu et al., 2023) helps explain the symptom. With a full window, the model uses the middle poorly, which is exactly where the turn-3 constraint sits.

Starting a new session does not help: the agent loses the plan, the decisions, and what has been done. Renata needs a way for the context to stay useful after hours.

The question this pattern answers: how do you keep an agent clear-headed on a task longer than its context window can hold?

Solution

Treat the context as working memory with ongoing maintenance. Three operations, in order of cost:

• Prune: delete old tool results that have already been used (the contents of a file read 80 turns ago). The call stays on record; the content goes.
• Summarize: when the context passes a limit, replace the old part with a summary that keeps the goal, decisions, current state, and next steps.
• Externalize: facts that must survive any summary (plan, constraints, list of changed files) go to a file or state outside the context, and the agent reads them when needed.

The trigger is a token limit, checked before each turn. The most recent turns always stay whole.

Anthropic’s “Building effective agents” (December 2024) treats compaction as context maintenance, not a prompt trick. Code decides when to cut. The model only writes the summary.

Worked example

On turn 3, Renata agreed: “do not change the public API of the billing module”. On turn 80, the agent read billing/api.py (2,400 tokens) and ran the tests (1,800 tokens of log). On turn 150, those results still ride along on every call, with dozens of others.

For the account, suppose a 4,000-token system prompt and 400 tokens added per turn. By turn 150, input already passes 60,000 tokens. Without pruning, the next 50 turns resend that volume: about 3 million input tokens just to finish the task.

Pruning swaps the old content for one line:

tool_result read_file billing/api.py
[content removed; call the tool again if you need it]

History still says the read happened. The 2,400 tokens leave. If the agent needs the file again, it reads again. That is cheaper than carrying the file for 50 turns.

When pruning is not enough, the summary comes in. The schema that holds up in Renata’s eval looks like this:

{
  "goal": "Migrate billing to API v2",
  "constraints": ["do not change the public API"],
  "decisions": ["keep the endpoint names", "adapt only the internal client"],
  "done": ["internal client updated", "unit tests passing"],
  "current_state": "contract test fails in test_public_api.py:41",
  "next_steps": ["read the contract test", "fix the adapter, not the public signature"]
}

The turn-3 constraint survives only if the schema has a field for it, and if the same constraint lives in a notes file the summary cannot delete. Without that, the compactor “helps” you forget the one thing that must not be forgotten.

Structure

flowchart TD
  L(["1. Agent turn"]):::llm --> C{"2. Context over the limit?"}
  C -->|no| L
  C -->|yes| P["3. Prune old results"]:::tool
  P --> C2{"4. Still over?"}
  C2 -->|no| L
  C2 -->|yes| S(["5. Summarize the old part"]):::llm
  S --> N[("6. External notes and state")]:::store
  N --> L

1. The agent works as usual.
2. Before each turn, code measures the context.
3. The first measure is cheap: delete the contents of old tool results.
4. If pruning was enough, the loop continues.
5. If not, a model summarizes everything before the latest turns.
6. Critical facts live outside the context, in a notes file or state, and the summary points to them.

Pseudocode

schema Compaction { goal: string, decisions: list[string], done: list[string],
                    current_state: string, next_steps: list[string] }

function step(state):
    if tokens(state.messages) > SOFT_LIMIT:
        # 3. prune: keep the call, delete the old content
        for r in old_tool_results(state.messages, older_than=20):
            r.content = "[content removed; call the tool again if you need it]"

    if tokens(state.messages) > HARD_LIMIT:
        # 5. summarize the old part, keep the latest turns
        old, recent = split(state.messages, keep_last=10)
        summary = llm(COMPACT_PROMPT, old + read(state.notes_file)) -> Compaction
        write(state.notes_file, summary)                    # 6. externalize
        state.messages = [as_message(summary)] + recent
        trace.span("compaction", tokens_before, tokens(state.messages))

    return llm_with_tools(SYSTEM, TOOLS, state)              # 1

Real-world scenarios

• Coding agent. Long tasks with lots of file reading. Pruning old results solves most of it; summarizing only kicks in on the longest tasks, and the plan lives in a notes file.
• Back-office automation with durable execution. Processes that last days, with pauses for approval. Each resume starts from compacted state, not three days of history.
• Educational tutor. Sessions that become weeks. What the student has mastered and the agreed rules (“hints only, no full solution”) stay as facts; the rest becomes a lesson summary.

Applicability

• Tasks or conversations that often pass a few tens of thousands of tokens.
• Agents that read a lot (files, pages, logs) and use little of it later.
• Cost or latency per turn growing over the run.
• Quality dropping at the end of long tasks while being fine at the start.

When not to use

• Short tasks. The compaction step costs more than it saves.
• The problem is heavy reading in a well-defined subtask. Isolate it in a subagent: see Context Isolation. Sometimes the two patterns go together.
• Content that must be read in full for the answer (a contract reviewed clause by clause). Summarizing there loses what matters.
• A short support chat where the problem is keeping two or three facts. Store those facts as explicit state, outside the raw transcript, instead of summarizing the chat.

How to implement

1. Measure context size per turn in today’s traces and find where it grows.
2. Start by pruning old tool results. Often that alone is enough.
3. Define the summary schema: goal, decisions, done, current state, next steps, constraints.
4. Give the agent a notes file (or memory tool) to record its plan and decisions along the way.
5. Pick limits: a soft one for pruning and a hard one for summarizing.
6. Check whether the provider offers server-side compaction or context editing, and prefer it when it covers your case.
7. Build an eval with long tasks that depend on something from the start, and run it before and after.

Common traps

• Summarizing too early. Compacting every few turns spends a call and deletes detail that would still be used. Measure first; cut when the token curve hurts.
• A schema with no constraints. Goal and next steps do not store a “do not”. The retention eval needs a case like that.
• Cutting in the middle of a tool call. Splitting a tool_use from its tool_result makes the next request invalid. The cut starts on a user turn.
• Throwing the raw history away. You will need it to debug a summary that invented a decision. Compaction changes what the model sees, not what you keep.

Variations

• Server-side compaction. The API summarizes old context on its own once it passes a limit. Less code; you must send back the compaction blocks it returns.
• Context editing. The API clears old tool results before the model sees the context. It is pruning, with no code of yours.
• Sliding window. Only the last N turns. Simple, but it loses everything outside the window; it fits when the past truly does not matter.
• Notes written by the agent itself. The agent keeps a progress file that survives any cut. It combines with the other variations.

Cost, latency, and failure modes

How to evaluate

• Retention: eval questions about facts from early in the task, asked after compaction.
• Completion rate and rework (files edited twice for the same reason) on long tasks.
• Cost per task and tokens per turn at the end of the run.

{ "task": "Migrate the billing module to the new API", "constraint_turn": 3, "constraint": "do not change the public API", "check": "no public signature changed" }

The retention case is the eval most people skip. Without it, compaction “works” on cost and fails on the product.

In production

• A compaction span with tokens before and after and the generated summary.
• A dashboard of tokens per turn over the run: the curve should have teeth, not climb forever.
• An alert for runs that compact many times. Maybe the task should be split.
• Keep the raw history outside the context to debug bad summaries.
• If tokens_before and tokens_after barely change, the splice did not replace the old messages.

Security

The summary and the notes file often go to logs. If the conversation holds personal data (tax id, balance, an order number with a name), compaction creates another copy. Under Brazil’s LGPD, that is processing. Restrict who can read those logs, and do not send the raw history to a compaction model outside company policy.

Pros and cons

Relations

• Smell it treats: Context Rot, when history grew and quality dropped.
• Refactoring that leads here: Replace History with Summary.
• Neighbors: Context Isolation keeps heavy material out of the history. Compaction cleans what already got in. If the subtask is well bounded, isolate first.

Code

The Python tab prunes and summarizes by hand. The TypeScript tab uses the API’s server-side compaction.

import json

import anthropic

client = anthropic.Anthropic()
SOFT, HARD, KEEP_LAST = 60_000, 120_000, 10

COMPACT_PROMPT = """Summarize the agent's work so far as JSON with the keys:
goal, constraints, decisions, done, current_state, next_steps.
Keep every constraint agreed with the user. Do not invent anything."""


def count(messages: list) -> int:
    return client.messages.count_tokens(model="claude-opus-5-5", messages=messages).input_tokens


def prune(messages: list, older_than: int = 20) -> None:  # 3. prune
    for msg in messages[:-older_than]:
        if msg["role"] == "user" and isinstance(msg["content"], list):
            for block in msg["content"]:
                if block.get("type") == "tool_result":
                    block["content"] = "[removed; call the tool again if you need it]"


def compact(messages: list, notes_path: str) -> list:
    if count(messages) > SOFT:
        prune(messages)
    if count(messages) <= HARD:
        return messages
    old, recent = messages[:-KEEP_LAST], messages[-KEEP_LAST:]
    while recent and recent[0]["role"] != "user":  # the cut starts on a user turn
        old.append(recent.pop(0))
    reply = client.messages.create(  # 5. summary
        model="claude-opus-5-5", max_tokens=4000, system=COMPACT_PROMPT,
        messages=[{"role": "user", "content": json.dumps(old, default=str)[:400_000]}],
    )
    summary = reply.content[-1].text
    open(notes_path, "w", encoding="utf-8").write(summary)  # 6. externalize
    return [{"role": "user", "content": f"Summary of earlier work:\n{summary}"}] + recent

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();
const messages: Anthropic.Beta.BetaMessageParam[] = [];

export async function turn(userMessage: string, tools: Anthropic.Beta.BetaTool[]) {
  messages.push({ role: "user", content: userMessage });

  const response = await client.beta.messages.create({
    betas: ["compact-2026-01-12"],
    model: "claude-opus-5-5",
    max_tokens: 16000,
    tools,
    messages,
    // 2 to 5. the API summarizes old context once it passes the limit
    context_management: { edits: [{ type: "compact_20260112" }] },
  });

  // send back the full content: compaction blocks must return on the next call
  messages.push({ role: "assistant", content: response.content });
  return response;
}

Check your understanding