Opening

A math tutor has been working with a student for twelve lessons. All of it in the same conversation.

In lesson one, she asked: don't show me the solution, just hints. Deal, said the tutor.

In lesson six, for a review sheet, she asked the opposite. For this sheet, you can show the full solution.

Now it's lesson twelve, and the history is almost ninety thousand tokens. She asks for help with question four. And the tutor switches between the two rules, with no clear logic.

Nobody told the tutor that the lesson six rule applied only to that sheet. Nobody summarized what the student has already mastered. The model gets everything, and tries to guess what still holds. In this lesson, you'll learn to recognize this smell, and what to put in place of an ever growing history.

The diagnosis

Here's the cycle. A new turn arrives.

Code appends it to the full history, which never shrinks.

The model reads tens of thousands of tokens, most of them irrelevant, or stale.

The answer gets worse, and costs more. And it goes into the history too, so the cycle continues.

How do you spot it? The conversation, or the agent session, only grows. Nothing ever leaves.

The model cites old information as if it were current. An order from three months ago, a file that has since changed.

Instructions given at the start get ignored, or mixed with later ones.

Cost per turn climbs through the session, even for short questions. And so does the time until the first word appears.

And answer quality drops in long sessions, while it's fine in short ones.

Why it happens

The default examples just append. The simplest chat code keeps the message list, and sends it all again. It works for the first ten turns.

Large windows feel safe. If a million tokens fit, there seems to be no reason to cut. But fitting isn't the same as being used well. Models use what sits in the middle of a long context less reliably. And the lesson one and lesson six instructions fight in exactly that middle. The window fits. Attention doesn't.

Deleting is scary. Without knowing what matters, keeping everything feels safer.

And agents produce a lot of text. Tool results, files they read, test output. All of it enters the history, and stays there, long after it stopped being useful.

What it costs

The cost starts with quality. Rules and facts buried in the middle are followed less reliably, and contradictory information produces unstable answers.

Then money, with illustrative numbers. A system prompt of two thousand tokens, and four hundred more each turn. At turn three, a call sends about three thousand two hundred tokens. At turn twelve, about six thousand eight hundred. At turn forty, about eighteen thousand.

The student's question fits in one line. The bill doesn't. Prompt caching makes a stable prefix cheaper, but history that changes every turn doesn't sit in that cached prefix. And at some point the session overflows the window, and the cut happens in the worst place, with no logic at all.

The same thing shows up in a bank's WhatsApp support. In month one, the customer mentioned a cancelled instant payment. In month three, they ask for their balance, and the bot still talks about that payment as the current problem. The new message was twelve words. The context held months of support.

A challenge

Here's one for you. A support chat always sends only the last ten messages, and drops the rest. Now customers complain that the bot forgets the order number they gave at the start. Is that context rot?

No. The context isn't rotten, it's too short. Cutting by a fixed window fixes the cost, but loses facts. What's missing is explicit conversation state, a short term memory. The order number belongs there, outside the raw history, where your code keeps it up to date.

The treatment

There are two refactorings. When the session is a long conversation, replace the history with a summary. Old turns become a summary, and facts that must last, like preferences and agreed rules, move into explicit state.

And when the context bloats with reading work, like files, pages or logs, extract a subagent. It reads in a clean context, and returns only the conclusion.

Here's lesson twelve, before and after. Before, ninety thousand tokens of raw chat, with two rules in conflict, buried in the middle. After, a short summary, the state, with the student's level and the current rule, and the most recent turns. The active rule lives in one field, that you can read, edit and test.

And watch for one mistake when you start cutting. Say you summarize every twenty turns. In month three, the summary still says: customer with a cancelled payment, treat as urgent. And the bot talks about it again, when the customer only asked for their balance. You only compacted the rot. A summary needs validity. What holds now goes into state, with a lifetime. And the past stays in the summary as past, or it leaves.

A coding agent

One more, from the agent world. A coding agent read forty files, found the cause of a bug, and is now writing the patch. All forty files still go out on every turn. What's the most useful next step?

Clear the old tool results, or move the reading into a subagent that returns only the conclusion. The files have done their job. Now they push the patch instruction into the middle of the context, where the model reads it worse. A bigger window doesn't fix that. And telling the model to ignore the files it won't use competes with tens of thousands of tokens that are still there. Code has to drop what's unused. Not the model.

Close relatives

And one case that looks similar, but isn't. A system prompt of eight thousand tokens, that never changes between turns, in conversations of about six turns. Cost per message is high. But the context doesn't grow during the session. The weight is fixed in the prompt. That's prompt bloat. Trim the prompt, and if the start is stable, cache it.

When to ignore

When can you ignore it? Sessions that are short by nature. A support chat that lasts five turns doesn't need compaction.

Tasks where the whole input is the document. Reviewing one contract at once is different from carrying an old conversation. There, the text is the work, not leftover turns.

And a prototype. Appending everything is the fastest way to test an idea. Just don't ship long sessions that way.

Wrap-up

So here's the idea to take with you. A long history isn't memory. It's a pile the model has to dig through on every turn, and pay for. Keep the facts that must last in explicit state, summarize the past as past, and let finished work leave the context.

In the next lesson, we'll do the first refactoring step by step: replace history with summary. See you there.