Opening

Now that you can measure, we start fixing. First, a smell that almost every team building on models has, and almost nobody notices until month-end.

A company extracts data from supplier contracts with two steps. The first one reads the contract and lists the main data. The second takes that text and fills in the record in the company's management system.

And between the two steps, there's this. A regular expression that looks for the word Total, a colon, and a number.

It worked for three months. Then one day, the model started writing total contract value, instead of just total. The regular expression found nothing. The total became empty. And one hundred forty contracts went into the system with no amount, before anyone noticed.

Nothing broke. No exception, no alert. The data was just empty. In this lesson, you'll learn to recognize this smell, why it's so tempting, and the one refactoring that removes the whole class of errors.

The diagnosis

Here's the shape of it. The first step produces text meant for a person to read.

And that text is the only interface between the two steps.

Code, or sometimes the next model, tries to guess where each piece of data is.

When the guess is right, everything works.

When the model changes how it writes, the data disappears, with no alarm.

How do you spot it? One step returns prose, and the next step uses a regular expression, a split, or find the line that starts with.

The code tolerates several names for the same field. Total, total value, grand total.

The second step is another model call, whose only job is to understand the first one's format.

Changing one word in a prompt breaks a parser in another file.

And missing fields silently become empty, instead of raising an error.

Why it happens

It usually starts with a prototype in a chat window. Someone tested the prompt, liked the text answer, and pasted it into the code. In a chat, text is the product. In a pipeline, text is a live wire.

Text feels flexible. You don't have to decide names, types, or what's required before starting. A schema makes you decide.

And it works, for a while. Models often repeat the format, until the day they don't. The model changed, the prompt changed, or just the input did. A contract that says amount instead of total.

And here's the trap for someone starting out. Asking for JSON in the prompt, reply with JSON only, and thinking that settled it. That's still text. The model can wrap it in formatting, add a comma, or leave out a key.

What it costs

The worst cost is silent bad data. Nothing breaks. One hundred forty contracts with no total raise no exception. Finance only sees the hole at month-end close.

Then there's the coupling nobody can see. The first step's prompt becomes part of the second step's contract, and nobody knows it. The parser lives in another file, another repository, maybe another team.

Then fear. Any prompt tweak or model swap becomes a risk. The team stops improving the extractor, because finance's regular expression will break.

And wasted tokens. Prose meant for a parser spends output on words nobody reads. The total contract value, taxes included, is. Dozens of tokens a single field doesn't need.

A challenge

Here's one for you. The team asked for JSON in the prompt, with the keys total and date, and the code parses it. Once a week, the run breaks on an extra comma, or a formatting fence around the object. Which next step removes the whole class of error?

Use the API's structured output, with a schema, and validate the fields in code. Format stops being a request to the model, and becomes a guarantee from the API.

Retrying until it parses hides the instability. And on the day the variation is a string where a number belongs, the parse succeeds, and the bad data goes in. Adding three examples of valid JSON lowers the rate, but never to zero. You'd still depend on the model obeying a format instruction.

The treatment

The treatment is a refactoring called replace free text with structured output. The contract between the steps becomes explicit.

The supplier's tax ID, the invoice's access key, which may be missing, the total, typed as money, and the list of due dates.

The API guarantees the shape. The code reads the total field, not a sentence. A phrase like total contract value simply has nowhere to go anymore. And a missing required field becomes an error right away. The document goes to human review, instead of entering the system as zero.

And often there's a bonus. If the second step only existed to turn the text into fields, it can go away. The first call already returns the fields.

But a schema doesn't replace an eval. A correct shape can still carry the tax ID of a company that isn't the issuer. The eval measures whether the value is right. The schema guarantees the value arrived as a value.

When to ignore

When can you ignore it? When the destination is a person. A chat reply or an email to a customer should be text. The smell is text between machines.

When the text is the product. One step writes an article draft, and another reviews it. Passing text between them is perfectly legitimate. Structure what code actually uses, like the list of problems found, or whether it was approved. Not the article itself.

And a throwaway prototype is fine. The day the parser goes into the real system, the schema goes in with it.

One last case, to sharpen the eye. A ticket classifier answers with a single word, billing, order or other, and code compares the text. Sometimes it returns billing with a capital letter and a period, and the ticket lands in the wrong queue. Is that this smell? Yes. Even one word is free text, if your code depends on its exact form. A schema with a closed list of values fixes it, with just one field.

Wrap-up

So here's the idea to take with you. Text is for people. When one step hands data to code, or to another step, the contract has to be a schema, not a sentence. Otherwise your system works until the day the model finds a nicer way to say total.

In the next lesson, we'll do the refactoring itself, step by step, with an eval before and after: replace free text with structured output. See you there.