Opening

A building materials distributor in Brazil receives a few thousand supplier documents a month. Printed electronic invoices, service invoices from city governments, each with a different layout, and supply contracts with price adjustment clauses.

Three people in finance type all of this into the management system. During month-end close, the queue falls behind, and payments go out with late fees.

The goal is to read each document and fill in the system with no manual typing. Under one clear rule. When in doubt, the document goes to human review. It never enters with wrong data.

This is the first real system of this trail. And almost every smell and refactoring from the trail shows up in it. You'll see the first version break four times, and how the final one turns silent errors into visible ones.

Version one

The first version took a week to build, and it got all ten demo documents right.

The file arrives by email or through a portal, and a library pulls out the text.

One prompt asks for the issuer, the line items, the taxes and the totals. And to check that the sum matches.

Regular expressions pull the values out of the answer text.

And the data goes into the system.

What broke

Week two. The model started writing invoice total amount, instead of total. The regular expression didn't find the field, and one hundred forty documents went in with no total. Nobody noticed for four days.

Week four. An invoice with thirty-eight line items went in with a total off by twelve cents. The model had added the items, and rounded differently. The error only surfaced during bank reconciliation.

Week five. Someone asked the prompt to keep two decimal places, and tested it on three invoices. Accuracy on the discount field dropped. But only on discounted invoices, and none of them were among the three.

Week seven. A tax ID with an invalid check digit was accepted, because the model validated it and said it was fine. The supplier didn't exist.

Now notice what the four incidents share. None of them raised an error. The system moved on with empty or wrong data, and another team found it, days later. In document extraction, a silent failure is the worst kind. Its cost grows every day nobody notices.

The smells

You already know these smells by name. Free text handoff. A prose answer, read by a regular expression. That's week two.

LLM doing deterministic work. Sums, rounding and tax ID validation, done by the model. That's weeks four and seven.

And the untested prompt. A change tested on three examples. Week five.

A challenge

Here's one for you. A colleague proposes keeping the sum check in the prompt. The model would read the line items, and answer ok, or mismatch. Why doesn't the final architecture do that?

Because adding is deterministic work. Code gets it right every time, for free. The model produces text that looks like a check, with no guarantee that it checked anything. Reading the line items stays with the model. The arithmetic leaves it. And comparing the two is what catches a misread.

The fixes, in order

Here's what the team did, in order. First of all, the eval. One hundred eighty real documents, anonymized, with the right value for each field, split by document type and by issuer. Version one was measured against them before anything changed.

Then, structured output. Extraction now returns an object in a schema, with amounts in cents and typed dates. The regular expression was deleted.

Then, the math moved into code. The model only reads printed values. Sums, taxes and tax ID validation became tested functions. And a mismatch between the calculated total and the printed one became an alarm.

The final architecture

Here's where it landed. The whole document goes to the model, which reads both the text and the image.

A short step classifies the type. A printed invoice, a service invoice, or a contract.

Each type has its own prompt and its own schema. Long contracts are read in parts, and the results combined.

Code adds up the items, checks the taxes, and validates the tax ID and the dates.

If everything matches, the data goes into the system, with a reference to the document and the prompt version.

If something doesn't match, the error goes back to the model once, with the exact validation message.

If the repair fails, the document goes to a person, with the problem field highlighted.

And every human correction becomes a new case in the dataset. The eval grows on its own, through use, with no separate labeling project.

And one decision in there deserves attention. Say the validator reports the item sum differs from the printed total. The repair rereads the page, and returns the same values. What now?

Don't accept it because the model confirmed the reading twice. Two identical readings don't prove they're right, and the invoice itself may truly be inconsistent. And don't retry until the sum matches. That teaches the model to change some number so the math closes, and it swaps a visible error for an invisible one. Send it to a person, with the field highlighted. And store the correction in the dataset.

And one more choice that saves a lot. Most Brazilian invoices also exist as a signed, structured file, authorized by the tax authority. When the distributor can get that file, it parses it in code and skips the model completely. There's nothing to interpret. Sending it to the model would only add cost and a chance of error. The model stays for service invoices, contracts, and documents with no structured file.

In production

So which number answers whether the system meets the rule, when in doubt, review? Not the average accuracy. The average hides where the error is, just like week five. And not the share of documents that skipped review. That measures automation, not safety. It rises when the system improves, and also when it starts letting errors in.

The number is how many documents entered without review, with a critical field wrong. That's the error the rule forbids, and it has to stay at zero. The review rate is what you optimize after, without ever letting that first number rise.

And when a vendor offers a cheaper model, you already know what to do. Run the one hundred eighty documents on both versions, with accuracy per field and per type. Then the saving is either worth it, or it isn't.

Wrap-up

So here's what this system teaches. Extraction and arithmetic are different jobs, and the mismatch between them is your best error detector. A loud error beats a silent one. The eval comes first, and pays for itself. Human review is part of the architecture. And every human correction is a test case. The goal was never zero review. It was zero wrong data.

In the next lesson, the last of this trail, we'll let the model write database queries, and see how to keep that safe: natural language to SQL. See you there.