LLM Doing Deterministic Work

Signs and symptoms

A Brazilian distributor runs an extractor for NF-e, the electronic invoice every Brazilian company must issue. It asks the model for the invoice line items, the total amount, and whether the sum of the items matches the total. An excerpt from the prompt (anonymized and translated):

Extract the invoice line items with quantity, unit price, and line
amount. Add up the line amounts, apply the discount, and check that the
result matches the invoice total. Also compute each item's ICMS (state
sales tax) with the rate shown. Convert dates to YYYY-MM-DD format.
Validate that the issuer's CNPJ (company tax ID) is valid.

Once every few hundred invoices, the sum is off by cents, one item’s tax is rounded differently, or an invalid tax ID gets marked as valid. The team cannot find a pattern, because there is none: it is the model doing arithmetic.

Signs of an LLM Doing Deterministic Work:

• The prompt asks to calculate, add, convert, format, or validate something that has an exact rule.
• Errors are rare, small, and patternless: a cent, a digit, a day.
• The same document gives different results on different runs.
• There is code in the system that redoes the math “to double-check” and sometimes disagrees.
• Dates, currencies, and identifiers come out in slightly different formats.

flowchart LR
  D(["1. Document"]):::io --> M(["2. Model extracts and calculates"]):::llm
  M --> C["3. Sum, tax, validation"]:::tool
  C --> R(["4. Result that is almost always right"]):::io

1. The document arrives.
2. The model does two things in one call: understand the document and do math.
3. The math is generated token by token, as text, with no guarantee of exactness.
4. Almost always the result matches. The “almost” is the problem.

Why the model gets simple math wrong

The model does not add. It predicts the next token. A number like 1,284.37 becomes a few tokens, and the tokenizer does not always split it in the same places. To “add”, the model writes digits that look plausible after what came before.

Most of the time, plausible and correct coincide. Current models get far more math right than the ones from a few years ago. But “far more” is not “always”, and a tax system needs always. A 40-item sum with a discount and rounding is exactly the kind of math where one digit slips.

Check +20 XP on first tryYour team switched to a bigger model and the invoice sum now fails once every 2,000 invoices instead of once every 300. The manager wants to call the problem solved. What do you answer?

That it is solved, because the rate is now acceptable

That the math has an exact rule and should leave the model, because a better rate is not a guarantee

That it is enough to ask the model in the prompt to double-check its own sum

Reasons

• Writing a sentence in the prompt is faster than writing a function with tests.
• The model gets it right in testing. With five sample invoices, the math comes out right. The error shows up at scale.
• The line between understanding and calculating is blurry. “Extract the total” looks like “calculate the total”, and it is not. The first is reading the printed number. The second is producing a new one.
• The model seems able to do everything. It can do a lot, but not with the guarantee of a function.
• Rules nobody wants to maintain. Validating a tax ID or counting business days takes code and tests. Pushing it into the prompt feels like skipping that work.

Cost of ignoring

• Financial and tax errors. One wrong cent on an invoice creates accounting rework and, at scale, trouble with the tax authority.
• Distrust. When the error is rare and random, nobody trusts any result and everything gets checked by hand.
• Cost. Output tokens spent on math that one line of code does for free. Output tokens usually cost several times more than input tokens.
• Flaky tests. The same case passes and fails on different runs, and the team learns to ignore red in CI.
• Rules that change without warning. Since July 2026, new Brazilian CNPJs can contain letters in their first 12 positions (Receita Federal Normative Instruction 2,229/2024). A rule like that must be updated in one place, with a test. In the prompt, you find out about the change when the model starts rejecting valid IDs.

Treatment

Refactoring	Choose it when
Replace LLM Call with Code	Whenever the operation has an exact rule. The model extracts the raw data; code calculates, formats, and validates. Target: Tool Use, when the model needs the result in the middle of its reasoning (for example, to explain a delivery date to a customer).

The same extractor, split

The rule for splitting: the model copies what is written in the document, and code produces any new number. In practice, the model’s output becomes a structured output with only values it read (field names stay in Portuguese, as in the real system):

{
  "emitente_cnpj": "12.ABC.345/01DE-35",
  "data_emissao": "03/09/2026",
  "itens": [
    { "descricao": "Caixa papelão 40x30", "qtd": "120", "valor_unit": "3,15", "aliquota_icms": "18" },
    { "descricao": "Fita adesiva 48mm", "qtd": "36", "valor_unit": "7,90", "aliquota_icms": "18" }
  ],
  "desconto": "25,00",
  "total_impresso": "634,40"
}

Notice the values come as text, exactly as printed (Brazil uses a comma as the decimal separator). The code converts, adds, validates, and compares:

import re
from decimal import Decimal, ROUND_HALF_UP

def brl(s: str) -> Decimal:
    return Decimal(s.replace(".", "").replace(",", "."))

def dv(base: str) -> str:
    soma = sum((ord(ch) - 48) * (2 + i % 8) for i, ch in enumerate(reversed(base)))
    resto = soma % 11
    return "0" if resto < 2 else str(11 - resto)

def cnpj_valido(cnpj: str) -> bool:
    c = re.sub(r"[./-]", "", cnpj.upper())
    if not re.fullmatch(r"[0-9A-Z]{12}[0-9]{2}", c) or len(set(c)) == 1:
        return False
    return dv(c[:12]) == c[12] and dv(c[:13]) == c[13]

def conferir(nota: dict) -> dict:
    itens = [brl(i["qtd"]) * brl(i["valor_unit"]) for i in nota["itens"]]
    total = sum(itens) - brl(nota["desconto"])
    icms = [(v * brl(i["aliquota_icms"]) / 100).quantize(Decimal("0.01"), ROUND_HALF_UP)
            for v, i in zip(itens, nota["itens"])]
    return {
        "total_calculado": total,
        "bate_com_impresso": total == brl(nota["total_impresso"]),
        "icms_itens": icms,
        "cnpj_ok": cnpj_valido(nota["emitente_cnpj"]),
    }

On this invoice, 120 times 3.15 is 378.00 and 36 times 7.90 is 284.40. The sum minus the discount is 637.40, not the printed 634.40. Before, the model might have “checked” and said it matched. Now the mismatch is a calculated fact, and the invoice goes to review with the reason attached.

Use Decimal, never float, for money. The ICMS rounding rule (here, half up to two places) is the tax team’s decision, and now it is written in a place you can test.

Check +20 XP on first tryIn the new version, the model returns valor_unit as text read from the invoice and code computes the totals. A colleague suggests asking the model to return each line amount already multiplied, to save code. Why is that a step back?

Because the multiplication goes back to being a number produced by the model, with no guarantee

Because the output gets bigger and more expensive

Because the model cannot multiply decimal numbers

Payoff

• Exact math, every time, testable with ordinary unit tests.
• The model focused on what only it can do: read a messy document.
• Smaller, cheaper outputs.
• Extraction errors separated from calculation errors, easy to diagnose.
• Rule changes (like the alphanumeric CNPJ) in one place, with a test.

When to ignore

• Estimates. “Roughly how many pages is this report?” does not need exactness.
• Math that depends on interpretation. Deciding which value in the document is the total is already interpretation; the sum afterwards is math.
• Prototype. To validate that extraction works, letting the model add is acceptable. Take it out before production.

Is it this smell?

Check +20 XP on first tryAn assistant answers customer questions about delivery times. The prompt says 'calculate the delivery date by adding 5 business days to the purchase date'. Sometimes it counts the holiday. Is this an LLM Doing Deterministic Work?

No, because it is a conversation with the customer

Yes

No, the problem is that the prompt does not list the holidays

Check +20 XP on first tryA classifier decides whether a comment is offensive. There is no exact rule, and the model sometimes disagrees with itself on borderline cases. Is this an LLM Doing Deterministic Work?

No

Yes, because the result varies

Check +20 XP on first tryThe model extracts each line amount from an invoice and code adds them up. The code's total sometimes does not match the total printed on the invoice. Where is the problem?

In the code's sum

In extraction or in the invoice itself, and the mismatch is a good sign

Check +20 XP on first tryAn HR agent asks the model to convert 'March 3rd' into 2026-03-03 before saving it to the database. The format comes out right 99% of the time. What is the simplest treatment?

Add date examples to the prompt

Use the model at temperature zero

Ask the model for the date as written and format it in code