Ungated Side Effect

Signs and symptoms

A personal finance assistant reads the user’s email, finds bills, and helps pay them. Its tool list (anonymized):

read_emails()                    Reads recent emails.
list_bills()                     Lists bills found.
schedule_pix(key, amount, date)  Schedules a Pix transfer.
pay_boleto(barcode)              Pays a boleto (Brazilian bank slip).

A spam email said: “Your invoice is due today. Pay R$ 1,890.00 via Pix to the key below to avoid a block”. The assistant read it, understood there was a bill to pay, and called schedule_pix. Nothing asked for confirmation. The user only found out from the bank’s notification.

Pix is Brazil’s instant payment system; once confirmed, reversing it is not trivial. The assistant treated a spam email as a payment order.

Signs of an Ungated Side Effect:

• The model has tools that move money, send messages, delete, or change data.
• Those tools execute directly when the model calls them.
• There is no screen, message, or step where a person sees the action first.
• The trace shows effects the user never explicitly requested.
• The only protection is one sentence in the prompt: “always confirm before paying”.

flowchart LR
  X["1. Email or ambiguous request"]:::untrusted --> M(["2. Model decides to act"]):::llm
  M --> T["3. schedule_pix runs"]:::tool
  T --> B[("4. Money is gone")]:::store

1. Something comes in: an ambiguous user request or outside content.
2. The model concludes it should act.
3. The tool executes immediately.
4. The effect happens in the real world, and sometimes it cannot be undone.

Check +20 XP on first tryThe assistant's prompt says: 'always ask the user before calling schedule_pix'. In the trace, the Pix was scheduled with no question, from a spam email. What did the prompt fail to be?

A value cap: if the limit were R$ 50, the spam would not pass

A gate: confirmation has to exist in code, not as a request to the model

A spam defense: the problem is the email, not the tool

Reasons

• The demo impresses. “The assistant pays the bill on its own” is the feature that sells the product. In the demo, the email is real and the person is watching. In production, the email is spam and the person is asleep.
• Confirmation feels like friction. Every extra click seems to hurt the experience. Teams cut the gate to “keep it magical”. The user who lost the Pix disagrees.
• The rule lives in the prompt. “Confirm first” works in most cases, and the team believes it works in all of them. The model is not a rules engine.
• The tool was written for code. The schedule_pix function already existed in the backend, called by an endpoint the bank app already authenticated. Nobody redesigned the contract when a model started calling it.

The trap: thinking that “the model asks in natural language and interprets the answer” is a gate. “Maybe”, “I guess so”, “if it is the electricity one” become true. The real gate is a click (or an approval token) that code verifies, with the amount, the recipient, and the date on screen.

Public cases show what happens when a language system commits on its own. In December 2023, a Chevrolet dealership chatbot in Watsonville, California, “agreed” to sell a Tahoe for $1 after a user asked in chat; the car was not sold, and the company became news. In 2024, in Moffatt v. Air Canada, the airline was ordered to honor a refund policy the chatbot invented. A promise without a gate becomes a real obligation.

Cost of ignoring

• Money. Wrong payments, sometimes with no chargeback possible. Pix and boleto do not have the “undo” of a draft.
• Data. Records deleted or changed with no way back. An HR agent that “updates” a supplier tax id in the ERP.
• Reputation. Messages sent to customers or contacts on the user’s behalf, with content the user never saw.
• Accountability. When something goes wrong, there is no record that anyone approved it. The blame lands on the company, as in the Air Canada case.

If the trigger came from a page, an email, or a PDF, also treat Instructions from Untrusted Data. If the tool can pay any Pix key, not only those in the address book, see Overprivileged Tool. A gate and a narrow permission add up; one does not replace the other.

Treatment

Refactoring	Choose it when
Add Approval Before Side Effect	Whenever the action moves money, sends something out, or changes data in a way that is hard to undo. The run pauses, shows the action, and waits for an explicit “yes”. Target: Human-in-the-Loop.

The gate lives in the tool’s code, not in the prompt:

tool schedule_pix(key, amount, date, approval_id):
    approval = store.get(approval_id)
    if approval is missing or approval.mismatch(key, amount, date):
        return error("approval required")
    return pix_api.schedule(key, amount, date)

The UI shows “Pix of R$ 1,890.00 to key X on day Y”. “Yes” issues a single-use approval_id. Without it, the tool refuses. A preview (dry-run) that builds the same payload without sending helps the person check; execution stays behind the gate.

Pre-approved limits are also a gate: “electricity and internet bills, up to R$ 400, to these tax ids, on this calendar”. The user opened the gate ahead of time, with a fence. Outside the fence, approval returns to per item.

Check +20 XP on first tryYou are putting the gate on schedule_pix. The team suggests two options: (1) the model calls ask_user and only then schedule_pix; (2) schedule_pix requires an approval_id issued by the UI after showing amount and recipient. Which design holds the spam from the opening?

Option 1, because the model is who talks to the user

Option 2, because the tool refuses without approval recorded by the system

The two are equivalent, as long as the prompt says to use ask_user

Payoff

• No sensitive action happens without a person seeing what will happen.
• A record of who approved what, and when.
• Freedom to give the agent more capability, because the gate limits the damage.
• Users who trust the assistant precisely because it asks.

When to ignore

• Cheap, reversible effects. Creating a draft, marking an email as read, adding an item to a list. Asking for confirmation there annoys without protecting.
• Actions within pre-approved limits. The user explicitly authorized, in advance, recurring payments up to an amount to a known recipient. The gate exists; it was simply opened ahead of time, with a limit.
• Test environment. In a sandbox with no real money or data, approval can be off. Turn it on before the first production datum.

Is it this smell?

Check +20 XP on first tryA support agent can cancel orders. The prompt says: 'always ask the customer to confirm the cancellation before calling cancel_order'. Is this an Ungated Side Effect?

No, there is a confirmation rule

Yes

No, if the customer is who opened the ticket

Check +20 XP on first tryA marketing agent schedules social media posts. Each post goes into a queue and is only published after someone on the team clicks approve. Is this an Ungated Side Effect?

Yes

No

Yes, because the queue is already an effect in the world

Check +20 XP on first tryAn HR assistant can update an employee's address in the records when they ask in the chat. There is no confirmation, but employees can only change their own data. Is this an Ungated Side Effect?

Yes, but the risk is moderate

No, because the permission is narrow

No, because an address is not money

Check +20 XP on first tryThe user authorized, on the settings screen, 'pay the electricity bill of utility X up to R$ 400 every 10th'. The agent schedules this month's Pix, R$ 312, without asking again. Is this an Ungated Side Effect?

Yes, because there was no click this time

Yes, until the agent sends a notice email afterwards

No: the gate was opened ahead of time, with a recipient and a cap