Rule Stacking

Signs and symptoms

A health insurer’s assistant answers member questions. This is an excerpt of its system prompt after a year of incidents (anonymized):

IMPORTANT: NEVER give a diagnosis.
ALWAYS be empathetic.
NEVER mention procedure prices.
ATTENTION: ALWAYS state the copay amount when asked.
DO NOT answer about medication.
IMPORTANT!!! For severe symptoms, tell them to go to the emergency room.
ALWAYS answer in at most 3 sentences.
NEVER leave the member without a complete answer.
... [17 more rules in capitals]

Notice three conflicts: “never mention prices” versus “always state the copay”; “at most 3 sentences” versus “never leave without a complete answer”; and the severe-symptoms rule, the most important of all, buried among 24 others with the same visual weight.

In a test, the assistant answered “I have had chest pain for an hour” with a three-sentence explanation of cardiology coverage. It followed the brevity rule and ignored the urgency one.

Signs of Rule Stacking:

• The prompt has many lines with IMPORTANT, NEVER, ALWAYS, ATTENTION, in capitals or with exclamation marks.
• Each rule was born from a specific incident.
• Some rules contradict each other, and nobody decided which one wins.
• Critical rules have the same emphasis as trivial ones.
• Adding a rule fixes one case and breaks another.
• The model follows some rules with excessive rigidity and ignores others.

flowchart LR
  I(["1. Incident"]):::io --> R["2. New rule in capitals"]:::tool
  R --> P[("3. Rule stack")]:::store
  P --> M(["4. Model picks which to follow"]):::llm
  M --> I

1. A case goes wrong in production.
2. The quickest fix is a new rule, with emphasis to “make sure” it works.
3. The rule joins the earlier ones, with no review of the whole set.
4. With many rules of equal weight, the model resolves conflicts its own way. New incidents appear.

Check +20 XP on first tryThe test sends 'chest pain for an hour'. The bot explains cardiology coverage in three sentences and does not send the person to the ER. Which rule won, and why?

The urgency rule, because it is marked IMPORTANT

The brevity rule, because it competed on equal footing with urgency and was simpler to obey

Neither: the model hallucinated coverage because it has no procedure table

Reasons

• A rule is the cheapest fix. One line resolves the incident at hand and closes the ticket.
• Emphasis seems to work. Capitals sometimes change behavior in a test, and it becomes a habit.
• Nobody owns the whole prompt. Each team adds its own rules without reading the others’.
• Without an eval, deleting is scary. Nobody knows which rules are still needed.
• Current models follow instructions more literally. Emphasis that helped older models now produces excessive rigidity.

DPD’s chatbot (2024) insulted a customer after an update. A prompt that accumulates incident fixes is fragile in the same way: one edit touches a rule, and another rule nobody was watching stops being followed.

Cost of ignoring

• Unpredictable behavior. With conflicting rules, the answer depends on which one the model prioritized that time.
• Risk on critical rules. The rule that matters competes for attention with twenty that matter less.
• Rigidity. Absolute rules applied out of context produce absurd answers (“I cannot talk about prices” to someone asking about their copay).
• Maintenance. Every change requires understanding how it interacts with every other rule.

The insurer’s conflicts, side by side

Rule A	Rule B	What the model does in practice
NEVER mention prices	ALWAYS state the copay	Refuses the amount, or invents a halfway answer
At most 3 sentences	NEVER leave without a complete answer	Trims the urgency guidance to fit
DO NOT talk about medication	Route severe symptoms	Talks coverage and skips “go to the ER”

Copay here is the member’s share of a procedure under a Brazilian health plan (coparticipação). The person is asking for a number the contract says you must give. The stack treats that number as if it were a clinic’s price list. Privacy rules (LGPD in Brazil) and health routing are not solved with one more line in capitals: the flow decision needs to be code.

Treatment

Refactoring	Choose it when
Replace Rules with Examples	The rules try to describe tone, format, or edge-case behavior. A few well-chosen examples show what twenty rules try to say. Target: Few-Shot Selector.
Replace Prompt Rule with Graph Edge	The rule is a flow decision (“for severe symptoms, do X”). It leaves the prompt and becomes a code step that cannot be ignored. Targets: Routing and Prompt Chaining.

Before either, build an eval from the incidents that produced the rules. It is what makes deleting safe: see Add Eval Before Changing the Prompt.

How to slice the insurer’s stack:

1. Urgency becomes a classifier plus a code route to the ER message. It no longer competes with “3 sentences”.
2. Copay becomes a tool that looks up the amount in the contract. The “never cite procedure prices” rule stops fighting a number the tool returns.
3. Tone (empathy, brevity) becomes two or three examples, not eight ALWAYSes.

Deleting comes after measuring. The temptation is to put the critical rule in capitals at the top. That can raise the rate in a test and still fail on a Friday night.

Check +20 XP on first tryOne rule says 'if fraud is suspected, transfer to the security team'. It sits among 15 style rules, and the model sometimes does not transfer. What do you do?

Put the rule in capitals at the top of the prompt

Repeat the rule three times, at the start, middle, and end

Turn the rule into a code step, with a suspicion classifier and routing

Payoff

• Critical rules guaranteed by code, not by emphasis.
• A short prompt that explains the context and goal instead of listing prohibitions.
• Consistent behavior, with no conflicts left for the model to resolve.
• Each incident becomes an eval case, not one more line in the prompt.

When to ignore

• A few rules, no conflict. Three or four clear, compatible instructions are normal in any prompt.
• Simple format constraints. “Answer in English” or “use at most 200 words” do not need to become code.
• Prototype. To discover the desired behavior, quick rules are acceptable. Consolidate before going to production.

Is it this smell?

Check +20 XP on first tryThe prompt has 6 rules, all about answer format, with no conflicts, and the model follows all of them in the eval. Is this Rule Stacking?

Yes, that is too many rules

No

Yes, because any ALWAYS or NEVER is already the smell

Check +20 XP on first tryThe prompt is 4,000 tokens: 30 paragraphs of company policy and 5 behavior rules. What is the main smell?

Rule Stacking

Prompt Bloat

Routing System Prompt

Check +20 XP on first tryTwo teams edit the same prompt. Billing adds NEVER offer a product. Sales adds ALWAYS offer an upgrade. The eval swings week to week. What is the diagnosis?

The eval dataset is weak and needs more golden cases

You need a larger model, able to reconcile the two rules

Rule Stacking on top of a prompt that also routes by topic: the emphasis cancels out

Check +20 XP on first tryYou built the eval from the 20 incidents that produced the rules. Removing 12 lines in capitals does not change the eval. What do you do with the 12?

Keep them, because a similar incident might come back someday

Move the 12 to the top, together, so the model does not forget

Delete the 12 and let the eval catch a regression if any of them was still needed