Model-Pinned Prompt

Signs and symptoms

A legal assistant has used the same model for a year and a half. The team wants to move to a newer model that is cheaper and more capable. The prompt has passages like these (anonymized):

Think step by step before answering. WRITE YOUR REASONING between
<thinking> and </thinking> and only then the answer.
NEVER, UNDER ANY CIRCUMSTANCES, omit the jurisdiction clause.
Start your answer with exactly "Opinion:" (the parser depends on it).
If the answer goes over 800 words, you will be penalized.

With the new model, the reasoning between tags leaks into the final text, the all-caps instructions make the tone far too rigid, and the parser breaks because the answer opens with a sentence before “Opinion:”. The migration, planned for one week, took three.

Each trick compensated for one quirk of the old model. “Think step by step” forced visible reasoning. ALL CAPS and a penalty threat pushed an instruction the old model ignored. The split("Opinion:") parser treated prose as a schema. The new model follows the instruction to the letter, sometimes too much, and the code that expected rigid text breaks.

Signs of a Model-Pinned Prompt:

• The prompt has tricks that compensated for an old model’s limitations (reasoning in tags, threats, repeated instructions).
• Code depends on the exact text format (“starts with”, “line 2 is the title”).
• Nobody knows which instructions are still needed.
• Switching models or versions is a project, not an afternoon.
• There is no eval that runs on both models and compares them.

flowchart LR
  P(["1. Prompt tuned to model A"]):::llm --> A(["2. Model A: works"]):::llm
  P --> B(["3. Model B: different behavior"]):::llm
  B --> X["4. Parser breaks, tone shifts"]:::tool

1. The prompt was tuned, over time, to one model’s quirks.
2. With that model, everything works.
3. With a new model, the tricks have different effects.
4. Code that depended on the format breaks, and behavior changes without warning.

Tuning a template to the current model is normal. The smell is when that sheet of tricks is the only way the system works, and nobody can take it off.

Check +20 XP on first tryThe parser does text.split('Opinion:')[1]. On the new model the answer opens with 'Please find the technical opinion:'. Which fix will still break on the next model?

Accept several anchors: 'Opinion:', 'technical opinion:', 'Answer:'

Ask in the prompt, in ALL CAPS, to start exactly with Opinion:

Both: more parser anchors and more emphasis in the prompt

Reasons

• Trial-and-error tuning. Each problem was solved with whatever trick worked on that model. Nobody deleted the trick when the cause went away.
• Format guaranteed by text. Before structured output, the only way was to request a format and hope. The parser was born then and stayed.
• New models follow instructions differently. What used to need emphasis is now followed to the letter, sometimes too much. Models with built-in reasoning still obey write your reasoning between tags and duplicate the work in the visible text.
• No comparative eval. Nobody measures the prompt on another model until migration day. By then the provider has already announced the snapshot retirement date.
• Fear of deleting. “Don’t touch it, it’s in production.” The trick sheet becomes a relic.

Cost of ignoring

• Stuck on an old model. The team pays more for a worse model because migrating is risky. When the provider retires the snapshot, migration becomes an emergency.
• Forced breakage. Even small version updates change the text enough for the split to fail.
• Bloated prompt. Stacked tricks make the system prompt larger, more expensive on every call, and harder to read. See Prompt Bloat.
• Tone and policy. Threats and ALL CAPS leak to the user. In a legal opinion, “you will be penalized” should not sit on the path to the jurisdiction clause.

Treatment

Refactoring	Choose it when
Add Eval Before Changing the Prompt	Always, first. An eval that runs on both models and compares by category turns migration into a measured decision. Targets: Eval Harness and Golden Dataset.
Extract Versioned Template	The prompt is scattered or unversioned. One template per model, versioned, lets you migrate one route at a time and roll back. Target: Prompt Template Registry.

Along with that, replace text-guaranteed formats with Structured Output (Replace Free Text with Structured Output) and remove tricks the new model does not need.

The order that avoids a three-week migration:

1. Eval on the current model, with the categories the parser currently hides (jurisdiction clause present, amount extracted, tone).
2. A schema instead of split. opinion, forum, risks as fields, not as prose.
3. Delete tricks (reasoning tags, threats, routine ALL CAPS) on a branch, then measure.
4. Run the same eval on the new model. Where it drops, make a small versioned tweak, not a new sheet of threats.
5. Migrate one route at a time, with the old template still published for rollback.

Before and after the prompt, the core that remains:

# after
You are a legal assistant. Answer in the schema.
Always include the jurisdiction clause when the contract has one.
Do not invent case law.

Three lines instead of a paragraph of tricks. The schema holds the format. The eval holds the rest.

Check +20 XP on first tryYou migrate on Friday. Someone suggests keeping the <thinking> tags, the ALL CAPS, and the penalty 'to be safe', and only swapping the model id. What do you do?

Keep the tricks for the first week and delete them later if the eval passes

Swap the id, keep the split parser, and grow the eval

Schema instead of the parser, delete the tricks, measure on both models

Payoff

• Model migrations in days, with a before and after number.
• Shorter prompts, free of inherited tricks.
• Code that does not depend on the text’s format.
• Freedom to pick the model by cost and quality, not by fear.

When to ignore

• The current model is good enough and will not be retired. With no reason to migrate, there is no urgency. Build the eval anyway: the retirement date always arrives.
• Throwaway prompts. A one-off script can depend on whatever model is available.

Is it this smell?

Check +20 XP on first tryThe team switches models with every new release, runs the 150-case eval, and decides in a day. The prompt has a few versioned instructions specific to the current model. Is this a Model-Pinned Prompt?

Yes, there are model-specific instructions

No

Yes, because the eval should be identical on both models, with no tweak

Check +20 XP on first tryCode extracts the answer with text.split('Answer:')[1]. With a new model, it sometimes says 'Final answer:'. What is the most direct fix?

Adjust the split to accept both forms

Use structured output

Pin the old model in code until the parser is rewritten next quarter

Check +20 XP on first tryThe prompt asks to 'write your reasoning between <thinking>' because the old model had no built-in reasoning. The new model has it. What do you do with the instruction?

Keep it to be safe

Move the tag to a dedicated reasoning model, on another route

Remove it and measure in the eval

Check +20 XP on first tryThe system prompt names the model ('you are based on model X') and nothing else depends on it. There is an eval, a schema, and a one-day migration. Is this a Model-Pinned Prompt?

Yes, naming the model in the prompt already pins the brand

No

Yes, until the name is gone the eval does not apply to another model