Hardcoded Few-Shots

Signs and symptoms

A software company’s ticket triage uses examples to classify. They live in a constant, in the middle of the code (anonymized):

# classifier.py
FEW_SHOTS = [
    ("I can't access the dashboard", "access"),
    ("Double charge on my card", "billing"),
    ("The sales report is empty", "bug"),
    # TODO: ask the ops team for integration examples
]

The operations team noticed that tickets about the ERP integration land in “bug” when they should go to “integration”. They know exactly which examples would help. To change them, they have to file a ticket with engineering, wait for someone to pick it up, review the PR, and wait for next week’s deploy. The TODO has been there for four months.

While the ticket sits, the model still sees only three classes. A ticket like “the ERP webhook does not fire the NF-e” (a Brazilian electronic invoice) becomes a report bug, because the nearest example is the empty sales report.

Signs of Hardcoded Few-Shots:

• Prompt examples live in constants or strings in the source code.
• Changing an example requires a pull request, review, and deploy.
• The people who know the domain (operations, support) cannot edit the examples.
• There is no history of why each example was added.
• The examples in code and the eval cases are mixed together, or are the same.

flowchart LR
  O((("1. Operations team"))):::human --> T["2. Ticket to engineering"]:::tool
  T --> PR["3. PR and review"]:::tool
  PR --> D["4. Weekly deploy"]:::tool
  D --> P(["5. New example in the prompt"]):::llm

1. Whoever knows the problem has no access to the examples.
2. The change becomes a request to another team.
3. An engineer makes a text change as if it were code.
4. The fix waits for the deploy cycle.
5. Weeks later, the example goes in.

The smell is not “examples in git”. It is the cycle: only engineering can change them, only on deploy, with no owner and no recorded reason.

Check +20 XP on first tryOps has 4 ready examples of ERP integration. To reach the classifier, the path is a ticket, a PR, and Thursday's deploy. What is the smell?

Untested Prompt, because nobody measured the 4 examples

Hardcoded Few-Shots: the people who know the domain cannot publish the example

Few-Shot Hoarding, because four is already too many

Reasons

• It started as a prototype. A constant in code was the fastest way to test. The prototype shipped with the TODO on line 7.
• The prompt treated as code. Since it lived in the repository, it followed the code’s cycle: PR, application CI, weekly deploy. Domain text does not need that cycle.
• There was nowhere else. Nobody built an example store with versions and permissions.
• Fear that ops will break production. With no eval at publish time, engineering becomes the only gate. The right gate is the eval, not the deploy.

Cost of ignoring

• Slowness. Simple fixes take weeks. On WhatsApp support, four months of ERP tickets classified as bugs is a queue on the wrong team.
• Lost knowledge. People who know which examples would help stop asking.
• Engineering busy with text. Expensive people make changes that are not engineering.
• Risk. With no separate eval, a new example can hurt another category without anyone noticing.
• Contamination. If the same 3 examples in code are the eval, the eval always passes and never sees the ERP.

Before and after the flow

	Hardcoded	Dataset
Who edits	Engineering, via PR	Ops, with review
When it ships	Next deploy	After the eval, in minutes
History	File blame, if anyone commented	Owner, reason, date
Relation to the eval	Often the same list	Separate lists

Treatment

Refactoring	Choose it when
Move Few-Shots to a Dataset	Whenever examples need to change often, or by people outside engineering. They go into a versioned set, editable with review, and enter the prompt at request time. Targets: Few-Shot Selector and Golden Dataset.

The destination has three rules teams forget when they “just dump it in a spreadsheet”:

1. Immutable version. Publishing creates erp-triage@14, it does not overwrite row 4.
2. Eval before it counts. The new integration examples must not tank “billing” accuracy. The eval set is a different set.
3. Domain experts edit. Engineering builds the pipe (schema, permission, selector). Ops chooses the examples.

A YAML file in the repository, with ops opening a PR on that file only and a job applying it after the eval, is already a dataset. A constant in classifier.py is not.

Check +20 XP on first tryYou moved the examples to a spreadsheet ops edits freely. The classifier reads the spreadsheet on every request. There is no review and no eval. What did you just swap?

Nothing serious: the smell was slowness, and slowness is gone

A slow cycle for a cycle with no version and no measurement

Hardcoded Few-Shots for Unversioned Prompt, and that is all

Payoff

• People who know the domain edit the examples, with review and history.
• Changes go live in minutes, after passing the eval, with no code deploy.
• Every example has an owner, a reason, and a date.
• Examples and eval cases stay separate.

When to ignore

• Stable, technical examples. Format examples that change once a year can stay in code.
• Small team, same person. If whoever knows the domain is also whoever deploys, the cost is lower. The gain from versioning and separating from the eval remains.

Is it this smell?

Check +20 XP on first tryThe examples live in a YAML file in the repository. The operations team can edit it through pull requests, and a job applies changes without an application deploy, after running the eval. Is this Hardcoded Few-Shots?

Yes, they are in the repository

No

Yes, because ops should not touch git

Check +20 XP on first tryThe examples live in a spreadsheet the operations team edits freely, and the system reads it on every request, with no review and no eval. What is the risk?

None, it is editable

Changes with no version and no eval

Only if the spreadsheet leaves the company, for privacy law

Check +20 XP on first tryThe prompt has 35 examples in code and nobody knows why each one is there. Which smell stands out?

Hardcoded Few-Shots

Few-Shot Hoarding

Prompt Bloat, and that is all

Check +20 XP on first tryThe classifier's only examples are three JSON-format strings in code, unchanged for a year. Ops has never asked to edit them. Is this Hardcoded Few-Shots?

Yes, any example in code is this smell

No, and you can ignore it with complete ease

Not the urgent case; the gain from versioning is small while nobody needs to edit