Vibe-Check Eval

Signs and symptoms

A recruiting company uses a model to pre-screen resumes. Before each change, the coordinator runs five resumes she knows well and checks that the score “makes sense”. Last week’s change was approved this way:

Test of prompt v8 (5 resumes):
- Ana, senior dev: 9   (ok)
- Bruno, junior: 5     (ok)
- Carla, analyst: 7    (ok)
- Diego, intern: 4     (ok)
- Elisa, senior: 8     (ok)
Approved. Ship it.

Three months later, an internal audit showed that candidates from an entire state received systematically lower scores. Prompt v8 had added “value experience at leading companies”, and the model associated “leading” with companies from two capital cities. None of the five test resumes came from that state.

This is not unique to LLMs. In 2018, Amazon discontinued an internal screening tool that penalized resumes with signals associated with women, such as the word “women’s”. The bias only became visible when someone measured by group, not when someone “checked that it made sense”.

Signs of a Vibe-Check Eval:

• Quality is judged by a person looking at a few examples.
• The examples are always the same, chosen for convenience.
• There is no written expected answer, only the feeling that “it makes sense”.
• There is no before-and-after number to compare.
• Nobody looks at results per group or category.

flowchart LR
  C["1. Prompt change"]:::tool --> V((("2. A person looks at 5 cases"))):::human
  V --> D["3. Deploy"]:::tool
  D --> P[("4. Thousands of real cases")]:::store
  P --> B(["5. Bias found months later"]):::io

1. A change is made.
2. A person checks a few cases they know.
3. It looks fine, and the change ships.
4. In production, cases are far more varied.
5. The problem surfaces late, with no data on when it started.

Check +20 XP on first tryFive familiar resumes came back 'ok' after the prompt change. Why is that not enough to ship?

Five is too few; 20 by eye would be enough

The cases are the same ones, with no written expected answer and no slice by group

You need a larger model to judge the five cases

Reasons

• It is fast and feels sufficient. Five cases take five minutes.
• The person knows the domain well. Their judgment is good, but five cases do not reveal patterns.
• Building an eval looks like research work. The team does not know where to start. Start with What an eval is.
• Distribution problems are invisible in a few cases. Bias, rare categories, and small regressions only show at volume.

Common trap: a golden dataset that is a favorites list

Two hundred cases picked “because they are representative” by the same person are still a vibe check with more rows. You need production sampling, rare categories on purpose, and group attributes (state, seniority, role type) so you can slice. If nobody labeled the state, nobody measures state bias.

In Mata v. Avianca (2023), lawyers filed a brief with case law invented by ChatGPT. The text “looked like” a legal filing. Nobody had a set of cases where the citation had to exist for real. A vibe check approves form; an eval demands fact.

Cost of ignoring

• Silent regressions. Drops of a few points pass unnoticed.
• Bias and discrimination. In decisions about people, unmeasured bias can be illegal, and privacy laws such as GDPR and Brazil’s LGPD give people the right to ask for review of automated decisions.
• Decisions by opinion. “It got better” versus “it got worse”, with no number.

A small eval already changes the conversation. Thirty real resumes, with an expected score and the state labeled, show what five favorites hide. That is not the final dataset. It is the end of shipping on a feeling.

Treatment

Refactoring	Choose it when
Add Eval Before Changing the Prompt	Always. A set of real cases with expected answers, graded automatically, with results per category and per relevant group. Targets: Eval Harness and Golden Dataset.

The short path, if you have half an hour this week:

1. Sample production, not the cases the team likes.
2. Write the expected answer (score, class, spans that must appear).
3. Run the old version and the new one on the same set.
4. Look at the overall number and the number by group. If one state dropped and the rest rose, the average is lying.

Manual reading stays: it finds failures the metric does not cover yet. Findings become new cases. Without the eval, reading decides the deploy. With the eval, it feeds the dataset.

Check +20 XP on first tryThe coordinator has 30 minutes this week to start a real eval. What is the best use of that time?

Sample real cases, write the expected answer, and run both prompt versions

Write a judge prompt and score live production

Ask the model to generate 200 synthetic resumes and use them as the dataset

Payoff

• A before-and-after number for every change.
• Results per category and per group, which surface bias and localized regressions.
• The domain expert’s experience becomes labels and rubrics, not a five-minute check.

When to ignore

• Early exploration. Looking at examples is great for understanding the problem. It just does not replace an eval when it is time to decide.
• As a complement to the eval. Reading outputs now and then remains useful for finding problems no metric captures. Alongside the eval, not instead of it.

Is it this smell?

Check +20 XP on first tryThe team has a 300-case eval and, on top of it, someone reads 20 outputs a week looking for new problems. Is this a Vibe-Check Eval?

Yes, there is manual reading

Yes, if the 20 outputs have no rubric

No

Check +20 XP on first tryThe eval has 200 cases, but the same person picked all of them as representative, and none has the candidate's home state labeled. What is missing?

Nothing, 200 cases is plenty

Real sampling and group attributes

Switch models and measure again by eye

Check +20 XP on first tryThe prompt changes in production with no test at all, not even a manual one. Which smell is more precise?

Vibe-Check Eval

Unversioned Prompt

Untested Prompt

Check +20 XP on first tryThe eval has 50 cases, all senior developers from São Paulo, and the change scored 50 out of 50. Production has interns from 12 states. Can you ship?

No, the dataset does not cover production

Yes, 100% accuracy is strong evidence

Yes, if the coordinator looks at 5 more junior resumes