Eval Harness

Intent

Run the test cases automatically on every prompt, model, or code change, and compare against the previous version. It is for any AI system that changes often and must prove it did not get worse.

Problem

Joana’s data team has a natural-language-to-SQL system. They already have a dataset: 120 real questions with the correct query. The problem is that running the eval is a half-hour ritual: open a notebook, point it at the right prompt version, wait, copy the results table into a spreadsheet, and compare it with last week’s spreadsheet.

In practice, the eval runs when someone remembers. The week three people changed the prompt, nobody ran it. Accuracy dropped, and they had to bisect by hand to find which of the three changes caused it.

Having the cases is not enough. They must run on their own, the same way, every time, and the result must reach whoever made the change before the merge.

The question this pattern answers: how do you turn an eval dataset into an automatic check that runs on every change?

Solution

Build an eval harness with four fixed parts: load the cases, run the real system on each one, grade each output with the right rule, and compare the result with the stored baseline. Wire the eval harness into CI: every change to a prompt, model, tool, or pipeline code triggers the eval and attaches the report to the pull request.

The eval harness runs the system as it runs in production (same code, same versioned prompt), not a simplified copy. And it stores every run, so you can open a failing case and see exactly what the model returned.

Worked example

On Monday Joana’s PR changes only the JOIN example. CI starts the eval harness, loads evals/sql@19, runs the 120 cases through the real pipeline, and pastes this on the PR:

category           baseline   current   delta
aggregation        94.2%      94.0%     -0.2%
join               88.1%      71.3%     -16.8%  REGRESSION
date_filter        91.0%      90.5%     -0.5%

The merge blocks. The report lists the 11 join cases that went from right to wrong. Joana opens the first, sees the generated SQL, and notices the new example flipped the fact table. Without the eval harness, that drop would only show up in Friday’s spreadsheet, mixed with the other two changes of the week. Three prompt edits in one week is the usual failure mode: the drop has three parents and nobody wants to own the bisect. A gate on the PR makes the parent obvious, because only one of those PRs ever merged.

SQL cases are not graded by query text. Two different SQLs can return the same table. The eval harness runs the generated SQL and the expected SQL on the test database and compares the rows. Case sql-041:

{ "id": "sql-041", "input": "How many orders did we have in March by state?", "expected_rows": "fixtures/sql-041.csv", "category": "aggregation", "grader": "rule" }

If the model writes date_trunc instead of EXTRACT(MONTH ...) and the result matches, the case passes. If it writes an extra JOIN and duplicates rows, it fails. Comparing SQL strings would have rejected the first (good) and sometimes accepted the second (bad).

Structure

flowchart LR
  PR(["1. Pull request"]):::io --> L["2. Load cases"]:::tool
  DS[("Versioned dataset")]:::store --> L
  L --> R(["3. Run the real system"]):::llm
  R --> G{"4. Grade each output"}
  G --> A["5. Aggregate per category"]:::tool
  BL[("Baseline")]:::store --> C{"6. Compare"}
  A --> C
  C --> REP(["7. Report on the PR and block"]):::io

1. A change is proposed.
2. The eval harness loads the current dataset version.
3. Each case runs through the real pipeline, in parallel, with repeats when output varies.
4. Each output is graded by exact match, a rule in code, or a judge.
5. Scores become rates per category, plus cost and latency.
6. The result is compared with the main branch’s baseline.
7. The report goes to the PR, and regressions above the threshold block the merge.

Pseudocode

schema Case { id: string, input: any, expected: any, category: string, grader: "exact" | "rule" | "judge" }
schema Result { case_id: string, output: any, score: float, cost: float, latency: float }

function run_eval(system_version, dataset, repeats=3):
    results = []
    parallel {                                       # 3. in parallel
        for case in dataset, r in range(repeats):
            out = system(system_version).run(case.input)
            score = grade(case, out)                 # 4
            results.append(Result(case.id, out, score, out.cost, out.latency))
    }
    return aggregate(results, by="category")         # 5

function grade(case, out):
    if case.grader == "exact": return 1.0 if out == case.expected else 0.0
    if case.grader == "rule":  return RULES[case.category](out, case.expected)
    return llm(JUDGE_RUBRIC, case, out) -> float     # judge, when there is no alternative

function check(pr):
    current = run_eval(pr.version, load("evals/dataset@latest"))
    baseline = load_baseline("main")
    diff = compare(current, baseline)                # 6
    report(pr, diff)                                 # 7
    if any(cat.drop > THRESHOLD for cat in diff): fail(pr)

Real-world scenarios

• Natural language to SQL. Grading runs the generated and expected queries on the same test database and compares the results, not the SQL text.
• Support bot with human escalation. Real anonymized conversations become cases; the eval harness checks the route, the tools called, and whether side-effect actions asked for confirmation.
• Automated PR reviewer. Old PRs with known bugs are the cases; the eval harness measures how many bugs the reviewer finds and how many useless comments it makes.

Applicability

• More than one person changes the prompt, model, or pipeline.
• You already have (or will build) a dataset with expected answers.
• Changes happen weekly, or more often.
• A production regression is expensive: money, bad data, customers.

When not to use

• There is no dataset yet. Start there: Golden Dataset and Add Eval Before Changing the Prompt.
• The system changes once a quarter. A script run by hand before the change may be enough.
• You want to measure behavior with real users. That is a production experiment or a Shadow Deploy, which complement the eval harness.

How to implement

1. Define the case format: input, expected result, category, and grading type.
2. Write an entry point that runs the real system from a version (of prompt and code).
3. Implement grading, starting with exact match.
4. Run in parallel, with a concurrency limit so you do not blow through the rate limit.
5. Aggregate per category and compute cost and latency per case.
6. Store the main branch’s result as the baseline.
7. Wire it into CI, triggered by changes to prompts, model configuration, and pipeline code.
8. Post the report on the PR, with the cases whose result changed.

Which grader to use

Start with what code can decide on its own. A classifier route: exact string. SQL: execute and compare rows. An agent: the tool trajectory (it called buscar_pedido, it did not call reembolsar without approval). Free text, when there is no other path: LLM-as-judge with a rubric, measured against a person on a sample.

The judge is the last resort. It costs an extra call per case and can agree with a bad answer. If a code rule covers 80% of cases, leave the judge on the remaining 20%, not the other way around.

Quick eval and the regression threshold

A subset of 20 canary cases runs on every commit, in a few minutes. The full 120 run before merge and overnight. The canary catches the obvious miss; the full set catches the rare category.

The drop threshold (2 percentage points in the sample code) is not a taste. Run the same version twice, with no change, and see how much the result moves. If natural spread is 1 point, a threshold of 0 rejects innocent PRs. If it is 4 points, a threshold of 2 lets a regression through. The threshold sits a little above the measured noise.

Common mistakes

• A simplified copy. The eval harness calls the model directly, without retrieval, without the router, without the tool. The eval passes and production fails. The entry point is the same classify or run as production.
• A notebook as CI. It works once. The week of three PRs, nobody opens it.
• Comparing SQL text. It rejects equivalent queries and sometimes accepts the wrong one that “looks” like the expected query.
• A frozen dataset. The eval harness is only as good as the cases. Every production bug becomes a case, in the order of the Add Eval Before Changing the Prompt refactoring. Without that, you measure an old world. See also the Vibe-Check Eval smell.
• A zero threshold with high temperature. The same case passes and fails. Repeats and averaging come in before you tighten the threshold.

Off-the-shelf frameworks (Promptfoo, LangSmith, your provider’s eval runner) save the scaffolding. What you do not outsource: the dataset, the grader, and the trigger on the PR.

Variations

• Quick eval and full eval. A small subset runs on every commit; the full dataset runs before merge or nightly.
• Trajectory eval. For agents, beyond the final result, check that the right tools were called and that dangerous actions asked for approval.
• Online eval. The same grading runs on a sample of production traffic, without expected answers, using rules and a judge. It measures drift.
• Off-the-shelf tools. There are eval frameworks that provide the scaffolding (datasets, graders, reports). Use one when it saves code; what matters is the dataset and the grading, not the tool.

Cost, latency, and failure modes

The numbers below are illustrative. Suppose 120 cases, 3 repeats, no judge, concurrency 8.

The cheap canary does not replace the full set. It only saves you from waiting on the long job to learn that the prompt no longer parses.

How to evaluate

The eval harness needs evaluating too:

• Judge agreement with a person, on a sample, when there is a judge.
• Stability: run the same version twice and see how much the result varies. That sets the regression threshold.
• Coverage: how many production bugs from the last quarter have a case in the dataset.

{ "id": "sql-041", "input": "How many orders did we have in March by state?", "expected_rows": "fixtures/sql-041.csv", "category": "aggregation", "grader": "rule" }

If the nightly eval drops with no code change, the suspects are the provider (an updated model) or the test database. That alert is as useful as the PR one: the baseline ages too.

In production

• Eval traces kept apart from production traces, tagged with the version tested.
• History of per-category results over time, to see the trend.
• An alert when the nightly eval drops with no code change: a sign of a change in the provider’s model or in the data.
• A routine to turn production incidents into new cases.

Pros and cons

Code

Both tabs implement a minimal eval harness with JSONL cases, exact-match grading, parallel execution, and a baseline comparison.

import asyncio
import json
from collections import defaultdict
from statistics import mean

from my_app.pipeline import classify  # 3. the real system, not a copy

THRESHOLD = 0.02  # maximum tolerated drop per category


async def run_case(case: dict, repeats: int, sem: asyncio.Semaphore) -> list[float]:
    async with sem:
        outs = await asyncio.gather(*(classify(case["input"]) for _ in range(repeats)))
    return [1.0 if o == case["expected"] else 0.0 for o in outs]  # 4. exact match


async def run_eval(path: str, repeats: int = 3) -> dict[str, float]:
    cases = [json.loads(line) for line in open(path, encoding="utf-8")]
    sem = asyncio.Semaphore(8)  # respect the rate limit
    scores = await asyncio.gather(*(run_case(c, repeats, sem) for c in cases))
    by_cat = defaultdict(list)
    for case, s in zip(cases, scores):
        by_cat[case["category"]].extend(s)
    return {cat: mean(v) for cat, v in by_cat.items()}  # 5


def main() -> None:
    current = asyncio.run(run_eval("evals/tickets.jsonl"))
    baseline = json.load(open("evals/baseline.json"))  # produced on the main branch
    failed = False
    for cat, base in sorted(baseline.items()):
        now = current.get(cat, 0.0)
        flag = "REGRESSION" if base - now > THRESHOLD else ""
        failed |= bool(flag)
        print(f"{cat:20} {base:6.1%} {now:6.1%} {now - base:+6.1%} {flag}")  # 7
    raise SystemExit(1 if failed else 0)


if __name__ == "__main__":
    main()

import { readFileSync } from "node:fs";
import { classify } from "../src/pipeline"; // 3. the real system

type Case = { input: string; expected: string; category: string };
const THRESHOLD = 0.02;

async function pool<T, R>(items: T[], limit: number, fn: (t: T) => Promise<R>): Promise<R[]> {
  const out: R[] = new Array(items.length);
  let next = 0;
  const workers = Array.from({ length: limit }, async () => {
    while (next < items.length) {
      const i = next++;
      out[i] = await fn(items[i]);
    }
  });
  await Promise.all(workers);
  return out;
}

export async function runEval(path: string, repeats = 3): Promise<Record<string, number>> {
  const cases: Case[] = readFileSync(path, "utf8").trim().split("\n").map((l) => JSON.parse(l));
  const scores = await pool(cases, 8, async (c) => {
    const outs = await Promise.all(Array.from({ length: repeats }, () => classify(c.input)));
    return outs.map((o) => (o === c.expected ? 1 : 0)); // 4
  });
  const byCat: Record<string, number[]> = {};
  cases.forEach((c, i) => (byCat[c.category] ??= []).push(...scores[i]));
  return Object.fromEntries(
    Object.entries(byCat).map(([k, v]) => [k, v.reduce((a, b) => a + b, 0) / v.length]),
  ); // 5
}

const current = await runEval("evals/tickets.jsonl");
const baseline: Record<string, number> = JSON.parse(readFileSync("evals/baseline.json", "utf8"));
const regressions = Object.entries(baseline).filter(([cat, base]) => base - (current[cat] ?? 0) > THRESHOLD);
console.table(Object.keys(baseline).map((cat) => ({ cat, base: baseline[cat], now: current[cat] })));
process.exit(regressions.length ? 1 : 0); // 6 and 7

Check your understanding