Eval Set Builder

Turn a text box of cases into an eval file, a grader and a runner script you fill your own endpoint into, with the cost of one full run worked out first.

CasesOne case per block, separated by a line of ---. A line of => splits the input from what you expect. An optional first line of #tags labels the case.

System message used for every case

Grader

Runs per caseMore than one, or you are measuring luck.

runs

Case sensitiveTrim whitespace before comparing

Input rateYour own rate. Nothing here ships a price.

$ / 1M in

Output rate

$ / 1M out

Expected output tokens per caseHow long you expect each answer to be.

tokens

Characters per tokenCharacters per token. An estimate — nothing here runs a tokeniser.

chars

Cost of one full eval run

$0.02

4 case(s) × 3 run(s) = 12 model calls. At your rates, with input tokens estimated at 4 characters each and output tokens taken from the field above.

Cases: 4
Model calls per run: 12
Grader: Output contains the expected text
Estimated input tokens per pass: ≈ 121
Estimated output tokens per pass: ≈ 320
Input cost per run: $0.0011
Output cost per run: $0.01
Cost per case: $0.0039
Cost of running it daily for a month: $0.46
Cases tagged #refund: 2 (50.0%)
Cases tagged #delivery: 1 (25.0%)
Cases tagged #edge: 1 (25.0%)
Cases tagged #unknown: 1 (25.0%)
Untagged cases: 0

eval.jsonlRead-only. Select all and copy.

run_eval.py — fill in the three constants at the topRead-only. Select all and copy.

"""Run the eval set. Fill in BASE_URL, MODEL and the env var name.

Nothing about this script came from a live model — it is your cases, your
endpoint and your grader, written out. Read it before you run it.
"""
import json, os, re, urllib.request

BASE_URL = "https://your-endpoint.example/v1/chat/completions"
MODEL    = "your-model-id"
API_KEY  = os.environ["YOUR_API_KEY_ENV_VAR"]
REPEATS  = 3
SYSTEM   = """You are the support assistant for an online shop. Answer in one or two sentences."""

def call(prompt: str) -> str:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "system", "content": SYSTEM},
                     {"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(
        BASE_URL, data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)["choices"][0]["message"]["content"]

def grade(kind: str, got: str, want: str, opts: dict) -> bool:
    if opts.get("trim"):
        got, want = got.strip(), want.strip()
    if not opts.get("case_sensitive", True) and kind != "regex":
        got, want = got.lower(), want.lower()
    if kind == "exact":    return got == want
    if kind == "contains": return want in got
    if kind == "regex":    return re.search(want, got) is not None
    if kind == "numeric":
        nums = re.findall(r"-?\d+(?:\.\d+)?", got)
        return bool(nums) and abs(float(nums[0]) - float(want)) <= opts["tolerance"]
    if kind == "json":
        try:    return str(json.loads(got)[opts["field"]]) == want
        except Exception: return False
    raise SystemExit(f"grader {kind!r} needs a model call — wire it up yourself")

cases = [json.loads(l) for l in open("eval.jsonl") if l.strip()]
passed = total = 0
for c in cases:
    for _ in range(REPEATS):
        got = call(c["input"])
        ok = grade(c["grade"]["type"], got, c["expected"], c["grade"])
        passed += ok; total += 1
        if not ok:
            print(f"FAIL {c['id']} {c.get('tags')}\n  want: {c['expected']}\n  got : {got}")
print(f"{passed}/{total} passed ({passed / total:.1%})" if total else "no cases")

The same set as YAML, for a harness that takes oneRead-only. Select all and copy.

What this assumes: the cost figures use YOUR rates and an estimated token count of characters ÷ 4 for the inputs plus the output length you typed — no price list is shipped with this page, and none of these numbers came from a model. The system message is counted once per case, because it is sent on every call. The judge grader is the one grader this page cannot run for you: it needs a second model call, so the script raises rather than pretending. Everything on this page runs in your browser. Nothing you paste or open is uploaded, logged or sent anywhere.

An eval set is worth having the moment you change anything — a model, a prompt, a retrieval step — because without one the only test is whether the three examples you happen to remember still look right. What stops most teams is not the concept but the hour of formatting, which is what this page removes: cases in a text box come out as a file with a grader attached and a script that runs it.

Run each case more than once

A single run of a set at any temperature above zero measures the model and the die roll together. Three runs per case is a floor, not a target, and the cost table above exists so you can see what the floor costs before deciding it is too expensive — usually it is a few cents, which is a poor reason to ship a regression.

Choose the grader that can actually fail

Exact match on a sentence of prose fails on punctuation and tells you nothing. Contains passes a refusal that happens to quote your expected phrase. An unanchored regex passes on any output long enough to contain the pattern by accident. Each of those is flagged above where this page can see it happening. The tags are the other half: a score of 84% means nothing until you can see that it is 100% on the easy cases and 20% on the ones tagged #edge, which is the number that actually predicts the support tickets.

Eval Set Builder

Run each case more than once

Choose the grader that can actually fail

The reasoning behind this

Related tools