How to Evaluate an LLM for Your Own Use Case

Every team that ships something on a language model eventually discovers that the leaderboard was not about them. The replacement is small, private and boring: fifty examples of your actual work, with a grader you wrote before you looked at any output.

Decide the unit before anything else

The first question is not which model, and not which metric. It is: what is one example? Get this wrong and everything downstream is measuring something you do not care about.

A support-reply system has an obvious unit — one incoming ticket, one drafted reply. A retrieval-augmented assistant does not: you can evaluate one question against the final answer, or you can evaluate the retrieval step and the generation step separately. Both are legitimate and they answer different questions. End-to-end tells you whether the product works; component-wise tells you which half to fix when it does not. An agent that takes twelve tool calls to complete a task has a third option again — score the trajectory, or score only the terminal state.

Pick one. Write it down as a sentence: “one example is one customer ticket plus the retrieved documents, and the output under test is the drafted reply.” If you cannot write that sentence, you are not ready to collect examples, and the set you collect will be a mixture of three different evals that you will have to split later anyway.

Why fifty, and what fifty buys you

Fifty is the number at which the set stops being anecdote and starts being a fixture, and it is small enough that one person can build it in an afternoon and read every single output. That last property matters more than it sounds — an eval set nobody has read end to end is an eval set with mislabelled items in it.

What fifty does not buy you is precision. Take a system that passes 40 of 50 items. The Wilson score interval at 95% confidence around that 80% runs from roughly 67% to 89% — about eleven points either side. Any change smaller than that is invisible at this sample size, no matter how convincing the spreadsheet looks.

So a fifty-item set is calibrated for one job: catching the large, obvious, structural failure. Does this model follow the output format? Does it refuse things it should not refuse? Does swapping to a model a tenth of the price destroy the product or barely dent it? Those are twenty-point questions and fifty items answers them cleanly. Deciding between two candidates that are three points apart needs a paired design and several hundred items, and pretending otherwise is how teams end up shipping regressions they were sure were improvements.

Choosing the fifty: stratify, never sample

The instinct is to take a random fifty from production traffic. Do not. Random sampling from a workload where 85% of requests are trivial gives you an eval set that is 85% trivial, which means it saturates immediately and every model you try scores in the nineties. A saturated eval has no resolution left; it cannot tell any two candidates apart.

Allocate deliberately across strata instead. A workable default split:

Record the stratum on every item. Aggregate score across strata is nearly meaningless when the mix is chosen rather than sampled — what you want out of a run is five numbers, and the interesting result is almost always that a model won three strata and lost two.

Write the grader before you see any output

This is the discipline that separates an eval from a vibe. For each item, write down what a pass looks like before any candidate model has produced anything. Once you have read a plausible-sounding output, your notion of “correct” will quietly deform to accommodate it, and you will not notice it happening.

Graders come in a rough hierarchy, and you should always take the cheapest one that actually works:

• Programmatic. Does it parse as JSON against the schema? Does the SQL execute and return the expected row count? Is the cited document id in the retrieved set? These are free, instantaneous, perfectly reproducible, and cover far more of a real eval than people expect. Exhaust this category first.
• Reference-based. A gold answer plus a comparison rule — exact match, numeric tolerance, set overlap. Cheap and trustworthy where a single right answer exists.
• Model-graded. A judge model with a rubric. Necessary for open-ended text, and the point at which you have added a second stochastic system that needs its own validation.
• Human. Slow, expensive, and still the ground truth everything above is approximating. Reserve it for calibrating the other three rather than for routine runs.

A good eval file is mostly rows one and two with a minority of row three. If your set is entirely model-graded, the first thing to ask is whether the task really has no checkable structure, or whether nobody looked for it.

What the first run is actually for

The first run is not for ranking models. It is for finding the bugs in your eval. Expect a meaningful fraction of the failures in run one to be your own: an item whose gold answer is wrong, a grader regex that rejects a correct answer over a trailing full stop, an item that two reasonable people would grade differently.

Read every failure by hand on the first run — all of them, including the ones you expected. Fix the eval, not the prompt. Only when a run produces failures that are all genuinely the system’s fault do you have an instrument, and only then does the number mean anything.

How the set grows

The set grows from production incidents and from nowhere else. Every time something goes wrong in a way a user noticed, that input becomes an item, with the correct output written down at the moment everyone still remembers what correct meant. This one habit is what makes an eval set appreciate over time rather than rot; the curation mechanics are their own page.

Resist the urge to grow it with synthetic variations. A hundred paraphrases of the same question look like a larger eval and are statistically almost one item — they fail and pass together, so they add cost without adding power. Growth should mean new failure modes, not new sentences.