Eval Frameworks Compared
4 min read · updated August 3, 2026
Every eval tool demos well, because the demo is a small dataset and a pass rate. What separates them is what happens in month nine, when you want last quarter’s numbers, a judge definition that outlives the vendor, and a CI job that does not phone anywhere.
Four families, not one market
Tools that get compared to each other frequently solve different problems. Sorting them first makes the comparison tractable.
| Family | Description |
|---|---|
| academic harnesses | EleutherAI's lm-evaluation-harness, Stanford CRFM's HELM. Built to run standard benchmarks reproducibly across models. Excellent at exactly that, and awkward for a bespoke task with a custom grader. Reach for these when you want a comparable public number, not when you want to test your product. |
| developer test runners | promptfoo, OpenAI Evals, UK AISI's Inspect, DeepEval. Config or code, run locally, results as files, designed to sit in CI next to your other tests. The closest analogue to a unit-test framework. |
| hosted platforms | LangSmith, Braintrust, Arize Phoenix and similar. Datasets, runs, traces, judges and a UI in one product, with production tracing joined to offline evals. The most capability per hour invested, and the most to unpick later. |
| domain libraries | Ragas and its relatives for retrieval-augmented systems. Ship opinionated metrics -- faithfulness, context precision, answer relevance -- with definitions already implemented. Useful as a starting vocabulary; the definitions are theirs, so read them before you report them. |
Because this page names live products, treat it as a map rather than a spec sheet — capabilities and licences here change on a scale of months, and the axes below are what stay stable.
The four axes of lock-in
1. Where your dataset lives
The best case is a plain file in your repository — JSONL or YAML, diffable, reviewable in a pull request, and greppable. The worst is rows in someone’s database, editable only through a UI, exported through an API that gives you their schema. Ask specifically: can a reviewer see a dataset change in a code review? If not, your eval set has left version control, and everything that makes a golden set maintainable has left with it.
2. Where your results live
This is the axis that hurts most and gets checked least. Eval results are an accumulating asset: a year of per-item scores is what lets you say when a regression started. If they live only in a hosted service, your history is a subscription. Insist on per-item rows — item id, sample index, score, per-grader marks, tokens, latency, model version — exportable in bulk. An export that gives you aggregates only is not an export.
3. Whether the judge is portable
A judge is a prompt, a model, a parse rule and a version. If the tool expresses it as a named built-in metric whose prompt you cannot read, you cannot reproduce your scores anywhere else, and you cannot tell whether a change in the tool changed your metric. A framework whose judges are files you can read is one you can leave.
4. Whether it reaches into your application
The sharpest distinction in the whole market. Some tools evaluate artefacts — you give them inputs and outputs. Others require their SDK inside your application to trace calls, which is genuinely powerful for joining production traces to offline evals, and means the vendor is now in your runtime dependency graph and your request path. That may be a fine trade. It should be a deliberate one, and it is not reversible in an afternoon.
The exit test
Before committing, run this. It takes a morning and it is the only evaluation of an eval tool that predicts the thing you will care about.
- Export the dataset. Is it a file you could feed to a fifty-line script? Time how long the conversion takes.
- Export a run at per-item granularity. If you cannot, stop — the history you are about to accumulate is not yours.
- Print every judge prompt the tool would use on your behalf. If any is opaque, your metric is undefined.
- Run the whole suite offline, in CI, with the vendor unreachable. If that is impossible, your release pipeline now has their uptime in it.
- Reproduce one number outside the tool, by hand, from the exported artefacts. If the number does not reproduce, you have not been measuring what you thought.
Choosing by situation
- Comparing models on public benchmarks. An academic harness. Do not build this; reproducibility across models is exactly what they were built for and exactly what is fiddly to get right.
- A product team wanting a CI gate. A developer test runner, or your own harness. Files in the repository, results as artefacts, no network dependency at gate time.
- Several teams, non-engineers writing test cases, production tracing needed. A hosted platform earns its keep. Go in with the export path tested, not assumed.
- A retrieval-augmented system, early. A domain library to borrow the metric vocabulary, then reimplement the two or three metrics you actually use so you own their definitions.
When to write your own
More often than the market implies. The core of an eval harness — load cases, call a model with k samples, apply graders, write per-item rows, aggregate with an interval, exit non-zero — is a couple of hundred lines, and it is written out in full on the next page. What frameworks add on top is dataset management, a UI, tracing, concurrency and integrations. Those are real, and they are worth paying for when you need them and worth nothing when you do not.
There is one hazard that cuts across all four families and deserves naming: metric definition drift. A tool that ships a metric called “faithfulness” will improve its implementation over time, which is good engineering and quietly makes your historical scores incomparable across the upgrade. Pin the tool version in your lock file, treat a bump as a change that re-runs the baseline, and record the tool version in every results row. This is the same discipline you would apply to a judge model, for exactly the same reason — the metric is code, and code that changes underneath a time series invalidates the series.
The hybrid most teams land on: own the dataset and the graders as files in the repository, use a tool for the parts that are genuinely annoying — parallelism, caching, dashboards, tracing — and keep every per-item result in your own storage as well as theirs. That keeps the switching cost at one adapter rather than a year of history.