Anti-Pattern: Evaluating by Vibes

“That looks better” is not a stupid way to evaluate a language model. It is the correct way for about a week. The problem is that nothing in the process ever announces that the week is over, so teams keep using it for a year and cannot explain why quality moves sideways.

Why it is rational at first

At the start of a project, reading the output is the highest-bandwidth evaluation available. You do not yet know what good looks like, so you cannot write a rubric; the failure modes have not been discovered, so you cannot enumerate them; the prompt is changing hourly, so any test suite you write is obsolete before it runs. Under those conditions, eyeballing twenty outputs teaches you more per minute than any instrument you could build, and building the instrument first would be premature.

This is worth conceding clearly, because the usual framing — that teams evaluate by vibes out of laziness — is wrong and does not persuade anyone. Vibes evaluation is a legitimate exploratory method that people continue using past its expiry date, which is a different failure and has a different fix.

The moment it stops being rational

Three conditions each independently end it, and typically the first one arrives within days of the feature being real.

• The change is smaller than your ability to see it. Early prompt changes move quality visibly. Later ones move it by a few percentage points, which is invisible in a sample of twenty and is exactly the size of change that matters once the obvious problems are gone. At this point looking is not a weak measurement; it is no measurement.
• More than one person is judging. Two people with different mental rubrics will disagree about whether a change was an improvement, and neither can show their work. Without a written rule, that argument is unresolvable and gets settled by seniority.
• You need to compare across time. Vibes have no record. When quality is questioned three months later, there is nothing to compare against — not even a note about what the outputs used to look like. Whether the model genuinely got worse becomes an unfalsifiable argument, which is why that argument recurs in every team that never wrote anything down.

Three errors it reliably produces

You fit to the examples you happen to look at

The outputs you inspect are not a random sample. They are the ones from the input you have open, which is usually the input that motivated the change. Fixing the prompt until that case is right, then shipping, is optimising on a sample of one — and the change frequently costs accuracy on cases nobody reopened. Without a held-out set there is no mechanism that would ever tell you this happened.

You mistake style for substance

Fluent, well-structured, confident output reads as better output. It is a robust and well documented bias in human judgement, and the same bias shows up when a model does the judging, which is why length and formatting effects are a standing subject in judge bias. A prompt change that makes answers longer and more assertive will feel like an improvement even when factual accuracy is unchanged or worse. Vibes evaluation has no way to separate the two, because the thing doing the evaluating is the thing being fooled.

Regressions are invisible until a user finds them

The characteristic vibes failure is a change that improves the case you were working on and breaks a case you fixed two months ago. There is nothing to catch it, so it ships, and it is discovered by a customer who remembers that it used to work. That report arrives without a version, without an input, and often weeks later — the most expensive possible form of feedback.

The smallest thing that beats it

The trap in fixing this is scope. Teams decide they need an eval framework, a labelled dataset and a judge model, none of which they have time to build, so they build nothing and keep using vibes. The honest minimum is much smaller than that, and the value curve is steepest at the very beginning.

cases/
  001-empty-input.json          { input, expect: { must_not: ["I'm sorry"] } }
  002-two-languages.json        { input, expect: { language: "input" } }
  003-missing-total.json        { input, expect: { fields: ["total"], total: null } }
  ...                           twenty of these, from real traffic

run:
  for each case:
     call the model once, at temperature 0
     apply the case's assertions      // string, regex, schema, or a number check
  report: passed, failed, and WHICH cases changed since the last run

Twenty cases with mechanical assertions is a weekend of work at most, runs in CI, and catches the regression class that vibes cannot see. Note what it does not attempt: it does not score quality, rank models or produce a number anyone reports upward. It answers one question — did this change break something that used to work — and that is the question vibes is worst at.

Three properties make the difference between a suite that survives and one that is abandoned after a month. Cases come from real inputs, not invented ones, so they keep representing the traffic. Assertions are about specific failures you have actually seen, named after them, so a failing case tells you what broke rather than that a score moved. And the run is fast and free enough to happen on every prompt change, because a suite that runs weekly is a suite nobody trusts. Growing this into something that can also rank models and gate releases is well-trodden ground — building an eval set for your own use case and regression testing for prompts both start from exactly this shape.

Keeping judgement where it belongs

The corrective for vibes evaluation is not to eliminate human judgement. It is to stop using it for the thing it is bad at — noticing a three-point change across a distribution — and keep using it for the things no assertion can do.

• Deciding what counts as a failure. A rubric is a human artefact. Every assertion in the suite above started as somebody looking at an output and deciding it was unacceptable.
• Reading the failures, not the successes. Once a suite exists, human attention belongs on the cases that failed and on a small sample of production traffic. Reading successful outputs is the least informative use of an expert’s time.
• Judging the things you cannot assert. Tone, whether an explanation is genuinely useful, whether a summary preserved the point. Structured human evaluation with a written rubric and more than one rater is still evaluation; it is vibes with a method, and the method is what makes it repeat.

The tell that a team has made the transition is a small one. Someone proposes a prompt change and the next sentence in the thread is about which cases it should be run against, rather than a screenshot of one output that looks good.