Test Generation With LLMs: Beyond Coverage Theatre

Ask for tests and you will get tests. Coverage will go up. Almost none of it will be information, and the reason is that coverage measures execution, which is the one thing a model can produce without understanding anything.

What coverage actually rewards

Line coverage counts lines executed during the test run. It says nothing about assertions. A test that calls a function inside a try and asserts nothing at all covers every line it touches, and a suite of those reports 90%.

When you ask a model for “tests with good coverage”, you have named a proxy, and a system optimising a proxy will satisfy the proxy. It is the same dynamic as any metric-driven target, except that the generator is fast enough to saturate it in one call. The fix is not a better prompt about coverage; it is a different objective.

The tautological test

The dominant failure is a test that asserts its own setup. Every mock is configured to return the value the assertion then checks:

// Proves that the mock was configured. Nothing else.
test("applies the discount", () => {
  const pricing = { discountFor: vi.fn().mockReturnValue(0.1) };
  const total = priceOrder({ subtotal: 100 }, pricing);
  expect(pricing.discountFor).toHaveBeenCalled();
  expect(total).toBe(90);            // 100 * (1 - 0.1), recomputed by hand
});

// Would survive priceOrder being replaced by: () => 90

The last comment is the test for the test. If a constant function passes it, it is not a test. That check — imagine the dumbest implementation that passes — catches the class immediately and by hand.

The close relative is the test that asserts the implementation rather than the behaviour: expect(repo.save).toHaveBeenCalledTimes(1) fails the moment you add a legitimate retry, which teaches the team that tests are noise and should be deleted when they go red.

Mutation score is the real objective

Mutation testing gives you the objective coverage cannot: it perturbs your source — flips a comparison, replaces a return with a constant, removes a call — and reports how many of those mutants your suite caught. A tautological test kills nothing. A test that asserts real behaviour kills a mutant per branch it constrains.

# Python
pip install mutmut && mutmut run --paths-to-mutate src/billing/
mutmut results          # survived mutants are the gaps your suite does not see

# JS / TS
npx stryker run --mutate 'src/billing/**/*.ts'

# Java: PIT   ·   Go: go-mutesting   ·   Rust: cargo-mutants

This is slow — it runs your suite once per mutant — so scope it to the module you just generated tests for rather than the repository. Run it once on a generated suite and the theatre becomes visible immediately: coverage 94%, mutants killed 20%. That gap is the number worth putting in a PR description.

The loop that follows is genuinely good use of a model: feed it a surviving mutant — “replacing discount > 0 with discount >= 0 does not fail any test” — and ask for a case that distinguishes them. That is a well-posed problem with a checkable answer, which is the shape models are reliable on.

Two prompt shapes that work

Enumerate cases, do not write code

Ask for the equivalence classes and boundaries of a signature, as a list, before any test code exists. Models are good at this — it is recall over a well-documented body of practice — and you keep the judgement:

Signature:  parseDuration(s: string): number   // returns milliseconds
Contract:   accepts <int><unit> where unit in ms|s|m|h; throws on anything else

List the equivalence classes and boundary values. For each, give the input
and the expected output or exception. Do not write test code.

  "0ms" -> 0                   lower boundary
  "1h"  -> 3_600_000           unit conversion, largest unit
  "999999999h"                 overflow past Number.MAX_SAFE_INTEGER?
  "1.5s"                       non-integer: contract says throw, verify
  " 1s" / "1 s" / "1S"         whitespace and case: unspecified — decide now
  "" / "s" / "1" / "1d"        malformed

The value is in the fourth and fifth rows: cases where the contract is undecided. A model enumerating inputs surfaces underspecification, which is worth more than the tests.

State the property, not the example

Where an invariant exists, property-based tests are a better fit for generation than examples, because the invariant is written once and the framework produces the cases and the shrinking:

from hypothesis import given, strategies as st

@given(st.lists(st.integers(min_value=0, max_value=10**6)))
def test_split_then_merge_is_identity(amounts):
    # invariant: splitting a payment and merging it back conserves cents
    assert sum(split_evenly(sum(amounts), len(amounts) or 1)) == sum(amounts)

Rounding bugs in money splitting are found by this in seconds and by example-based tests approximately never. Ask the model for candidate invariants — round-trip, idempotence, conservation, monotonicity, commutativity — and write the ones that are actually true of your domain.

The flakiness generated suites add

A generated suite arrives all at once, which means its flakiness arrives all at once too, and a suite that fails once a week for no reason destroys the value of the whole thing — people stop reading red builds. Three classes account for nearly all of it, and all three are predictable from what the model could not see.

• Real time. datetime.now() in the assertion, a sleep chosen to be “long enough”, a test that fails on the last day of a month or during a DST transition. The model has no way to know your clock is not frozen, so it writes against the real one. Freeze it explicitly — freezegun, vi.useFakeTimers() — and say so in the prompt.
• Shared state between tests. A module-level fixture mutated in place, a database row created by test A that test B counts, an environment variable set and not restored. These pass in file order and fail when the order changes, which is why the ordering plugin is worth running deliberately rather than disabling.
• Unordered results asserted as ordered. assert rows == [a, b, c] against a query with no ORDER BY, or an assertion on the serialisation of a set. Passes on your machine and on nine of ten CI runs.

Catch all three before merging, in about a minute, by running the new tests under conditions the author did not choose:

# shuffle the order, five times, with a different seed each run
pytest tests/test_invoice.py -p randomly --count=5

# run them in isolation as well as together: a test that only passes
# alongside its neighbours is depending on their side effects
pytest tests/test_invoice.py --forked

# JS: vitest --sequence.shuffle --repeat=5
# and pin the clock so a test cannot pass because of what day it is
TZ=Pacific/Chatham pytest tests/          # a +12:45 offset finds date bugs

That last line is not a joke. Running the suite in a timezone with a non-integer offset, on a machine whose locale is not English, finds a category of date and formatting assumption that no amount of reading finds — and generated code makes those assumptions more often than hand-written code, because the corpus it learned from is overwhelmingly UTC and en-US.

What not to generate

• Snapshot tests. A generated snapshot asserts that today’s output equals today’s output, and the standard response to a red snapshot is to update it. Coverage theatre with a worse failure mode.
• Tests for private functions. They pin the implementation and block the refactors the tests were supposed to enable.
• Tests for generated code. If a protobuf stub is wrong, the fix is in the generator.
• A test written after the bug fix, by the same session that wrote the fix. It will pass. Whether it would have failed before is the only question, and running it against the parent commit is how you answer it.

Where generation genuinely earns its place is the unglamorous middle: fixtures, table-driven cases once the table exists, and filling in the combinations of a matrix you specified. Those are transcription tasks, and transcription is what this is good at.