Hypothesis Generation: Can a Model Have an Idea?

The question is asked as though it had one answer. It has at least four, because “novel” is doing four incompatible jobs in the sentence, and separating them turns an argument into a set of answerable questions.

Four things novel can mean

Almost every disagreement about this subject is two people using different rows. A demonstration that a model produces hypotheses experts rate as non-obvious says nothing about the fourth row, and a single confirmed discovery says nothing about the rate.

Mechanism one: recombination

The first plausible mechanism does not require the model to be creative in any interesting sense. It requires it to have read more than any individual can.

This is an old idea with a canonical example. In the 1980s Don Swanson noticed that one body of literature reported that dietary fish oil reduced blood viscosity, and a separate body reported that elevated blood viscosity was a feature of Raynaud’s syndrome — and the two literatures did not cite each other. Nobody had stated the implied connection because nobody read both. The hypothesis fell out of the structure of the literature rather than from any new data, and the approach became known as literature-based discovery.

Language models are unusually well placed for this. The bottleneck Swanson identified is human reading capacity, and that is precisely what a model does not share. Cross-field analogy — this pathway resembles that control system, this ecological model resembles that epidemiological one — is a recombination task, and recombination over a very large corpus is something a next-token model does naturally.

Two honest limits. The output is a hypothesis and nothing more: the connection being stateable does not make it true, and Swanson’s proposals still needed testing. And the same generative process produces connections that are merely superficial, at a rate nobody has characterised, so the expensive part — deciding which of the proposals is worth an experiment — is untouched.

Mechanism two: search against a verifier

The second mechanism is narrower, much stronger, and is the only place novelty has been demonstrated rather than argued about.

Where a proposal can be checked automatically, a model can be run as a proposer inside a loop and the loop can afford to reject almost everything it produces. In combinatorics and algorithm design a candidate construction can be expressed as a program whose quality is computed by running it, and evolutionary loops built this way have produced constructions that beat the previous human-designed best on open problems. In a proof assistant the checker is the kernel, and the same pattern applies — the subject of why formal verification is the interesting part.

What makes these different from the recombination case is that novelty is not a matter of expert opinion. The new object is better by a computable measure than everything previously known, and anyone can re-run the check. That is the fourth row of the table, settled, and it is available exactly where a cheap automatic verifier exists — which is the same condition that governs the rest of this cluster.

The study that would settle it

For the general case, the design is not mysterious. It is just expensive, which is why so little of it exists.

• Generate both arms under matched conditions. Human and model hypotheses on the same problems, with the humans given comparable time, information and incentives. Comparing a model against researchers doing it as an afterthought measures the researchers’ motivation.
• Blind the raters, and check the blinding. Style gives generated text away, which unblinds the study silently. Ask raters to guess the source and report how often they were right; if they can tell, the ratings measure something other than quality.
• Pre-register the criteria. Novelty, plausibility and interest are three different scales and they move in opposite directions — the most novel ideas are usually the least plausible. Collapsing them into one score hides the trade-off that is the whole question.
• Test them prospectively. The only decisive endpoint is what fraction of hypotheses from each arm survives experimental testing. This is what almost no study does, because it means running an experimental programme, and it is the difference between measuring what experts find interesting and measuring what is true.
• Report cost per validated hypothesis. A system that generates a thousand ideas will contain a good one; the question is what it costs to find it, including the expert time spent filtering. The best idea in a large set is not a comparable unit to the best idea in a small one.

Using it now, without settling it

None of the above stops the tool being useful today, as long as it is used for divergence rather than for selection. Generate widely — twenty framings of a problem, ten mechanisms that could explain an observation, the objections a hostile reviewer would raise — and then apply the filter yourself, because the filter is where the domain expertise lives and it is the part that has not been demonstrated.

The most reliable value is not the idea you take. It is the idea you reject for a reason you had not previously articulated. That is genuinely useful, it costs nothing, and it requires no position on whether a model can have an idea.