Headlines, Titles and Subject Lines

Models are good at producing forty headlines and bad at telling you which one to use. Split the job: generate wide with a model, select with evidence from your own archive, and if your traffic is below about six thousand impressions per variant, accept that you are exercising judgement rather than running an experiment.

Generation and selection are different jobs

Generation rewards volume and variety. Selection rewards calibration against a specific audience. A single prompt that says “write the best headline for this article” collapses both into one step and returns the most typical headline for that subject — which is the one your competitors already used.

The productive shape is a funnel: thirty to forty candidates spread across deliberately different strategies, cut to five by rule, then ranked by a selector grounded in what has actually worked for you. The model is doing two different things at those two stages and needs two different prompts.

The generation prompt

Variety does not come from asking for variety. It comes from naming the strategies and demanding a fixed quota of each, which stops the output collapsing into forty rewordings of the first idea.

Write 40 candidate headlines for the article below. Produce exactly
five in each of these eight strategies, labelled:

1. FLAT      — states what the article establishes, no rhetoric.
2. NUMBER    — leads with the most specific figure in the piece.
3. VERB      — the reader does something: how to X, X without Y.
4. TENSION   — names a conflict or a cost the reader already feels.
5. WRONG     — asserts that a common belief is mistaken (only if the
               article actually shows this — otherwise return fewer).
6. NOUN      — names the thing precisely, for people searching for it.
7. QUESTION  — the literal question the reader typed. No rhetorical
               questions.
8. LONG      — 12 to 16 words, specific enough to need no subheading.

Constraints:
- Every headline must be supported by the article. Do not promise a
  number, a comparison or a result the article does not contain.
- No colon-splice headlines ("X: Why Y Matters").
- No superlatives that are not measured in the piece.
- Under 65 characters except in LONG.

ARTICLE
...

The first constraint is the one to enforce hardest. Headline generation is where a model most readily promises what the piece does not deliver, because a promise is a high-probability headline shape. Any candidate you cannot point at a paragraph for is deleted before selection, not during it.

Why you probably cannot A/B test this

Everybody is told to test headlines. Here is what a test costs, derived rather than asserted, using the standard approximation for comparing two proportions.

For 80 per cent power at a 5 per cent two-sided significance level, the sample needed per arm is approximately

n per arm ≈ 16 × σ² / δ²

For a click-through rate, σ² ≈ p(1 − p), where p is the baseline rate
and δ is the absolute difference you want to be able to detect.

Baseline p = 0.04   (a 4% click-through rate)
Target    δ = 0.01   (detect a lift to 5%, a 25% relative improvement)

σ² = 0.04 × 0.96 = 0.0384

n ≈ 16 × 0.0384 / 0.0001
  = 0.6144 / 0.0001
  = 6,144 impressions per variant

Two variants: 12,288 impressions for ONE test.

Every assumption there is labelled and every step is arithmetic you can redo with your own numbers. The consequences follow directly. Detecting a smaller lift is quadratically more expensive: halving δ to 0.005 multiplies the requirement by four, to about 24,600 per variant. Testing four headlines instead of two both splits your traffic and multiplies your comparisons. And a newsletter with 5,000 subscribers cannot run a single adequately powered subject-line test on one send, however the tooling presents it.

What people actually do is run an underpowered test, see a difference, and act on it. At these sample sizes the observed difference is mostly noise, and the practice of always shipping the apparent winner locks in whichever variant got lucky. That is worse than judgement, because it feels like evidence.

Building a selection set from your archive

If you cannot test forward, use backwards evidence. You have one asset nobody else has: every headline you have published and how it did.

1. Export title, date, and one outcome metric per piece. Impressions and clicks for search, opens for email, whatever your platform records.
2. Normalise for exposure, because raw counts mostly measure how long the piece has been live and how big the list was that week. Use rate, not total.
3. Normalise for subject. A piece about pricing will out-click a piece about architecture regardless of its headline, so compare within topic clusters rather than across your whole archive.
4. Form pairs, not scores. Take pieces from the same cluster published within a few months of each other where one clearly beat the other, and record the winner. Thirty to sixty pairs is enough to be useful.
5. Hold back a quarter of the pairs. You need them for the validation step below and you must not look at them first.

Pairs rather than scores because a language model asked to give an absolute score returns a number clustered around seven out of ten regardless of input, and because a pair cancels out the topic effect that a score cannot.

The pairwise selection prompt

You are predicting which of two headlines performed better with one
specific audience. Below are past pairs from that audience where the
outcome is known. Learn the audience's preferences from them.

PAST PAIRS (winner first)
1. WON:  Why our search rewrite made results worse for a month
   LOST: Lessons learned from rebuilding our search stack
2. WON:  The 400ms we could not explain
   LOST: Debugging latency in a distributed system
... (20 to 40 more)

Now the new pair. Answer with:
- winner: A or B
- confidence: high / low
- reason: which learned preference decided it, in one clause

A: <headline A>
B: <headline B>

Two properties make this worth doing. The exemplars are outcomes rather than opinions, so the model is generalising from your audience instead of from headline-writing advice on the open web. And the reason field is often more valuable than the verdict — after twenty comparisons you have a written description of what your readers respond to, which is a house style you did not have before. This is ordinary few-shot prompting with the examples chosen for one purpose.

Run every candidate against every other in a round robin and rank by wins. Twenty candidates is 190 comparisons, which is cheap; if it is not cheap, cut to the eight you would actually consider first. The pairwise construction is the same one used whenever a model is used as a judge, and it inherits the same properties: pairs are far better calibrated than absolute scores, and position bias is real enough that you should run each pair in both orders when the result matters.

Checking the selector beats a coin

A selector that agrees with your instincts is not evidence. Run it on the held-back pairs, where you know the answer and it does not, and count.

20 held-out pairs. The selector picks the true winner k times.
Null hypothesis: it is guessing (p = 0.5).

P(k ≥ 15 | n = 20, p = 0.5) = 21,700 / 1,048,576 = 0.021
P(k ≥ 14 | n = 20, p = 0.5) = 60,460 / 1,048,576 = 0.058

So: 15 or more out of 20 beats chance at the 5% level.
    14 out of 20 does not, and 14 is what you will get.

That is exact binomial arithmetic, not an approximation. It sets a deliberately unflattering bar, and most selectors will fail it on the first attempt. A failure is useful: it means you should stop treating the ranking as a prediction and start treating it as a shortlist generator, which is still worth having.

If it passes, keep it and re-validate when your audience changes. Recheck at least annually, and immediately after any change in where your traffic comes from — a selector trained on search behaviour is not a selector for a social feed.

What no model can judge for you

• Whether the headline is honest. The highest-scoring candidate is frequently the one that overpromises, because overpromising works in the short run and your archive records the short run. Only you know what the article delivers.
• What it costs you when it does not deliver. A headline that wins clicks and loses trust is a bad trade that no click-based metric can see, and the loss is realised on the next piece rather than this one.
• House constraints. Legal sensitivities, a live news situation, an ongoing dispute, an internal agreement about how a competitor is described. None of that is in the context.
• Whether the article is worth a headline at all. Optimising the title of something that should not be published is the most common way this whole exercise is misused — the topic of the slop page.