The Probability You Actually Need

Four ideas cover almost everything: conditional probability, the chain rule, independence, and Bayes’ theorem. A language model is the second one, and the fourth explains why a detector that is right 95% of the time can still be wrong about most of the things it flags.

Conditional probability is the whole model

P(A | B) = P(A and B) / P(B)

“Given that B happened, how likely is A?” The division renormalises: you have thrown away every world where B did not happen, so what is left must be scaled back up to sum to 1.

Every output of a language model is a conditional probability. The quantity a model computes is P(next_token | all preceding tokens), and the “given” part is the context window. This is why nothing outside the context exists to the model: the conditioning set is exactly the tokens you sent, and a fact you did not include is not conditioned on and cannot influence the answer.

Concrete, from a three-token vocabulary:

P(mat, given "the cat sat on the")   = 0.659
P(rug, given "the cat sat on the")   = 0.242
P(sofa, given "the cat sat on the")  = 0.099
                                       -----
                                       1.000

Change the context to "the cat sat on the wet"
and all three numbers change. Same model, no retraining.

The chain rule is autoregressive generation

The probability of a whole sequence factorises into a product of conditionals, each one conditioned on everything before it:

P(t1, t2, ..., tn) = P(t1)
                   * P(t2 | t1)
                   * P(t3 | t1, t2)
                   * ...
                   * P(tn | t1 ... t(n-1))

That identity is exact, requires no assumptions, and is the entire justification for generating one token at a time. A model that can produce each conditional can produce the joint probability of any sequence, and can sample from it.

It also explains why long generations have astonishingly small probabilities:

Ten tokens, each at p = 0.9:

  0.9^10 = 0.348678

Fifty tokens, each at p = 0.9:

  0.9^50 = 0.005154

In log space (which is why everyone works in log space):

  ln(0.9) = -0.105361
  50 * -0.105361 = -5.268
  exp(-5.268) = 0.005154

A 500-token answer where every token is a comfortable 0.9 has a joint probability around 1e-23. Sequence probabilities are only usable in log space, they must be length-normalised before being compared across different lengths, and the failure to normalise is the classic reason a beam search prefers short answers.

Independence, and where it is assumed wrongly

If A and B are independent:  P(A and B) = P(A) * P(B)

Multiplying probabilities is only valid under independence, and in practice independence is usually assumed rather than checked. Three places this goes wrong in AI work, all of them common:

• “Sample five times and take the majority”. Self-consistency assumes the five samples fail independently. They do not: the same misunderstanding of the prompt produces the same wrong answer five times, and the majority vote confirms it. The technique works well where errors are sampling noise and not at all where they are systematic.
• Cascading two models for reliability. If model A is 95% accurate and a checker B is 95% accurate, the failure rate is only 0.05 * 0.05 = 0.0025 when the two fail independently. When B is the same family of model looking at the same ambiguous input, the correlation is high and the real figure is much closer to 0.05.
• Retrieval and generation errors. Treating a RAG pipeline as independent stages understates the joint failure rate, because the queries that retrieve badly are disproportionately the queries the generator would also handle badly.

Bayes, and the 95%-accurate detector that is wrong 72% of the time

P(A | B) = P(B | A) * P(A) / P(B)

Bayes’ theorem turns a conditional round. You know how often the test fires when the thing is true; you want to know how often the thing is true when the test fires. Those are different numbers and the gap between them is where nearly all bad intuition about classifiers lives.

Set up a concrete, realistic case. You have a hallucination detector. It flags 95% of genuinely wrong answers (sensitivity) and wrongly flags 5% of correct ones (a 95% specificity). Your system produces a wrong answer 2% of the time.

Assumptions:
  P(wrong)          = 0.02
  P(flag | wrong)   = 0.95
  P(flag | right)   = 0.05

Marginal probability of a flag:

  P(flag) = P(flag|wrong)*P(wrong) + P(flag|right)*P(right)
          = 0.95 * 0.02          + 0.05 * 0.98
          = 0.019                + 0.049
          = 0.068

Bayes:

  P(wrong | flag) = P(flag|wrong) * P(wrong) / P(flag)
                  = 0.019 / 0.068
                  = 0.2794

72.1% of everything this detector flags is a correct answer.

Count it out in whole numbers if the algebra is unconvincing. Take 10,000 answers: 200 are wrong and 9,800 are right. The detector flags 190 of the 200 wrong ones, and 490 of the 9,800 right ones. Total flags: 680, of which 190 are genuine. 190 / 680 = 27.9%.

The base rate did that, not the detector. At a 20% error rate instead of 2%:

P(flag) = 0.95*0.20 + 0.05*0.80 = 0.19 + 0.04 = 0.23
P(wrong | flag) = 0.19 / 0.23 = 0.826

Same detector, 83% precision instead of 28%.

Which is the practical rule: a classifier looking for something rare spends most of its output on false positives, no matter how good it sounds in the spec. If you are building a hallucination check or a prompt injection filter, the base rate of the thing you are catching determines whether the alerts are worth reading, and it is the first number to estimate.

Expectation, for anything you have to budget

E[X] = sum_i p_i * x_i

Expectation is a weighted average of outcomes, and it is the tool for any cost that depends on something uncertain. A concrete routing decision:

Cheap model handles a request for $0.002 and succeeds 80%
of the time. On failure you retry with an expensive model
at $0.030. What does a request cost on average?

  E[cost] = 0.002 + 0.20 * 0.030
          = 0.002 + 0.006
          = $0.008

Always using the expensive model:  $0.030
Cascade:                           $0.008
Saving:                            73%

Break-even success rate s, where cascade = expensive:
  0.002 + (1 - s) * 0.030 = 0.030
  (1 - s) * 0.030 = 0.028
  1 - s = 0.9333
  s = 6.7%

The cascade wins unless the cheap model succeeds
less than 7% of the time.

Two things the expectation hides, both of which matter. The average says nothing about the tail: 20% of users wait for two calls, so the p95 latency roughly doubles even though the mean cost fell. And it assumes you can detect failure cheaply — if detection requires the expensive model anyway, the arithmetic collapses.

Four probability traps in AI work

• Confusing a probability with a confidence. A token probability of 0.99 means the model finds that continuation overwhelmingly likely given its training distribution. It is not a 99% chance the statement is true, and logprobs are a calibration signal at best.
• Averaging probabilities that came from different conditionings. The mean of per-token probabilities across answers of different lengths is not a meaningful quantity. Use the mean log probability, and say what you normalised by.
• Ignoring the prior when reading an evaluation. A model that scores 90% on a benchmark where the majority class is 85% has demonstrated very little. Always compare against the trivial baseline.
• Treating a sampled estimate as exact. Any probability estimated from n examples carries an error bar of roughly sqrt(p(1-p)/n), and on typical benchmark sizes that bar is wider than the differences being reported.