Skip to content
Multigrid

The maths behind AI

The twenty pieces of maths that actually appear in a working language model, each derived on the page with arithmetic you can reproduce.

There are two kinds of maths-for-AI writing and neither is much use. The first is a formula dump: the definition, the summation sign, no numbers, no consequence. The second is the hand-wave: an analogy about temperature being creativity, with nothing you could check. The gap between them is where the useful version lives, and it is not complicated — it is intuition first, then the arithmetic on a concrete example, then what that means for something you care about.

That is what every page here does. The numbers are small enough to follow on paper and are worked out line by line, so if a step is wrong you can find it. Where a page needs a hardware or model figure, the figure is an input to the sum with a name on it, not a claim about a product. Change it and the derivation still holds.

You do not need a degree for any of this. You need to be willing to multiply four numbers together and look at the answer. Almost every surprising fact about how these systems behave — why doubling context roughly quadruples attention cost, why a one-point benchmark win means nothing, why a model file is 14 GB rather than 28 — is four numbers multiplied together by somebody who bothered.

Matrix Multiplication Is the Whole Job

Every operation in a transformer written as a matrix multiply, with real shapes, FLOP counts, and where the 2 x params x tokens rule comes from.

11 min read

Vectors, Dot Products, and Why Similarity Is an Angle

The dot product worked by hand in two dimensions, the angle it encodes, and the exact line where that becomes embedding similarity.

9 min read

The Softmax Function, Line by Line

Softmax worked on three numbers, the max-subtraction trick shown to give the identical answer, and the overflow it prevents with real fp16 and fp32 limits.

9 min read

Cross-Entropy: What Training Actually Minimises

From surprise measured in bits to the number in a training log, with the loss of an untrained model derived rather than quoted.

9 min read

KL Divergence, and What Distance Between Distributions Means

KL divergence computed both ways round on the same pair of distributions to show it is not a distance, then placed exactly where it appears in RLHF and distillation.

10 min read

Perplexity: Reading the Number Properly

Perplexity is exp of the mean negative log-likelihood, and it is not comparable across tokenizers — both shown with arithmetic on the same document.

9 min read

Entropy and Information, With Tokens as the Example

Entropy computed by hand on a short string with every logarithm shown, then the same computation applied to token distributions and to your own text.

9 min read

The Probability You Actually Need

Conditionals, the chain rule and Bayes' theorem, each introduced by the specific LLM problem it solves rather than by a card-and-dice example.

10 min read

Floating Point: fp32, fp16, bf16, fp8 and int4

Bit layouts drawn out for every format in use, with the range-versus-precision trade shown as numbers and the exact problem bf16 was designed to fix.

11 min read

Numerical Stability: Where Neural Nets Quietly Break

Overflow, underflow and catastrophic cancellation shown with real fp32 and fp16 values, plus the guards hidden inside every framework and why each one is there.

10 min read

Derivatives and the Chain Rule, for People Who Forgot

One scalar chain-rule example checked numerically, then carried unchanged through a two-layer network to every gradient in it.

10 min read

Eigenvectors, PCA, and Why Dimensionality Reduction Works

PCA built from a covariance matrix on five points, with both eigenvalues verified by projecting the data, then a script for your own embeddings.

10 min read

Distance Metrics: Euclidean, Cosine, Manhattan, Hamming

Four metrics computed on the same pair of vectors, and the algebraic identity that makes three of them give the same ranking once vectors are normalised.

9 min read

Why Attention Is Quadratic

The operation count of attention derived from its shapes, converted into seconds at a labelled throughput, so the cost of doubling context becomes a number.

11 min read

Sampling From a Probability Distribution, Concretely

Inverse-CDF sampling worked with one random number, then temperature, top-k, top-p and min-p applied to the same four tokens step by step.

10 min read

Bootstrap Resampling, Done by Hand First

Uncertainty from a single dataset with no distributional assumption, worked by hand on ten measurements before the twenty-line script.

9 min read

Confidence Intervals on a Benchmark Score

A confidence interval computed on a 200-example benchmark, showing arithmetically that a one-point win is indistinguishable from noise.

11 min read

Logits, Odds and Log-Odds

From raw model scores to probabilities and back, with the odds interpretation that explains exactly what a logit bias of +2 does.

9 min read

Vector Norms, and Why Normalising Embeddings Matters

L1, L2 and infinity norms computed on the same vector, and the one-line normalisation that fixes a retrieval bug nothing throws an error about.

9 min read

How Big Is the Prompt Search Space?

The combinatorics of prompt search, and the second reason exhaustive search fails: at 32,768 configurations the best score is noise.

10 min read

Integer Quantisation and Rounding Error

The affine map derived from a real min and max, the s/2 error bound, the 6.02 dB per bit rule, and why 4-bit weighs 0.578 bytes rather than 0.5.

12 min read

Other topics