Skip to content
Multigrid

Machine learning foundations

The layer under the language models: how a model is fitted, how it fails, and how you find out which of those is happening.

Everything a language model does is machine learning, and almost none of it is new machine learning. The objective a frontier model is trained on is the cross-entropy loss from a 1980s classification textbook. The thing that makes fine-tuning overfit is the thing that made a decision tree overfit. Benchmark contamination is train/test leakage with a bigger corpus. If you learn this layer once, a surprising amount of what looks like language-model folklore turns into a special case of something you already understand.

So these pages are written for someone who arrived from the LLM side and wants the foundation underneath rather than a course. Each one derives the thing it is about — the actual algebra, with the assumptions labelled — and then says where that same object shows up when you are fine-tuning, evaluating or retrieving. Where a claim needs evidence, the paper is named. Where nobody has run the experiment, the page gives you the code to run it instead of pretending.

Machine Learning vs Deep Learning vs AI: Three Nested Sets

The three terms are not synonyms and not rivals — they nest, and the boundaries are drawn by two specific questions you can ask about any system.

4 min read

Supervised, Unsupervised and Reinforcement Learning

The four learning paradigms sorted by the only thing that decides which one you can use: what data you are actually able to collect.

4 min read

How a Neural Network Learns: Gradient Descent by Hand

One full step of gradient descent on two parameters, with the arithmetic written out, then the same three lines generalised to a billion.

4 min read

Backpropagation Explained Without Calculus Anxiety

Backprop is the chain rule applied in a particular order, and the order is the whole trick — here it is on a four-parameter network, with numbers.

5 min read

Loss Functions: Choosing the Right One

What each common loss punishes, what it therefore converges to, and the four-line derivation showing why cross-entropy's gradient is simply prediction minus target.

5 min read

Overfitting, Underfitting and the Bias-Variance Trade-off

The bias-variance decomposition derived from scratch, the two learning-curve shapes it predicts, and the code to draw your own.

4 min read

Train, Validation and Test: Why Three Splits

Why two splits are not enough, how much optimism repeated model selection buys you, and the six leakage patterns that make an offline result meaningless.

5 min read

Cross-Validation and When It Is Overkill

The standard error of a held-out estimate, derived, so the choice between one split and k-fold becomes arithmetic — plus the variance problem k-fold does not solve.

4 min read

Precision, Recall, F1 and When Accuracy Lies

One worked confusion matrix on a 1% positive class, where the useless model scores 99% and the useful one scores 28%.

5 min read

ROC and Precision-Recall Curves: Reading Them Right

One formula explains why a classifier can hold a beautiful ROC curve and a hopeless precision at the same time — and why prevalence decides which curve to look at.

5 min read

Feature Engineering: Still the Highest-Leverage Work

A feature is a change of coordinates that puts the answer inside a model class that could not otherwise reach it — shown by construction on XOR, then applied to four transformations that keep paying.

5 min read

Gradient Boosting: Why XGBoost Still Wins on Tables

How boosting builds a model out of residuals, and what the published tabular benchmarks actually report about deep learning on the same data.

5 min read

Regularisation: L1, L2, Dropout and Early Stopping

Four techniques with one goal, each derived far enough to show what it actually does to a weight — including why L2 is called weight decay and why L1 produces exact zeros.

5 min read

Learning Rate: The One Hyperparameter That Matters Most

The stability bound derived on a quadratic — why training diverges above 2/L, why the useful range is narrow, and how to find yours in a few hundred steps.

4 min read

Batch Size, Steps and Epochs: What Each Actually Controls

The one equation relating all three, why batch size is really a gradient-noise setting, and the two published learning-rate scaling rules that disagree about it.

4 min read

Optimisers: SGD, Adam, AdamW and Why It Changed

Momentum, adaptive scaling and decoupled weight decay, each derived — including the specific bug in Adam that AdamW fixed and why it mattered enough to change every training script.

5 min read

Transfer Learning: Standing on a Bigger Model's Shoulders

Three different things are called transfer learning, separated by which parameters are allowed to move — and the parameter counts that make a linear probe and a LoRA adapter the same idea.

5 min read

Class Imbalance: Techniques That Actually Help

Resampling, class weighting and threshold tuning are one operation applied at three points — the Bayes correction shows why, and which point is cheapest.

4 min read

Model Interpretability: SHAP, LIME and Their Limits

What each method computes, the axioms one of them uniquely satisfies, and the published attack showing an explanation can be clean while the model is not.

4 min read

When Not to Use Machine Learning

Four preconditions a problem must meet before machine learning is the right tool, and the value formula that decides it when they are all met.

5 min read

Other topics

Machine learning foundations · Multigrid