Batch Size, Steps and Epochs: What Each Actually Controls

Three words, constantly confused, and one equation relates them. Once the equation is on the page, the question “should I increase the batch size or the number of epochs?” turns out to be two unrelated questions wearing one coat.

The equation, worked

steps per epoch = ceil(N / B)
total updates   = E × ceil(N / B)

N = 50,000 examples

B =  32,  E = 10   →  1,563 steps/epoch  →  15,630 weight updates
B = 256,  E = 10   →    196 steps/epoch  →   1,960 weight updates
B = 256,  E = 80   →    196 steps/epoch  →  15,680 weight updates

Read the middle line and the confusion dissolves. Going from batch 32 to batch 256 while keeping epochs fixed does not train the model harder or faster in any meaningful sense — it sees exactly the same data and performs eight times fewer updates. If nothing else changes, the model is usually worse, and the reason is not mysterious: it took an eighth as many steps.

• Epoch — one full pass over the training data. A unit of data consumption. Says nothing about how much learning happened.
• Batch size — examples per gradient estimate. A statistical setting and a memory setting at once.
• Step — one weight update. The only one of the three that is a unit of learning, and the one worth logging against.

Batch size is a noise setting

The minibatch gradient is an average of B per-example gradients, so it is a sample mean, and the standard error of a sample mean is σ/√B. Four times the batch halves the noise; sixteen times quarters it. Diminishing returns are built in — and this is the entire statistical content of the batch size choice.

Which means gradient noise is not purely a defect. A noisy gradient perturbs the trajectory, and that perturbation is a mild regulariser which helps small-data training escape sharp regions of the loss surface. Very large batches make the gradient nearly exact, remove the perturbation, and can generalise worse for reasons that are still argued about. So “the largest batch that fits” is a hardware heuristic, not a statistical one.

There is also a ceiling on what a larger batch buys, and it has a name. McCandlish, Kaplan, Amodei and the OpenAI Dota team (2018, “An Empirical Model of Large-Batch Training”) describe a critical batch size: below it, doubling the batch roughly halves the number of steps needed, so you get near-perfect parallel speedup; above it, the gradient is already accurate enough that extra examples buy almost nothing and you are spending compute for wall clock. They estimate it from the gradient noise scale, a quantity you can measure on your own training run. The practical shape of the result is the one to carry: there is a regime where more parallelism is nearly free and a regime where it is nearly wasted, and which one you are in depends on the task rather than the hardware.

Two scaling rules that disagree

If you change B, you must revisit η, because you have changed the noise the learning rate was tuned against. Two published rules exist, and they say different things:

• Linear scaling. Multiply B by k, multiply η by k. Goyal et al. (2017, “Accurate, Large Minibatch SGD”) used this with a warmup period to train ImageNet at very large batch sizes with SGD and momentum. The intuition: a k× larger batch means k× fewer steps per epoch, so each step should cover k× the ground.
• Square-root scaling. Multiply η by √k instead, the rule associated with Krizhevsky (2014). The intuition: keep the ratio of update size to gradient noise constant, and noise falls as √B.

They are not reconcilable and neither is universal — they were stated for different optimisers and regimes, and the literature does not hand you one answer. The practical reading: after a large batch-size change, re-run the range test rather than applying a rule. Use the rules to know which direction to look and roughly how far, not as a substitute for measuring.

The hardware half

Batch size is also the biggest lever on training memory, because activations must be stored for the backward pass and there is one set of them per example in the batch. Memory is roughly linear in B, which is why out-of-memory errors are almost always fixed by halving it.

When the batch size you need statistically is larger than the one that fits, gradient accumulation gives you the first without the second:

accum = 8                       # effective batch = 8 × microbatch

opt.zero_grad()
for i, micro in enumerate(loader):
    loss = model(micro).loss / accum      # divide, or the effective LR is 8x
    loss.backward()                       # gradients accumulate in .grad
    if (i + 1) % accum == 0:
        opt.step()
        opt.zero_grad()

The division by accum is the line everyone forgets. Without it the accumulated gradient is eight times too large, which is indistinguishable from having raised the learning rate eightfold — and by the stability bound, that is often straight past the divergence point.

Epochs in fine-tuning

The epoch counts that make sense differ wildly by regime, and carrying a habit across them is a common mistake.

• Pretraining is typically a single pass, or close to it, over a corpus far too large to repeat. “Epochs” barely applies; the meaningful axis is tokens seen.
• Fine-tuning on a task dataset is usually one to three epochs. Past that, held-out loss rises while training loss keeps falling — textbook overfitting, and the shape it takes here is losing capabilities you did not intend to touch.
• Small classical models on tabular data may need hundreds of epochs, because each epoch is a handful of steps.

The habit that survives all three: log against steps and against tokens or examples seen, never against epochs alone. An epoch is a different amount of learning every time you change the batch size, and a chart indexed by epochs hides that.