Neural Network Playground

Build a tiny network, train it in your browser with real backpropagation, and watch the decision boundary bend.

Hidden unitsThe one control. At 1 the boundary is a straight line no matter how long you train it; the interesting jump is 1 → 3.

4 units

Dataset

Learning ratefull-batch gradient descent — a slider rather than a field, because the useful range here is a hundredth wide

0.50

Seedinitial weights and data — the link reruns this exactly

Held-out accuracy

45.0%

61.3% on the training set after 0 epochs, with 17 parameters. Press Train and watch it move; press Reset and it starts from the same weights every time.

Every square is the network's output at that point, evaluated live. Dots are the training data. This is the real decision boundary of the weights currently in memory, not a picture of one.

Epochs run: 0
Parameters: 17
Training points: 160
Training accuracy: 61.3%
Held-out accuracy: 45.0%
Training loss (cross-entropy): 0.6648
Held-out loss: 0.7189

What this assumes: one hidden layer of tanh units, a sigmoid output, binary cross-entropy, plain full-batch gradient descent, no regularisation, no momentum and no schedule — chosen so that every number above comes from a loop you could write out by hand. Weights are initialised from the seed in the URL, so the same link produces the same run rather than a different one each visit. Training accuracy on 160 points and held-out accuracy on 120 fresh points from the same generator; nothing here is a claim about any real model.

Set hidden units to 1 and pick the circle dataset, then train as long as you like. It will not work, and the reason is visible rather than asserted: one tanh unit followed by a linear output can only produce a single soft threshold across the plane, so the boundary is a straight line and a circle cannot be a straight line. This is the same wall the field hit in 1969 with the perceptron and XOR. Now move the slider to 3. The boundary bends. Nothing else changed.

That is what a hidden layer buys: each unit contributes one oriented soft step, and the output layer takes a weighted sum of them, so with enough units you can enclose a region by adding steps from several directions. It also explains why the improvement flattens — going from 1 to 3 units changes what is representable at all, while going from 8 to 12 mostly changes how quickly it is found. Watch the two accuracy rows rather than the picture: past a point, extra capacity starts helping the training number more than the held-out one, which is the same U-curve the overfitting demo draws explicitly.

What this leaves out is scale and everything that comes with it: no mini-batches, no depth, no normalisation, no attention, and 49 parameters where a modern model has hundreds of billions. The mechanism is genuinely the same — a forward pass, a loss, gradients by the chain rule, a small step downhill, repeated — but do not read the training dynamics here as a miniature of a real training run. They are not.

Neural Network Playground

The reasoning behind this

Related tools