Convolutional Networks, and Where They Still Win

A convolution layer slides one small set of weights across the whole input and reuses it at every position. That single decision — share the weights, look only locally — is worth about seven orders of magnitude in parameter count on a typical first layer, and everything CNNs are good and bad at follows from it.

The shapes: what goes in and what comes out

A convolution operates on a four-dimensional tensor: batch, channels, height, width. An RGB image at 224 by 224 arrives as (N, 3, 224, 224). A layer’s weights are a four-dimensional tensor too — (C_out, C_in, kh, kw) — and the output is (N, C_out, H_out, W_out).

The output size is not a matter of taste. It is fixed by the kernel, the stride and the padding:

H_out = floor((H_in + 2*pad - kernel) / stride) + 1

224 input, 7x7 kernel, stride 2, pad 3:
  floor((224 + 6 - 7) / 2) + 1 = floor(223/2) + 1 = 111 + 1 = 112

So a 7 by 7 stride-2 layer with 64 filters turns (N, 3, 224, 224) into (N, 64, 112, 112). Channels are the axis you grow; spatial extent is the axis you shrink. A whole CNN is that swap repeated: 3 channels at 224 by 224 becomes 512 channels at 7 by 7, at which point a pooling layer collapses the spatial axes entirely and a linear layer reads off a class.

Parameters and FLOPs, worked

The parameter count of a convolution does not depend on the size of the image at all, which is the whole point:

params = C_out * C_in * kh * kw   + C_out (bias)
       = 64 * 3 * 7 * 7           + 64
       = 9,408                    + 64  = 9,472

Now price the alternative. A fully connected layer producing the same output from the same input needs one weight per input-output pair:

inputs  = 224 * 224 * 3   = 150,528
outputs = 112 * 112 * 64  = 802,816
params  = 150,528 * 802,816 = 120,850,000,000  (about 121 billion)

Nine and a half thousand weights against a hundred and twenty-one billion, for the same input and the same output shape. The ratio is roughly thirteen million to one. That is not an efficiency tweak; it is the difference between a model that can be trained on the images that exist and one that cannot.

The arithmetic, though, is not small, because the shared weights are applied at every position:

MACs = H_out * W_out * C_out * C_in * kh * kw
     = 112 * 112 * 64 * 3 * 7 * 7
     = 118,013,952        (about 118 million multiply-accumulates)
FLOPs ~= 2 * MACs = about 236 million, for one image, in one layer

That gap between parameters and FLOPs is the CNN’s signature. Convolutions are compute-heavy and weight-light, which is the opposite of a transformer’s decode step, where the weights are enormous and the arithmetic per token is small. It is why CNNs saturate a GPU easily and why they were the workload that made GPUs matter for deep learning in the first place.

The receptive field, and why depth is forced

A unit in the first layer sees a 7 by 7 patch. A unit in the second sees a 7 by 7 patch of first-layer units, so it sees more of the original image — but only a little more. With stride 1 and kernel k, the receptive field after L layers is 1 + L * (k - 1).

3x3 kernels, stride 1:
  after 5 layers   RF = 1 + 5*2  = 11 pixels
  after 20 layers  RF = 1 + 20*2 = 41 pixels
  to reach 224     L  = 111 layers

A hundred and eleven layers to let one unit see the whole image is absurd, and that is why every CNN downsamples. A stride-2 layer or a pooling layer halves the spatial resolution, which doubles the rate at which the receptive field grows in original-image pixels. Five downsamples and the receptive field grows 32 pixels per layer instead of two.

This is the structural contrast with attention, which is worth holding on to. In a single attention layer, every position can already see every other position. In a convolution, long-range interaction is something you buy with depth.

Three changes that made deep stacks work

A plain stack of 3 by 3 convolutions does not scale, and three modifications are what turned it into an architecture that could be made deep and cheap. All three are still in use inside models that are not CNNs at all.

The 1 by 1 convolution

A kernel of size 1 does no spatial mixing whatsoever. It is a linear layer applied independently at every position, mixing channels only, and it costs C_out * C_in parameters. That makes it the cheap way to change the channel count before doing something expensive:

Direct 3x3, 256 -> 256 channels:
  256 * 256 * 9 = 589,824 parameters

Bottleneck block, same input and output shape:
  1x1  256 -> 64    256*64      =  16,384
  3x3   64 -> 64     64*64*9    =  36,864
  1x1   64 -> 256    64*256     =  16,384
                                  --------
                                    69,632   ...8.5x fewer

Squeeze the channels, do the spatial work in the narrow space, expand again. This is the residual bottleneck block, and the same shape appears as the down-and-up projection in adapter layers and in the MLP of a transformer block, inverted.

Residual connections

Write the layer as y = x + f(x) rather than y = f(x). The gradient now reaches earlier layers through the identity path without passing through f at all, which is the same additive highway the LSTM cell state uses and for the same reason. Before residuals, networks past roughly twenty layers trained worse than shallower ones; after them, fifty and a hundred layers trained without difficulty. Every transformer block has one for exactly this reason.

Depthwise separable convolutions

Split the convolution into its two jobs. A depthwise convolution applies one k by k kernel per channel with no channel mixing; a pointwise 1 by 1 then mixes channels with no spatial extent.

Standard 3x3, 256 -> 256:
  256 * 256 * 9 = 589,824

Depthwise 3x3:      256 * 9   =  2,304
Pointwise 1x1:      256 * 256 = 65,536
                               --------
                                 67,840    ...8.7x fewer

The ratio is 1/(k*k) + 1/C_out, so it improves with kernel size
and with channel count.

That single substitution is what put convolutional vision on phones. It is also a clean statement of what a convolution really is — two independent mixings, spatial and channel — and once they are separated the same decomposition shows up everywhere, including in the separation of attention from the MLP in a transformer block.

What the convolutional prior bought and cost

Two assumptions are hard-coded into the layer, and neither is learned:

• Locality. A unit is allowed to look at a k by k window and nothing else. Pixels that are far apart cannot interact until several layers later.
• Translation equivariance. The same filter runs everywhere, so a feature detected in the top-left is detected by the same weights in the bottom-right. Shift the input and the feature map shifts with it.

What that bought: sample efficiency. The model does not have to learn from data that an edge is an edge regardless of where it appears, because it cannot represent anything else. On small labelled datasets this is decisive, and it is why a CNN trained on ten thousand images is usually still ahead of a vision transformer trained on the same ten thousand.

What it cost: the ceiling. The prior that stops the model needing data also stops it using data. A convolution cannot learn a relationship between two distant regions in one step no matter how many examples it sees, and it cannot learn to weight one region more than another based on content, because the weights do not depend on the input. Attention removed both constraints, and above a certain data scale removing them wins.

Where CNNs still win outright

Where they stop

Translation equivariance is weaker in practice than it is on paper. Padding breaks it at the borders, striding breaks it whenever the shift is not a multiple of the stride, and a pooling layer discards exactly the sub-pixel information that would have preserved it. A CNN is approximately shift-equivariant, not exactly.

It is also equivariant to nothing else. Rotate the input and the feature maps do not rotate; scale it and they do not scale. Those invariances have to be trained in with augmentation, which costs data — and the moment you are spending data to teach invariances, the argument for hard-coding one of them starts to weaken.

Finally, CNNs have not shown the same smooth returns to scale that transformers have. Adding parameters to a convolutional stack helps, then helps less, and there is no equivalent of the scaling behaviour that makes it worth training a model ten times larger. That, rather than any single benchmark, is why the frontier moved.