Few-Shot Image Classification: Learning a New Category From Five Photos

Few-shot classification does not fine-tune anything. It embeds your five examples, averages them into a point, and classifies a new image by which point it is nearest to. The mechanism is a nearest-centroid rule in a learned space, and everything interesting about it is in the word learned.

The setup, and what N-way K-shot means

The vocabulary is precise and worth getting right, because papers quote scores that are meaningless without it. An N-way K-shot task gives the model N classes it has never seen, with K labelled examples of each — the support set — and asks it to label a query image drawn from those N classes. A 5-way 1-shot score and a 5-way 5-shot score on the same model differ by a wide margin, and a 2-way task is a coin flip at chance while a 20-way task is not. When somebody reports “94% few-shot accuracy”, the number is uninterpretable until N and K are attached to it.

Note also what the setup does not include: no gradient step is taken at test time. That is the operational appeal. Adding a new product to a visual inspection line means computing five embeddings and writing them to a table, which takes milliseconds and cannot damage the accuracy of any existing class — unlike a fine-tune, which can.

The prototype mechanism

Prototypical networks, from Snell, Swersky and Zemel (2017), are the cleanest version. Push each support image through a frozen encoder to get a vector. Average the K vectors of a class to get its prototype. Classify a query by taking the squared Euclidean distance to each prototype and applying a softmax over the negated distances. That is the entire inference path; there are no learned parameters in it beyond the encoder.

The choice of squared Euclidean rather than cosine is not arbitrary in the original paper — with a squared Euclidean distance the resulting classifier is a linear model in the embedding, which is why averaging is the right way to combine support examples at all. Swapping in cosine distance, as many implementations do, changes that property and usually requires a learned scale factor to compensate.

A 2-way 5-shot classification, worked

Real embeddings have hundreds of dimensions; four is enough to show the arithmetic. Two classes on a tile inspection line, five support images each, encoder output in four dimensions:

class "cracked"  support embeddings
  (0.9, 0.1, 0.2, 0.1)
  (0.8, 0.2, 0.3, 0.0)
  (0.9, 0.0, 0.1, 0.2)
  (0.7, 0.1, 0.3, 0.1)
  (0.8, 0.1, 0.2, 0.1)
  sum       = (4.1, 0.5, 1.1, 0.5)
  prototype = (0.82, 0.10, 0.22, 0.10)

class "clean"    support embeddings
  (0.1, 0.9, 0.1, 0.2)
  (0.2, 0.8, 0.0, 0.3)
  (0.1, 0.9, 0.2, 0.1)
  (0.0, 0.7, 0.1, 0.2)
  (0.1, 0.8, 0.1, 0.2)
  sum       = (0.5, 4.1, 0.5, 1.0)
  prototype = (0.10, 0.82, 0.10, 0.20)

query q = (0.75, 0.20, 0.25, 0.15)

d2(q, cracked) = 0.0049 + 0.0100 + 0.0009 + 0.0025 = 0.0183
d2(q, clean)   = 0.4225 + 0.3844 + 0.0225 + 0.0025 = 0.8319

softmax over (-0.0183, -0.8319)
  difference           = 0.8136
  exp(0.8136)          = 2.256
  p(cracked)           = 2.256 / (2.256 + 1) = 0.693
  p(clean)             = 0.307

The query is 45 times closer to the cracked prototype by squared distance, and the classifier reports 69% confidence. That gap between “overwhelmingly nearer” and “69%” is not a rounding artefact and it is the most useful thing on this page.

Why that 69% is not a probability

The softmax is applied to raw negated distances, so its sharpness depends entirely on the numeric scale of the embedding space. These vectors have components under 1, so the squared distances are small fractions, so the logit gap is small, so the output is close to uniform. Multiply every embedding by 10 — a change that leaves every ranking and every nearest-neighbour decision identical — and the squared distances scale by 100, the logit gap becomes 81.36, and the same classification reports a confidence indistinguishable from 1.0.

One decision, two confidences, differing by an arbitrary constant. That is why implementations insert a learned temperature or scale on the distances, and why you should never threshold a prototypical network’s output for an abstain or human-review path without fitting that scale on held-out data first. It is the same machinery as temperature scaling for a normal classifier, applied to a head that has an even weaker claim to be calibrated because it never saw a training label for these classes at all.

The backbone decides everything

The prototype rule contributes almost nothing. What decides whether five photos are enough is whether the frozen encoder already places your new category in a tight region — and that depends on what it was trained on. An ImageNet-classification backbone learns features that separate its own thousand categories, so it separates a new dog breed easily and a new grade of surface finish barely at all, because nothing in its training objective ever rewarded telling two shades of brushed metal apart.

This is also the honest answer to why a CLIP-style image encoder often beats a purpose-trained one at few-shot: its contrastive objective over a very broad image distribution produces a space where far more distinctions are already linear. The practical test costs an hour: embed 50 images per candidate class with the encoder you propose to use, and look at whether the within-class distances are smaller than the between-class distances. If they are not, no amount of prototype cleverness will help, and the fix is a better encoder or a fine-tune — the same measurement that decides whether embedding-based similarity search will work on your catalogue.

Where five photos are not enough

• The five are not five. Five frames from one video of one object are, in embedding terms, close to one example: the prototype inherits that object’s exact lighting and pose. Five genuinely different instances beat fifty frames of one.
• The class is defined by something the encoder discards. Encoders trained with aggressive colour jitter learn to ignore colour. If your new class is “the red variant”, the augmentation used in pretraining has deliberately destroyed the only signal you need.
• The prototype is a mean, and means are fragile. One mislabelled support image moves the prototype by a fifth of the way to the wrong class. With K=5 there is no redundancy to absorb it, so support sets deserve more label scrutiny per image than a training set does.
• A linear probe may simply beat it. With five examples per class the prototype rule is hard to improve on, but by thirty or fifty per class, fitting a logistic regression on the same frozen embeddings usually wins — it can learn that some dimensions matter more than others, which a plain mean cannot. It costs seconds to fit. The prototype’s remaining advantage is that it updates instantly when a class gains an example, so it suits a catalogue that changes daily better than a fixed class list does.
• Nothing here handles “none of the above”. The softmax is over the N classes you supplied, so an image of something else is assigned confidently to whichever prototype it happens to be nearest. Open-set rejection needs an explicit distance threshold, and that threshold has the scale problem described above.