Skip to content
Multigrid

Core computer vision tasks

Detection, segmentation, pose and similarity — what each task actually outputs, and why a classifier that tests well still fails in production.

The first question to settle about a vision system is not which architecture it uses. It is what shape comes out of it. A classifier returns one label for a whole image; a detector returns a variable-length list of boxes with scores; a semantic segmenter returns an integer per pixel; an instance segmenter returns a list of masks; a pose model returns an ordered array of coordinates with a confidence each. Those are four genuinely different data structures, and almost every design argument that looks like it is about accuracy turns out, on inspection, to be about which of them the downstream code needed.

These pages work outward from the output. Each one names the metric that scores that output, does the arithmetic on a small example you can check by hand, and then goes to the place where the mechanism breaks — the crowded frame, the touching objects, the subgroup with a different score distribution, the confident wrong answer. Where a figure comes from a published evaluation it is named and dated; where it is an assumption in a worked calculation it is labelled as one.

Object Detection Algorithms: How a Model Draws a Box

How a detector goes from a grid of anchor boxes to a short list of final boxes, with the intersection-over-union and non-max suppression arithmetic worked out.

10 min read

Semantic and Instance Segmentation: The Difference That Matters

What each kind of segmentation puts in the output tensor, worked on a small pixel grid where two touching objects merge under one and separate under the other.

10 min read

Few-Shot Image Classification: Learning a New Category From Five Photos

How a prototype-based classifier adds a class from five examples without gradient updates, with the distances and the softmax worked out on real vectors.

10 min read

Image Similarity Search With Embeddings

How images become vectors, what an index does with them, and why the definition of similar is set by the encoder's training objective rather than by your query.

10 min read

Defect Detection in Manufacturing With Computer Vision

How to set a defect detector's operating point from the cost of an escape versus the cost of a false reject, worked through on a stated line rate and defect rate.

10 min read

Facial Recognition Accuracy and Where Bias Comes From

A single accuracy figure for a face recognition system conceals demographic differentials that the published audits measured directly; this argues the differential is the number, and the mechanism that produces it.

11 min read

Counting Objects in an Image With a Vision Model

Why detection-based counting has a hard density ceiling, worked out in pixels, and what density regression does instead.

10 min read

Human Pose Estimation Explained

How keypoint models produce a skeleton from heatmaps, and how OKS scores it — worked with COCO's real per-joint tolerances.

10 min read

Calibrating Image Classifier Confidence Scores

Why a classifier that says 0.9 is right far less than 90% of the time, with expected calibration error and temperature scaling computed on a worked table.

10 min read

Synthetic Image Data for Training a Classifier

When rendered or generated images help a vision model, why the domain gap appears, and how to work out the real-to-synthetic mixing ratio for your own batch size.

10 min read

Image Annotation Formats: COCO, YOLO and Pascal VOC Explained

The same bounding box written three ways, converted step by step, plus the four differences between the formats that silently corrupt a converted dataset.

10 min read

Panoptic Segmentation: Combining Semantic and Instance Masks

Why a panoptic output assigns every pixel exactly once, and how Panoptic Quality turns that constraint into a single number.

9 min read

Detecting Image Manipulation and Splicing

The compression, sensor and lighting signals that reveal a composited region, and the conditions under which each of them stops working.

10 min read

Visual Question Answering: How a Model Answers a Question About an Image

How the visual and text encoders are joined, from element-wise products through cross-attention to feeding image tokens straight into a language model.

10 min read

Retail Shelf Monitoring: Detecting Out-of-Stock Items From Camera Images

Why shelf monitoring splits into void detection and product identity, and how the pixels-per-centimetre calculation decides how many cameras an aisle needs.

10 min read

What It Costs to Run Object Detection at Scale

A cost model for detection built from images per day, measured throughput and a stated hourly rate, with the decode, utilisation and storage terms that usually dominate it.

10 min read

What Classification, Detection and Segmentation Each Actually Output

One image through all three tasks, with the exact tensor shapes, coordinate conventions and post-processing each one needs.

9 min read

Detecting a Damaged or Occluded Object in an Image

Why occlusion costs recall through the IoU threshold and through non-maximum suppression, and what changes in annotation, training and inference.

10 min read

Cropping and Aligning Faces Before Recognition

How five landmarks and a similarity transform normalise a face into a canonical crop, worked with real template coordinates, and what the transform cannot fix.

10 min read

Zero-Shot Object Detection With Open-Vocabulary Models

How replacing a detector's classification weights with text embeddings lets it find a class it never saw labelled, and why one score threshold stops working when you do.

10 min read

Detecting Anomalous Frames in a Quality-Control Camera Feed

How a per-frame anomaly score becomes an alarm, with the control-limit arithmetic that decides whether the line gets one alert a shift or one every twelve seconds.

11 min read

Why an Image Classifier That Tests Well Fails in Production

Four causes account for nearly every large gap between test accuracy and production accuracy, and each one is distinguishable in an afternoon.

11 min read

Other topics