Retail Shelf Monitoring: Detecting Out-of-Stock Items From Camera Images

Shelf monitoring looks like an object detection problem and is mostly a camera placement problem. The model you can choose; the number of pixels landing on a product facing is decided by the mount, and it decides what the model can possibly do.

Two ways to define the target

There are two quite different things a system can be asked to find, and conflating them is the most common design error.

Void detection segments empty shelf: regions where the shelf backing, the rail, or the pegboard is visible where product should be. It is a two-class segmentation problem on a fairly consistent visual target, it needs no product identity at all, and it generalises across stores. What it cannot tell you is which product is missing, and it will not notice a facing that is fully stocked with the wrong item.

Planogram compliance detects every product facing, identifies each one, and compares the observed arrangement against the planogram — the store’s specification of what sits where. It answers the useful question directly and requires SKU-level recognition across thousands of near-identical packages.

Systems that work in practice run both: void segmentation is cheap enough to run on every frame and is the trigger, and product recognition runs on the triggered region to decide whether the gap is a genuine out-of-stock, a misplaced item, or a shopper’s hand. That split also matters for the alerting, because a void is not automatically an action — a slot that is empty for ninety seconds during a restock is not a problem.

The pixels-per-centimetre calculation

Start from the lens. A camera with horizontal field of view theta at distance d from a flat shelf face covers a horizontal width of

width = 2 * d * tan(theta / 2)
resolution = sensor_width_px / width

Assume a fixed camera mounted on the opposite shelf bay looking across a 2.4 m aisle, so d = 2.0 m to the facing plane, with a 90° horizontal field of view and a 3840-pixel wide sensor. Those three numbers are assumptions about a mount, not measurements; the arithmetic is the part to keep.

width      = 2 * 2.0 * tan(45 deg) = 4.00 m = 400 cm
resolution = 3840 px / 400 cm         = 9.6 px/cm

a 7 cm wide product facing = 67 px across
a 2 mm high character on the label = 1.9 px  -- unreadable

Sixty-seven pixels is comfortable for a detector and hopeless for reading a label, which settles a design question immediately: identity has to come from packaging appearance, not from text or barcodes. To read 2 mm characters you would want on the order of 5 pixels per character stroke, so roughly 25 px/cm — nearly three times this mount delivers. You get there by moving the camera to 0.7 m, which shrinks its coverage to 1.4 m and triples the camera count.

Now the trap that catches more projects than the lens does. Detectors run at a fixed input size, and a 3840-pixel frame resized to a 640-pixel network input is downscaled 6×. That 67-pixel facing becomes 11 pixels, which is below the size at which any standard detector reliably fires. The effective resolution is set by the network input, not the sensor.

3840 px frame -> 640 px input        : 67 px facing becomes 11 px
tiled 3 x 2, each 1280x1080 -> 640  : 67 px facing becomes 34 px
                                      at 6x the inference cost per frame

Tiling is the standard answer and its price is exactly the tile count. That multiplier is the dominant term in what detection costs at scale, and it is worth settling before choosing hardware rather than after.

How many cameras cover one aisle

With 4.00 m of coverage per camera, allow a 15% overlap so that a product falling on a seam is fully visible in at least one view. That leaves 3.40 m of unique coverage per camera. For a 12 m aisle bay:

cameras per side = ceil(12.0 / 3.40) = ceil(3.53) = 4
cameras per aisle (both sides)          = 8
30-aisle store                          = 240 cameras

Every one of those numbers moves with the mount distance, and it moves in opposite directions for coverage and resolution: halving d to 1.0 m doubles px/cm to 19.2 and halves coverage to 2.0 m, so the store needs roughly 480 cameras instead of 240. That single trade — resolution against camera count, both driven by mount distance — is the design decision, and it is decided by whether you need label text. Fixed cameras are not the only option: shelf-edge cameras looking across the opposite bay, a robot traversing the aisle on a schedule, and a handheld or trolley-mounted capture all change the geometry completely, and the same arithmetic sizes them.

Why SKU identity is a retrieval problem

A grocery store carries tens of thousands of SKUs, many of which differ only in flavour text on otherwise identical packaging, and packaging changes several times a year. A closed-set classifier with one output per SKU would need retraining on every packaging change and every new line, which is untenable.

The workable structure separates localisation from identity. Localisation is a single-class detector: find every product facing, without caring what it is. That is exactly the task the SKU-110K dataset defines — Goldman and colleagues released 11,762 images of retail shelves with roughly 1.73 million bounding boxes, all labelled with one class, in “Precise Detection in Densely Packed Scenes” (CVPR 2019). Identity is then a nearest-neighbour lookup: crop each detected facing, embed it, and match against a gallery of reference product images. Adding a SKU means adding vectors to an index, not retraining a model. See image similarity search with embeddings.

Dense shelves break one specific piece of standard detector machinery. Identical products stand adjacent with nearly identical boxes, and non-maximum suppression exists to delete boxes that overlap an accepted one. Six cans of the same soup in a row can be suppressed down to two, which reads as an undercount and, if you are inferring stock level from facings, as a false out-of-stock. Raising the NMS threshold admits duplicates elsewhere; the mechanism and the alternatives are in detecting occluded and overlapping objects, and the counting formulation is in counting objects in an image.

What breaks in an actual store

• Shoppers. A person stands in front of the shelf for most of the busiest hours. No single frame is trustworthy; the state of a slot should be a temporal statistic — the median over the frames in which the slot was actually visible — not a reading. That also means the system is least reliable exactly when stock moves fastest.
• Parallax and depth. A product pushed to the back of a deep shelf is invisible from a shallow angle and reads as a void. This is a real cause of false alerts and it is a geometry problem, not a model problem: it is fixed by mounting higher, by a second view, or by accepting that back-of-shelf stock is out of scope.
• Occluding fixtures. Price rails, shelf talkers and promotional strips cover the bottom of every facing. Since they are fixed, the practical answer is a per-camera mask rather than asking the model to cope.
• Lighting. Aisle lighting varies with fixture age, with daylight near the entrance, and with the chiller cabinets, which are both colder in colour temperature and reflective. A model trained in one store degrades in the next for this reason before any other; it is the same mechanism described in why a classifier that tests well fails in production.
• Phantom inventory. The genuine business value is usually not the count. It is the disagreement between what the camera sees and what the inventory system believes, because that gap is invisible to every other system in the store and is where the lost sales are.