Detecting Anomalous Frames in a Quality-Control Camera Feed

Getting a good anomaly score per frame is the part that has a literature. Turning thirty scores a second into an alarm that somebody on the line will still trust at the end of a shift is the part that decides whether the system stays switched on, and it is statistical process control, not computer vision.

Scoring a frame without defect labels

You will not have a balanced dataset of defects. Defects are rare, they are varied, and the interesting ones are the kinds nobody has seen yet — which rules out supervised classification as the primary mechanism. The formulation that fits is one-class: model the distribution of normal frames and score how far a new frame sits from it.

• Feature-embedding methods. PaDiM fits a multivariate Gaussian at each spatial position over features from a pretrained backbone and scores by Mahalanobis distance. PatchCore — Roth and colleagues, “Towards Total Recall in Industrial Anomaly Detection” (CVPR 2022) — keeps a coreset-subsampled memory bank of normal patch features and scores by nearest-neighbour distance. Both train on normal images only and produce a per-pixel heat map as a by-product, which matters enormously for operator trust: an alarm you can point at is actionable, a scalar is not.
• Reconstruction methods. An autoencoder or diffusion model reconstructs the frame and the residual is the score. The characteristic failure is a model with enough capacity to reconstruct the defect too, at which point the residual vanishes exactly where you needed it.

The reference benchmark is MVTec AD — Bergmann and colleagues, “MVTec AD: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection” (CVPR 2019) — with 5,354 images across 15 categories and 73 defect types, annotated with pixel-accurate ground truth. It is a benchmark of controlled photographs of objects, not of a conveyor under factory lighting, and scores on it do not transfer to your line. See defect detection in manufacturing for the per-item framing, and the general treatment of anomaly detection.

From a score to a decision on a stream

Per-item inspection asks one question once per part: is this one defective. A continuous feed produces a score every frame whether or not a part is in view, and the decision problem is different in kind. The right frame of reference is a hundred years old: you are monitoring a process, the score is your measured statistic, and the question is when to signal that the process has changed.

That means three separate steps, and conflating them is where systems go wrong. Characterise the score distribution on a known-good run. Set a control limit from a stated alarm budget. Decide what an out-of-limit reading means operationally — because an anomaly detector says “unlike normal”, not “defective”, and an alarm with no disposition path becomes an alarm that is ignored.

Setting the control limit, worked

Collect scores over a clean run, giving a mean and a standard deviation. The reflexive choice is a Shewhart limit at three sigma. For a normally distributed statistic the tail beyond three sigma is 0.00135 on one side, 0.0027 on both, and the average run length to a false alarm is the reciprocal:

ARL0 = 1 / 0.0027 = 370 samples

at 30 frames per second:
  370 / 30 = 12.3 seconds between false alarms

A false alarm every twelve seconds is an alarm nobody looks at by the first tea break, and this is the actual reason so many pilot systems are quietly disabled. Three sigma is a sensible default when your sampling rate is five measurements an hour. At thirty a second it is not a threshold, it is a noise generator.

Work backwards from the alarm budget instead. Suppose you will tolerate one false alarm per eight-hour shift:

frames per shift = 8 * 3600 * 30 = 864,000
target per-frame false alarm rate p = 1 / 864,000 = 1.16e-6

normal quantile for a one-sided tail of 1.16e-6  ~  4.72 sigma
two-sided (also alarming on unusually low scores) ~  4.86 sigma

So the limit is set by the frame rate and the alarm budget, not by convention. Change the camera to 60 fps and the same budget needs a higher limit; sample one frame per part at two parts a second and it needs a much lower one.

Why EWMA beats a raised threshold

Raising the limit to 4.86 sigma buys quiet at the cost of sensitivity: a defect that shifts the mean score by half a sigma will never trip it. Aggregating over time is the better trade, because it separates a one-frame spike from a sustained change. An exponentially weighted moving average is the standard instrument:

z_t = lambda * x_t + (1 - lambda) * z_(t-1)

steady-state sd of z  =  sigma * sqrt( lambda / (2 - lambda) )

with lambda = 0.1:  sqrt(0.1 / 1.9) = sqrt(0.05263) = 0.2294
so a 3-sigma-equivalent limit on z is at 3 * 0.2294 * sigma
                                       = 0.688 * sigma from the mean

Read what that buys. A single frame four sigma above the mean moves z by only 0.4 sigma and does not trip a limit at 0.688 sigma. A sustained shift of one sigma — a nozzle slowly drifting, a lens gradually fouling, a batch of material with a different finish — drives z towards one sigma and trips it within a few dozen frames. The EWMA finds the small persistent change that no single-frame threshold can see, and ignores the glint off a passing trolley that every single-frame threshold trips on.

The assumption that breaks. Both the run-length arithmetic and the EWMA variance formula assume independent observations. Frames 33 milliseconds apart are nothing of the sort: same part, same lighting, same dust on the lens. The effective number of independent samples per second is far below the frame rate, so the real false-alarm rate is worse than the formula says, and the same applies to the popular “alarm if three consecutive frames exceed” rule — the cube of the per-frame rate is a serious underestimate when consecutive frames are correlated.

The practical correction is to decorrelate before doing any statistics: subsample to one frame per part, or one per 200 milliseconds, whichever is coarser than the autocorrelation length of your score. Measure that length on a clean run rather than assuming it. Then the frame rate in the arithmetic above becomes the sampling rate, and every number gets more forgiving.

Drift, and the recalibration trap

The score distribution moves for reasons that have nothing to do with quality. Daylight through a roof light changes over the shift, the lens fouls, a new lot of raw material has a slightly different finish, a lamp ages. All shift the mean and the spread, and a fixed limit set in January alarms continuously by June.

Recalibrating on a rolling window of recent frames handles that, and introduces the trap: a genuine fault that develops slowly is absorbed into the baseline and never alarms. The system adapts to the defect. Two mitigations, and it is worth having both:

1. Recalibrate only on frames confirmed good by something outside the vision system — parts that passed downstream inspection, or a periodic reference target moved through the field of view. That breaks the circularity.
2. Keep a fixed baseline from a validated clean run alongside the rolling one, and alarm when the two diverge beyond a set amount. That turns the drift itself into a monitored signal instead of an invisible accommodation.

Two operational points to close on. First, log every alarm with its frame and the operator’s disposition. Within a few months that log is the labelled defect dataset you did not have at the start, and it is drawn from exactly the distribution you care about. Second, an anomaly score is not a probability of a defect and should not be presented as one — the mapping from score to likelihood depends on a defect rate the model never saw, which is the same base-rate problem set out in why a classifier that tests well fails in production, and the calibration question is treated in confidence calibration.