Anomaly Detection in Production Data

Anomaly detection is not limited by its ability to find anomalies. It is limited by how many false alarms a human will tolerate before muting the channel, and that number is derivable from your sampling rate and your threshold before you write any code. On 200 metrics sampled every minute, a three-sigma rule produces 778 alerts a day.

The constraint is the alert budget

Every other machine learning task has labels and an accuracy question. This one usually has neither: nobody labelled which minutes were anomalous, the anomalies you care about are by definition rare, and the cost of a miss and the cost of a false alarm are wildly different and both unknown. That absence of labels is also what separates this from drift detection, which asks a related question about a distribution rather than about a point.

What is known is the budget. A team on call will act on perhaps three to five alerts a day before it starts closing them unread, and an alert channel that is ignored is worse than no channel, because it creates the belief that something is watching. So the design question is not “which method detects best”. It is “which method fits in five alerts a day and still catches the thing that matters”.

The arithmetic of a threshold

Assume one metric sampled every minute: 1,440 points a day. Assume the normal behaviour is roughly Gaussian — which it is not, and that matters later, but it gives the optimistic bound. A two-sided k-sigma rule fires on the tail probability of the normal distribution.

k     P(|z| > k)      alerts/day, 1 metric     alerts/day, 200 metrics
2      0.0455          65.5                     13,100
3      0.0027           3.89                        778
4      6.33e-5          0.091                        18.2
5      5.73e-7          0.000825                      0.165

  1,440 samples/day x P(|z| > k) = alerts/day for one metric
  x 200 metrics = the number that reaches the on-call channel

Three sigma — the default in most tutorials and most dashboards — produces 778 pages a day across a modest fleet of metrics, every one of them a false alarm by construction, before anything has actually gone wrong. Four sigma still produces eighteen. Only at five sigma does the noise floor drop below one alert a week, and a five-sigma rule will miss most real incidents, which rarely move a metric that far.

The two levers that fix this are both cheap.

PERSISTENCE. Require k consecutive points beyond the threshold.
  3 consecutive 3-sigma points, under independence:
    0.0027^3 = 1.97e-8 per position
    x 1,440 positions x 200 metrics = 0.0057 alerts/day
    = one false alarm roughly every 175 days

  Independence is false for real metrics -- consecutive minutes are
  correlated -- so treat this as a lower bound, not a promise. The
  direction is right and the magnitude is enormous.

AGGREGATION. Score 5-minute means instead of raw minutes.
  Points per day:  1,440 -> 288        (5x fewer chances to fire)
  Noise sd:        s -> s/sqrt(5)      (a real shift stands out more)
  Combined effect at fixed sensitivity: roughly an order of magnitude
  fewer false alarms, at the cost of up to 5 minutes of detection delay.

Four methods, ordered by what they cost you

The ordering below is derived from each method’s own default settings and the arithmetic above — not from anyone’s incident history. Recompute it with your own metric count and sampling rate; the shape does not change.

4. Raw threshold on a seasonal series — structurally unusable

A fixed bound, or a z-score against the all-time mean, applied to a metric with a daily cycle. Traffic at 14:00 is genuinely four sigma above the 24-hour mean, so the detector fires every afternoon, forever. The alert rate here is not a probability; it is one per cycle per metric, guaranteed — 200 a day and always the same 200. Every team builds this first and every team turns it off within a fortnight.

3. Isolation Forest at default contamination — 2,880 a day

A genuinely good multivariate method: it isolates points by random splits, and points that isolate in few splits are outliers. It handles many correlated metrics at once, which the univariate methods cannot.

The trap is contamination. It is not an estimate of how many anomalies exist; it is an instruction about what fraction of the input to flag. Scikit-learn’s default of "auto" and the widely copied contamination=0.01 both mean the same thing in practice: one per cent of your data becomes an alert.

contamination = 0.01 on 1,440 points/day = 14.4 alerts/day/metric
x 200 metrics = 2,880 alerts/day

The model has no opinion about how many anomalies there are.
You told it. Set contamination to your alert budget divided by
your sample count, and it becomes a usable tool:

  budget 3 alerts/day over 200 metrics x 288 five-minute points
  = 3 / 57,600 = 0.000052

2. Robust z-score on the deseasonalised residual — 778 a day at 3σ

Decompose the series into trend, seasonal and residual, then score only the residual. This removes the structural failure of method 4 and brings the alert rate back to the honest tail probability of the noise. It is a univariate method, so it needs a rule per metric, but the rule is one line and it is interpretable in the incident channel.

1. Deseasonalised residual, robust scale, persistence — under one a week

The same method as 2, with the two levers applied: score five-minute aggregates, use a median-absolute-deviation scale rather than a standard deviation, and require three consecutive breaches at 4σ. The derived false-alarm rate falls below one a week across the whole fleet, and it still catches any shift that persists for fifteen minutes — which is every incident anyone would want to be woken for.

Robust scale, because the mean is contaminated

The arithmetic above assumed you know σ. In practice you estimate it from history — and the history contains the anomalies you are trying to find, which inflates the estimate and hides the next one. One large spike in the training window can raise a standard deviation enough to silence the detector for a month.

The median absolute deviation does not have this problem. It ignores anything past the middle of the distribution by construction, and it is rescaled to be comparable to a standard deviation by a constant that is derivable: for Gaussian data the MAD equals 0.6745σ, so σ ≈ MAD / 0.6745 = 1.4826 × MAD.

import numpy as np

def robust_z(x: np.ndarray) -> np.ndarray:
    med = np.median(x)
    mad = np.median(np.abs(x - med))
    scale = 1.4826 * mad
    if scale == 0:                    # a constant or near-constant series
        return np.zeros_like(x, dtype=float)
    return (x - med) / scale

Building the one that works

import numpy as np
import pandas as pd
from statsmodels.tsa.seasonal import STL

def detect(series: pd.Series, period: int, k: float = 4.0,
           persist: int = 3) -> pd.Series:
    """Flag points where the deseasonalised residual breaches k robust
    sigmas for persist consecutive samples. series must have a regular
    DatetimeIndex; period is the number of samples per cycle."""
    stl = STL(series, period=period, robust=True).fit()
    resid = stl.resid

    med = np.median(resid)
    mad = np.median(np.abs(resid - med))
    scale = 1.4826 * mad
    if scale == 0:
        return pd.Series(False, index=series.index)

    z = (resid - med) / scale
    breach = z.abs() > k
    # True only where this point and the previous persist-1 are all breaches
    run = breach.rolling(persist).sum() == persist
    return run.fillna(False)

# 5-minute aggregates of a metric with a daily cycle: 288 samples/day
agg = raw.resample("5min").mean().interpolate()
alerts = detect(agg, period=288, k=4.0, persist=3)
print(agg.index[alerts])

1. Aggregate first. Choose the window from how long you are willing to wait to be told. Five minutes is usually right; one minute almost never is.
2. Remove the seasonality you know about. Daily and weekly cycles at minimum. If the metric has both, STL on the daily period plus a day-of-week term in the residual model.
3. Fit the scale on a clean window. A period you know was quiet, refreshed on a schedule. Refitting on a rolling window that includes the current incident is how a detector learns to accept the outage as normal.
4. Require persistence. Three consecutive breaches, derived above as the difference between one alert every 175 days and 778 a day.
5. Group before paging. When one upstream failure moves forty metrics, forty alerts is not forty pieces of information. Collapse by service and send one.
6. Route by consequence, not by score. A four-sigma move in a metric nobody acts on belongs in a weekly digest. This is the same decision as picking a threshold from costs, and the same arithmetic applies.

Evaluating without labels

You cannot compute precision without knowing which alerts were real, so build the label set as you go. Every alert gets a disposition from whoever looked at it: real and actioned, real and ignorable, false. After a month you have a labelled set and the detector becomes an ordinary supervised problem you can tune with precision and recall.

For recall — the alarms you did not get — the only honest source is the incident log. Take every incident from the last quarter, replay the detector over the metrics from that window, and count how many it would have caught and how early. That number is worth more than any benchmark, because it is measured on the failures your system actually has.