Log Anomaly Detection With Machine Learning

Almost every published log-anomaly method is a model of template frequency or template order. Which of the two you pick determines which incidents you can detect at all, and neither one detects the incident where the logs look completely normal.

The shape of the problem

Raw log lines are not a feature vector. The pipeline that makes them one has three stages, and skipping any of them is why generic anomaly scoring underperforms on logs. First, template extraction assigns every line an id. Second, grouping collects lines into the unit you are scoring — a fixed time window, a sliding window, or a session keyed on an identifier such as a block id, a request id or a trace id. Third, encoding turns each group into either a count vector over template ids or an ordered sequence of them.

The grouping choice is not a detail. Session grouping detects failures in the life of one object; time grouping detects failures in the behaviour of the system. A block that gets replicated twice instead of three times is invisible in a per-minute count over a busy cluster and obvious in a per-block session. A cluster-wide latency regression is the other way round. Most production systems need both, keyed differently, running side by side.

Count vectors and the PCA method

The count-vector encoding gives each group a vector of length equal to the number of known templates, holding how many times each fired. Suppose a healthy block session in a distributed filesystem is:

templates            T1  T2  T3  T4  T5   (allocate, receive,
healthy session       1   3   3   1   0     packet-ack, delete, exception)
degraded session      1   2   2   1   0
error session         1   3   3   0   2

The degraded session is the interesting one. Nothing in it is an error — no template fires that should not — but the ratio between T1 and T2 moved from 1:3 to 1:2, which for a system with a replication factor of three means a replica did not land. A threshold on error counts cannot see that. A model of the correlations between counts can.

That is the method Wei Xu and colleagues published at SOSP 2009 in “Detecting Large-Scale System Problems by Mining Console Logs”. Fit PCA to the count vectors of normal sessions, keep the top k components as the “normal subspace”, and score a new session by the squared length of its residual after projection — the part of the vector the normal subspace cannot explain. Sessions whose counts obey the usual proportions have small residuals whatever their absolute size; sessions that violate a proportion do not. The general treatment of that projection step is in principal component analysis, and the broader family is covered in anomaly detection.

The two things to get right are the choice of k and the normalisation. Counts across templates differ by orders of magnitude — a heartbeat fires thousands of times per session, a shutdown once — so an unnormalised PCA models the heartbeat and ignores everything else. Term-frequency weighting, or simply log-scaling the counts, is what makes the rare templates visible to the fit at all.

Sequence models

Count vectors throw away order, and order carries real information: a session that closes a file before writing it is anomalous even though its counts are identical to a healthy one. The sequence approach treats the template id stream as a language and learns to predict the next id from the previous h. DeepLog, published by Min Du, Feifei Li, Guineng Zheng and Vivek Srikumar at CCS 2017 (“DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning”), uses an LSTM for this and applies a rule that is worth stating exactly: the model outputs a probability distribution over the next template id, and the observed id is flagged as anomalous if it is not among the top g most probable candidates.

That top-g rule is the whole tuning surface. Small g means high recall and a false-positive rate that tracks how much legitimate concurrency your logs contain; large g means almost nothing is flagged. The concurrency point deserves emphasis: in a multi-threaded service, interleaving means the “correct” next id is genuinely non-deterministic, and a sequence model trained on an interleaved stream is learning the scheduler as much as the program. Sequence models work best on per-entity sessions, where the interleaving has already been undone by the key.

A first-order Markov chain over template ids gets a surprising fraction of the same benefit at a fraction of the cost, and it is inspectable — you can read the transition matrix and see which transition was improbable. The same construction applied to user behaviour is worked out in clickstream sequence modelling.

Evaluating without labels

You almost never have labelled anomalies. What you have is a period you believe was healthy and a handful of known incidents. That asymmetry dictates the evaluation.

Train on the healthy period and hold out a later healthy period. The alert rate on held-out healthy data is your false-positive rate, and it is the number that decides whether anyone keeps the detector switched on. If a detector scoring one-minute windows fires on 1% of them, that is 14 alerts a day per stream, which is not a detector, it is a pager-fatigue generator. Work backwards from what an on-call person will tolerate — a few per week — and set the threshold there, then ask what recall survives on the known incidents. If the answer is “none”, the encoding is wrong, not the threshold.

Precision on a rare positive class is dominated by the base rate rather than by the model, which is worked through with numbers in security event log anomaly detection. The general metric definitions are in classification metrics.

The four ways this fails in production

• The template set is not stationary. Every deploy can add, remove or reword log statements. A model whose input dimension is “number of known templates” has to handle a dimension appearing at runtime, and the usual answer — reserve an “unseen template” bucket and treat a spike in it as its own signal — is better than retraining in a panic.
• Volume changes are confounded with behaviour changes. Counts scale with traffic. Unless the encoding is normalised by group size or by request count, a marketing campaign and a retry storm look the same.
• The healthy training window contained the problem. Slow degradations get learned as normal. This is the failure that makes people distrust the whole approach, and the only defence is periodically scoring the training window against a model fit on an earlier one.
• Silence is the anomaly. The most severe incidents often reduce log volume: the component that would have logged is dead, or the log shipper is the thing that broke. A detector that only scores what arrived cannot see an absence. Pair it with an explicit heartbeat expectation per source, which is the cheapest high-value detector in this whole area and almost never the one people build first.