Detecting Anomalous Patterns in Security Event Logs

Security log detection fails for a reason that has nothing to do with the model: the thing you are looking for is astronomically rare, and rarity destroys precision faster than any amount of accuracy can rescue it. Every design decision here follows from that arithmetic.

The event sources and what they say

Authentication and process events carry most of the signal, and they are specific and checkable rather than generic “security logs”. On Windows, the security event log records successful logons as event ID 4624 and failed logons as 4625, each carrying a logon type that matters more than the event id itself — type 2 is interactive at the console, type 3 is over the network, type 5 is a service, and type 10 is remote interactive, which is what an RDP session produces. Event ID 4672 records that special privileges were assigned to a new logon, which is the closest thing to an “administrator logged in” event, and 4688 records process creation. Sysmon, documented by Microsoft at Sysinternals, adds richer process creation (event ID 1) with command lines and hashes, and network connections (event ID 3) attributed to a process.

On Linux the equivalents are auditd records and the authentication messages from sshd and PAM, plus process execution via the audit subsystem’s execve rules. The detection logic is the same; only the field names change. What matters is that in both cases you have three primitives: who authenticated, from where and how; what process ran, with what command line, under what parent; and which host talked to which.

Three signal shapes

Authentication shape. Brute force and password spraying are both failed-authentication signals with opposite geometries, and a detector tuned for one misses the other completely. Brute force is many failures against one account from few sources — a spike in 4625 with a single target and a high failure-to-success ratio. Password spraying is few failures against many accounts, often one or two attempts per account to stay under lockout thresholds, so no single account’s counter ever looks unusual and the signal only exists when you aggregate by source and count distinct targets. MITRE catalogues both under T1110, Brute Force. The general lesson generalises past security: whenever an adversary can choose the aggregation key that hides them, you need detectors on more than one key.

Rarity of a process or a parent-child pair. Most hosts run a small, stable set of executables. A binary never seen before on this host, or a familiar binary with an unfamiliar parent — a shell spawned by a web server process, a scripting host spawned by an office application — is a high-value, low-volume signal. Implement it as a per-host and per-fleet frequency table over (parent image, child image) pairs and score by how rare the pair is fleet-wide and whether it is new on this host. This is the same “first occurrence” detector that is the highest-value cheap signal in general log anomaly detection.

Access-graph novelty. Lateral movement shows up as edges that did not exist before: this account has never logged into that host, this host has never connected to that subnet. Maintaining a rolling set of observed (account, host) and (host, host) pairs and alerting on new edges catches a broad class of activity, and it is resilient in a way that volume-based detection is not, because a single quiet connection is exactly as novel as a noisy one. Valid stolen credentials — MITRE’s T1078, Valid Accounts — produce no failed logons at all, so this is often the only signal available.

The base-rate arithmetic

Assume a fleet of 10,000 hosts, a detector evaluated once per host per day, a true-positive rate of 99%, and a false-positive rate of 0.1% per host-day. Assume further that two hosts are genuinely compromised in a year. These are stated assumptions; the point is the shape of the result, and it holds for any plausible substitution.

host-days per year   10,000 hosts x 365 = 3,650,000

false positives      3,650,000 x 0.001      = 3,650 per year
                                            = 10 per day

true positives       2 compromises x 0.99   = 1.98 per year

precision            1.98 / (1.98 + 3,650)  = 0.00054  = 0.054%

alerts an analyst reviews per real detection = 1 / 0.00054 = 1,844

Nineteen hundred false alerts per genuine one, from a detector that is 99% sensitive and 99.9% specific. Nothing is wrong with the detector. The base rate is roughly one compromise per 1.8 million host-days, and no achievable false-positive rate makes a single-stage detector usable against that.

Which is why real detection is staged rather than tuned. Suppose three weakly-dependent detectors each with a 0.1% false-positive rate and each with 90% sensitivity, and require two of the three to agree within a window. If their false positives were independent, the pair rate falls toward 10⁻⁶ per host-day while sensitivity only falls to about 0.97 — precision improves by roughly three orders of magnitude. The independence assumption is the weak point and is usually optimistic, since detectors sharing an input share their errors, but the direction is right and it is the only lever with the necessary size. Raising a single threshold moves precision by a factor of two; requiring corroboration moves it by a factor of a thousand.

Baselining without a clean period

Every method above needs a notion of “normal”, and the uncomfortable fact is that you cannot prove your baseline period was uncompromised. An adversary present during the learning window becomes part of the baseline, and the detector will never flag them again.

Three partial defences. Baseline per-host but score against the fleet as well, so behaviour that is normal for one host and unique across 10,000 still surfaces. Prefer signals that are structurally rare rather than statistically rare — a first-ever (account, host) edge is meaningful even in a poisoned baseline, because the edge either existed in the observed history or did not. And re-derive baselines periodically from a rolling window rather than freezing one, then diff consecutive baselines: a stable set of processes that gained a member three months ago is a question worth asking retrospectively, and nothing in a real-time detector will ever ask it.

Assume the adversary reads your detector

This is what separates security log analysis from every other subject in this cluster. The distribution is not merely non-stationary, it is adversarial: a threshold of five failed logons per minute produces four failed logons per minute, and a detector keyed on process name produces a renamed binary. Detection logic is a specification of what to avoid.

The practical consequences are unglamorous. Prefer signals tied to things the adversary needs rather than things they choose — a process must make network connections to exfiltrate, whatever it is called, and an account must authenticate somewhere to move. Randomise thresholds within a band rather than publishing a round number. Watch for the absence of expected telemetry as its own detection, since disabling logging is itself a technique and a host that stopped reporting is a more urgent signal than most of what it would have reported. And accept that the value of these detectors is measured in the analyst hours they survive, which brings the whole design back to the base-rate arithmetic above rather than to any property of the model.