Detecting Duplicate Alerts Across Monitoring Tools

One disk fills up and three phones buzz: the infrastructure monitor says a filesystem is at 95%, the APM says a service’s error rate is up, and the synthetic check says a URL returned 500. Collapsing those into one alert is an entity-resolution problem, not a string-matching one.

Three tools, one failure

Deduplicating alerts within a single tool is nearly free, because the tool controls its own vocabulary. Prometheus Alertmanager, for example, identifies an alert by the fingerprint of its label set, so two instances of the same alert from two redundant Alertmanager peers are recognised as identical without configuration — and its group_by mechanism collapses distinct-but-related alerts into one notification, with a documented group_wait of 30 seconds before the first send and group_interval of 5 minutes before an update (see the Alertmanager configuration reference).

Across tools none of that applies. Each product has its own identifier, its own severity scale, its own idea of what the affected entity is, and its own timestamp semantics. There is no shared key, so the join has to be constructed. That construction is the whole of this page, and it has three parts: normalise the entity, normalise the condition, correlate in time.

What a fingerprint is

A fingerprint is a deterministic hash over a chosen subset of an alert’s fields, designed so that two alerts describing the same thing produce the same value. Everything depends on choosing the subset, and the two mistakes are symmetrical.

Include too much and nothing ever matches: a fingerprint over the whole payload includes the alert id, the timestamp and the message text, so it is unique by construction and does nothing. Include too little and unrelated failures merge: a fingerprint over the host alone collapses “disk full” and “certificate expired” on the same box into one alert, and somebody fixes one and closes both.

The useful subset is (normalised entity, normalised condition class). Normalising the entity means resolving whatever each tool identifies — a hostname, a Kubernetes pod name, a container id, an IP, a service name, an ARN — to a canonical id through an inventory or CMDB lookup. This is the part that is real work and the part people skip. A pod name like api-7d9f8c-x2k4l and a service name like api and a host like ip-10-0-3-14.ec2.internal are three views of one thing, and no amount of string similarity connects them; only a lookup does. Normalising the condition means mapping each tool’s alert name to a small controlled vocabulary — disk_pressure, error_rate, latency, availability, cert_expiry — which is a mapping table you maintain, typically a few dozen rows.

Collapsing the three

incoming alerts

A  source: infra-monitor
   host: ip-10-0-3-14.ec2.internal
   check: "disk_used_pct"   value: 96
   severity: "warning"      ts: 14:02:11Z

B  source: apm
   service: "checkout-api"   env: "prod"
   condition: "Error rate > 2% (5m)"
   severity: "P2"            ts: 14:03:47Z

C  source: synthetic
   url: "https://shop.example.com/checkout"
   condition: "HTTP 500"
   severity: "critical"      ts: 14:04:02Z

normalisation

A  entity: svc:checkout-api   (host -> instance -> service, via inventory)
   class:  disk_pressure      severity: 3

B  entity: svc:checkout-api   (already a service)
   class:  error_rate         severity: 2

C  entity: svc:checkout-api   (url -> route -> service, via routing table)
   class:  availability       severity: 1

fingerprints

A  sha1("svc:checkout-api|disk_pressure")  = 4f1c...
B  sha1("svc:checkout-api|error_rate")     = a087...
C  sha1("svc:checkout-api|availability")   = 9d33...

exact fingerprint match collapses nothing — three distinct classes.

entity-level correlation, 10-minute window

  all three share entity svc:checkout-api within 111 s
  -> one incident, three signals
     title    = highest-severity signal (C, availability)
     cause    = earliest signal        (A, disk_pressure at 14:02:11)
     severity = max(1, 2, 3) = 1
     notified = once

The important result there is that fingerprint equality was not what did the work. Fingerprint equality catches genuine repeats — the same tool re-firing, two collectors both reporting the same check — and it should run first because it is exact and cheap. Cross-tool collapse is a different operation: a correlation on the shared normalised entity within a time window, producing an incident that contains several distinct alerts rather than discarding two of them.

Discarding is the mistake to avoid. The disk-pressure alert is the one that tells you what to fix, and it is the lowest severity and the one a naive “keep the most severe” rule throws away. Keep every signal, notify once, and order the contained signals by timestamp so the earliest — usually the cause — is at the top.

The correlation window

The window is the parameter that decides how wrong this gets. Too short and a slow cascade arrives as separate incidents: a disk fills at 14:02, the service degrades at 14:09, and a five-minute window separates them. Too long and unrelated failures on a busy service merge all day, which produces an “incident” that is really a shift report and that nobody can close.

Two refinements make it behave better than a fixed number. First, make the window trailing rather than fixed: keep an incident open while signals keep arriving and close it after a quiet period, which is exactly a session window over alerts and inherits its merge behaviour — a late signal can join two incidents you already published, so your incident store needs to support merging, not just appending. Second, weight by topology: two alerts on entities with a known dependency edge should correlate over a longer window than two unrelated entities, because a cascade takes time to propagate and an unrelated coincidence does not become more likely with delay.

Timestamp trust is a practical trap here. Tools disagree about whether the timestamp is when the condition started, when the rule evaluated, or when the notification was sent, and those can differ by the whole for duration — several minutes, as worked out in incident detection from a log stream. Ordering signals by received time and calling the first one the cause will systematically name whichever tool has the shortest evaluation delay. If a tool exposes a condition-start time, use it and record which field you used.

When collapsing is the wrong answer

• Different blast radius. One alert about one pod and one about the whole service look correlated and mean different things. Collapsing them can turn a service-wide outage into a single-instance ticket. Carry the scope as part of the normalised entity — instance: versus svc: — and let a broader scope absorb narrower ones rather than the reverse.
• A shared dependency, not a shared cause. Forty services alerting within a minute is one incident if they share a failed database and forty if a deploy broke them independently. No time-window rule distinguishes those; only the dependency graph does. When you do not have one, err toward collapsing by time and showing the list, so a human sees forty names and decides.
• Suppression that outlives the cause. A collapsed incident that stays open suppresses new alerts on the same entity, so a second, unrelated failure during a long incident is silent. Always allow a new alert class to re-notify even inside an open incident, and expire incidents on a hard ceiling regardless of activity.

The evaluation to run before trusting any of this: take a month of historical alerts and the incident records humans actually created, replay the rules, and count how many real incidents were split and how many unrelated pairs were merged. Both errors are visible in that comparison and neither is visible in production, where a merged pair looks exactly like a quiet night.