Skip to content
Multigrid

Logs and event streams

Parsing, clustering, windowing and burst detection over high-volume machine output.

A log line is written for a human and then read by a machine, and almost every difficulty in this subject comes from that mismatch. The line has no schema. Its variable parts — an IP address, a request id, a duration in milliseconds — sit inline in a sentence, so two lines that describe the same event are byte-different, and the first job of any pipeline that wants to count something is to recover the sentence they have in common. That job has a name, template extraction, and a standard algorithm, and everything downstream — clustering, anomaly scoring, deduplication, alert correlation — is arithmetic over its output.

The second difficulty is time. Events are produced in one order, transported in another, and need to be aggregated in a third. A stream processor that aggregates by arrival time is easy and quietly wrong; one that aggregates by event time has to decide how long to wait for stragglers, and that decision is a watermark. These pages work through both halves with the real parameter names and the real arithmetic: what Drain’s similarity threshold changes, what a sliding window double-counts and by exactly what factor, and what a stated events-per-second rate costs to keep for ninety days.

Log Parsing and Template Extraction Explained

How an online parser turns free-text log lines into a small set of reusable templates, worked through Drain's fixed-depth tree.

10 min read

Log Anomaly Detection With Machine Learning

How template-frequency and template-sequence models find abnormal system behaviour in logs, and what each one cannot see.

10 min read

Clustering Log Messages for Faster Triage

A runnable script that collapses a pile of log lines into a ranked list of distinct message shapes using edit distance, with blocking so it finishes.

9 min read

Event Stream Processing Architectures Explained

How partitioning, keyed state and checkpointing fit together in a stream processor, with a worked throughput and partition-count calculation.

11 min read

Windowing Strategies for Streaming Aggregation

Tumbling, sliding and session windows produce different answers from one event stream — here is the arithmetic, including exactly what each one double-counts.

11 min read

Reconstructing a User Session From Clickstream Events

How inactivity timeouts turn a flat event table into sessions, worked on a real sequence, plus the boundary cases that make the count wrong.

10 min read

Detecting a Burst of Events in a Stream

Deriving a burst threshold from a measured baseline rate and its variance, why the Poisson assumption usually fails, and what Kleinberg's automaton does instead.

10 min read

Detecting an Incident From a Log Stream Before an Alert Fires

Where the minutes go between a failure starting and a threshold alert paging someone, and how much of that a log-pattern detector actually recovers.

10 min read

Deduplicating Noisy Log Lines Before Indexing

How template-plus-window deduplication collapses repeated lines before they reach an index, and what information the collapse destroys.

9 min read

Detecting Duplicate Alerts Across Monitoring Tools

How to collapse one underlying failure reported by three different monitoring systems into a single alert, using normalised fingerprints and a correlation window.

9 min read

Real-Time Fraud Detection From Event Streams

How windowed velocity features are computed on a stream, why point-in-time correctness decides whether the model works in production, and what the label delay does to evaluation.

10 min read

Change Data Capture for Feeding an AI Pipeline

Log-based versus query-based CDC, with the latency arithmetic for both and the operational hazards of replication slots, deletes and schema changes.

10 min read

Detecting Anomalous Patterns in Security Event Logs

The classical log signals for intrusion — authentication shape, rare processes, lateral movement — and the base-rate arithmetic that decides whether a detector is usable.

10 min read

Modeling Clickstream Sequences to Predict the Next Action

Building a first-order Markov transition matrix from click sequences by hand, what smoothing does to it, and where the memoryless assumption stops being adequate.

10 min read

Forecasting Log Volume for Capacity Planning

Deriving ingestion, storage and cost growth from a stated log-volume trend, and finding the date a fixed pipeline capacity runs out.

9 min read

What It Costs to Store and Query High-Volume Event Logs

A full cost derivation from a stated events-per-second rate, retention period and per-GB and per-TB-scanned prices, showing why query cost usually beats storage cost.

10 min read

Other topics