Windowing Strategies for Streaming Aggregation

A window is a rule for cutting an unbounded stream into finite pieces you can aggregate. Which rule you pick changes the number, and two of the three common rules make it impossible to sum the outputs without counting some events more than once.

Two clocks, and why it matters

Every event carries two timestamps. Event time is when it happened, recorded at the source. Processing time is when the stream processor got to it. They differ by the network, by queueing, by a mobile client that was offline for two hours, and by whatever backlog the consumer is working through.

Windowing by processing time is trivially easy and produces results that are not reproducible: replay the same input and the boundaries fall in different places, so the answer changes. Windowing by event time is reproducible — the same input always yields the same windows — and introduces the only genuinely hard problem in this subject, which is deciding when a window is finished, given that you cannot know whether another event with an earlier timestamp is still on its way. The conceptual framework everyone now uses for this comes from Akidau et al., “The Dataflow Model” (VLDB 2015), and it is implemented with the same vocabulary in Apache Beam and Apache Flink.

The three assigners on one stream

Take one user’s events, with event-time offsets in seconds from a round minute, and count them.

event   A    B    C    D    E    F    G
t (s)   2    9    35   62   68   240  246

Tumbling windows of 60 seconds assign each event to exactly one window, by floor(t / 60). Windows do not overlap and there are no gaps.

[0,60)    A B C        count 3
[60,120)  D E          count 2
[120,180) —            count 0   (or no output, depending on the runner)
[180,240) —            count 0
[240,300) F G          count 2

sum of window counts = 7 = number of events

Sliding windows of size 60 seconds with a slide of 30 seconds start a new window every 30 seconds, each covering the previous 60. Apache Flink states the mechanism plainly: sliding windows overlap when the slide is smaller than the window size, and in that case elements are assigned to multiple windows.

[-30,30)  A B          count 2
[0,60)    A B C        count 3
[30,90)   C D E        count 3
[60,120)  D E          count 2
[90,150)  —            count 0
...
[210,270) F G          count 2
[240,300) F G          count 2

sum of window counts = 14 = 2 x number of events

Session windows have no fixed boundaries at all. They are defined by a gap: events are merged into the same session while consecutive events are closer together than the gap, and a period of inactivity longer than the gap closes it. With a 60-second gap:

A(2) B(9) C(35) D(62) E(68)   gaps 7, 26, 27, 6  all < 60  -> one session
E(68) -> F(240)               gap 172 > 60             -> session closes
F(240) G(246)                 gap 6                    -> second session

session 1  [2, 68]    count 5   duration 66 s
session 2  [240, 246] count 2   duration  6 s

sum of session counts = 7 = number of events

Three assigners, one stream, three different sets of numbers, all correct. The tumbling answer says “three events in the first minute”. The sliding answer says “the busiest 60-second stretch contained three events”. The session answer says “there were two bursts of activity, one of five events lasting 66 seconds”. Only the third one can tell you a session length, and only the first two can be laid out on a fixed time axis.

Exactly what each one double-counts

• Tumbling windows double-count nothing. Each event is assigned to exactly one window, so the window outputs partition the stream and summing them reproduces the total exactly. What tumbling windows get wrong instead is the boundary: an event at t = 59 and one at t = 61 are two seconds apart and land in different windows, so any pattern that straddles a boundary is split. A burst of ten events spanning 58–62 seconds appears as two windows of five, and a threshold of eight fires on neither.
• Sliding windows double-count by exactly size ÷ slide. Each event falls inside every window that covers its timestamp, and with size 60 and slide 30 that is 60/30 = 2 windows. Size 60 slide 10 gives 6. Size 300 slide 1 gives 300. This factor is not an approximation, it is the definition, and it is why summing sliding window outputs to get a daily total is always wrong by that factor. It also means the storage and compute cost of a sliding aggregation is size/slide times that of the equivalent tumbling one, which is the real reason to think before setting a one-second slide on a five-minute window.
• Session windows double-count nothing, but they merge. A session window is not decided until the gap has elapsed, and a late event arriving in the middle of what looked like a gap can join two previously separate sessions into one. So a runner that emits early results has to be able to retract the two sessions it already published and emit one merged one. If your downstream sink cannot handle a retraction — if it appends rows to a table — the two old sessions and the merged session all exist, and now you have triple-counted the events in both.

The practical rule that falls out: aggregate on tumbling windows when the number has to be summable, use sliding windows only for “rate over the last N” questions where you read one window at a time and never sum them, and use sessions only where your sink supports upserts keyed on a session id.

Watermarks

A watermark is the processor’s claim that no event with an event time earlier than W will arrive from now on. It is an assertion, not a fact, and it is usually generated heuristically: take the maximum event time seen so far and subtract a fixed out-of-orderness bound, or track a percentile of observed lateness.

When the watermark passes the end of a window, the window fires and its aggregate is emitted. That single sentence contains the entire trade-off. A conservative watermark — maximum minus five minutes — is almost never wrong but adds five minutes of latency to every result. An aggressive one — maximum minus two seconds — gives near-real-time output and drops the tail of stragglers. There is no setting that is both fast and complete, because the information required to be complete has not arrived yet.

The failure that surprises people: a watermark derived from the maximum observed event time stalls when a partition goes idle. One silent source with no events means its watermark never advances, the global watermark is the minimum across sources, and every window in the job stops firing while the others keep producing data. Runners handle this with an idleness timeout that lets a quiet partition be excluded from the minimum, and it is worth checking whether yours is configured, because the symptom — a pipeline that is healthy by every metric and emits nothing — is very hard to diagnose from first principles.

Late arrival and what you do about it

An event whose event time is earlier than the current watermark is late. Flink’s documentation is explicit that allowed lateness “specifies by how much time elements can be late before they are dropped, and its default value is 0” — so on a default event-time job, every straggler is silently discarded, and the only way you learn the rate is to route them somewhere with a side output.

You have three options and they are all defensible. Drop is right when the aggregate is a monitoring signal and a 0.2% undercount is irrelevant. Allow lateness and re-fire keeps the window state alive past the watermark and emits an updated result each time a late event lands; this is correct but means downstream sees several results for one window, and whether that is safe depends entirely on the accumulation mode. In accumulating mode each firing restates the running total, so a downstream sum over firings triple-counts; in discarding mode each firing carries only what is new since the last, so downstream must sum and must not overwrite. Getting this backwards is the most common source of a streaming number that is quietly two or three times too large. Side-output the late events to a separate stream and reconcile in batch, which is the Lambda answer and the only one that gives you both a fast number and an eventually exact one.

Whatever you choose, instrument it. The one metric that makes windowing debuggable is a histogram of (processing time − event time) at ingest, because every parameter on this page — the out-of-orderness bound, the allowed lateness, the session gap — is a percentile of that distribution, and without it every value is a guess.