Event Stream Processing Architectures Explained

Every design decision in a stream processor comes back to two constraints: a partition is the unit of ordering, and state is the thing that makes a restart expensive. Get the arithmetic on those two right and the rest of the architecture is a consequence.

The log is the architecture

A stream processing system is a partitioned, append-only log with consumers that track offsets into it. In Kafka’s terms, a topic is split into partitions; each partition is an ordered, immutable sequence; a message’s key determines its partition by hash; and a consumer group assigns each partition to exactly one consumer instance. The design consequences follow directly, and they are worth stating rather than assuming.

• Ordering is per-partition, never global. Two events with different keys have no defined relative order, however close their timestamps. If your logic requires that a user.created is processed before a user.updated, both must carry the same key.
• Partition count caps consumer parallelism. A consumer group with more instances than partitions has idle instances. This is the single most common capacity mistake: someone scales the deployment to 32 replicas against 12 partitions and wonders why throughput did not move.
• Repartitioning is not free. Increasing the partition count changes the hash mapping for existing keys, so a key that used to be on partition 3 lands on partition 17 while events for it are still in flight on 3. Any keyed state you were accumulating is now split across two consumers.
• A hot key is a hard ceiling. One key’s traffic cannot be spread. If 40% of your events carry the same tenant id, one partition carries 40% of the load no matter how many you create.

The primary reference for these guarantees is the Apache Kafka documentation, and equivalents hold for Pulsar and Kinesis with different names — Kinesis calls a partition a shard and publishes a per-shard write ceiling that makes the hot-key problem explicit.

A worked throughput calculation

Assume a service producing 120,000 events per second at an average serialised size of 800 bytes, and a processing step that takes 1.2 ms of CPU per event. Everything below is arithmetic from those three stated assumptions; substitute your own and the shape of the answer does not change.

ingest bandwidth   120,000 ev/s x 800 B      = 96 MB/s
                   x 3 replicas                = 288 MB/s written to disk

CPU required       120,000 ev/s x 1.2 ms       = 144 core-seconds per second
                   -> 144 cores fully busy
                   at 60% target utilisation   -> 240 cores provisioned

partitions needed  one consumer thread does 1/0.0012 = 833 ev/s
                   120,000 / 833               = 144 partitions minimum
                   with 2x headroom for skew   -> ~288 partitions

per-partition rate 120,000 / 288               = 417 ev/s
                   417 x 800 B                 = 333 KB/s per partition

Two things fall out of that. The partition count is set by the slowest stage in the topology, not by the ingest rate: if one enrichment step takes 12 ms instead of 1.2 ms because it makes a network call, the same stream needs ten times the partitions or that step needs to be async. And the 60% utilisation target is not padding — a stream processor at 95% CPU cannot catch up after any pause, so lag grows monotonically and never recovers without either shedding load or adding capacity.

The number to alarm on is consumer lag measured in time, not in messages. A lag of 400,000 messages means nothing on its own; a lag of 90 seconds tells you exactly how stale every downstream answer is. Derive it as lag-in-messages divided by current consumption rate, and alert on the derivative as well as the level — lag that is flat at 30 seconds is a system in equilibrium, lag that is rising at 2 seconds per second is a system that will be hours behind by morning.

State is the hard part

A stateless map over a stream is easy and rare. Anything that aggregates, joins, deduplicates or detects a pattern holds state keyed by the partitioning key, and that state has to survive a restart.

The standard design keeps state in an embedded key-value store local to each task — Flink and Kafka Streams both use RocksDB for this — and makes it durable by periodically writing a consistent snapshot to object storage, or by mirroring every change into a compacted changelog topic. Recovery reloads the snapshot and replays the log from the offset the snapshot recorded. Two numbers govern how painful that is: the checkpoint interval, which bounds how much replay is needed, and the state size, which bounds how long the reload takes. A job with 400 GB of state and a 5-minute checkpoint interval can take longer to recover than the outage that caused it.

The corollary is that unbounded state is a latent outage. A join that keeps every key it has ever seen, or a deduplication set with no expiry, works perfectly for six months and then cannot restart. Every piece of keyed state needs a time-to-live, which in practice means every join needs a stated window and every dedup needs a stated retention — the same discipline worked through in log deduplication.

Delivery semantics, precisely

At-most-once commits the offset before processing: a crash loses events. At-least-once commits after: a crash reprocesses them. Exactly-once is not a guarantee that each event is processed once — that is impossible across an unreliable network — but that the observable effects occur once, achieved by making the state update and the offset commit a single atomic transaction.

The important limit is that the guarantee stops at the boundary of the transactional system. A processor writing to Kafka can be end-to-end exactly-once because the offset commit and the output write join one transaction. The same processor calling an external HTTP API cannot, because that call is not in the transaction and will be retried after a failure. The only reliable pattern there is to make the external effect idempotent, usually with a deterministic idempotency key derived from the event, and to accept at-least-once with idempotent sinks as the real design.

Two topology shapes

The Lambda shape runs a batch path and a streaming path over the same input, using the streaming path for a fast approximate answer and the batch path to overwrite it later with an exact one. It is honest about the fact that late data exists, and its cost is that every piece of business logic is implemented twice in two languages and drifts.

The Kappa shape keeps only the streaming path and handles correction by reprocessing: retain the source log long enough that you can replay it from an earlier offset into a fresh output, then swap. This is cheaper to maintain and it puts a hard requirement on retention — you can only reprocess what you kept, so the source retention period is now a correctness parameter, not a storage preference. The trade-off between the two is really a trade-off about where correction lives, and the same question in a simpler setting is covered in batch versus streaming.

Whichever shape you pick, the semantics that decide what your aggregates actually mean are the windowing ones — tumbling, sliding and session windows with watermarks — and they are worth settling before the topology, not after.