Streaming Pills Fundamentals
A series for data engineers who want to understand streaming from first principles, not as "batch, but faster." Synthesized from Martin Kleppmann's Designing Data-Intensive Applications and two production write-ups on stream processing. Each pill starts from a real question.
Series map
flowchart TD
subgraph F["Foundations"]
direction LR
P0["0 · Batch = Special Case"] --> P1["1 · Events vs Messages"] --> P2["2 · Two Clocks"]
end
subgraph L["Log-Based Processing"]
direction LR
P3["3 · Topics vs Queues"] --> P4["4 · Micro-batching Fails"]
end
subgraph S["State & Time"]
direction LR
P5["5 · Stateful Engines"] --> P6["6 · Windows"] --> P7["7 · Late Events"]
end
subgraph D["Design Decisions"]
direction LR
P8["8 · Joins"] --> P9["9 · Three Questions"]
end
F --> L --> S --> D- 00
Streaming Pill 0: Batch Is a Special Case of Streaming
Most engineers learn streaming as batch, but faster. Kleppmann flips that model in Designing Data-Intensive Applications: batch is the special case, streaming is the default. This pill sets up the whole series.
- 01
Streaming Pill 1: Events Are Not Messages
A message tells one consumer what to do. An event tells everyone that something happened. Mixing the two up is how a streaming pipeline turns into a distributed RPC system.
- 02
Streaming Pill 2: Event Time and Processing Time Are Different Clocks
Every event carries two timestamps: when it happened, and when your system saw it. Only one of them is trustworthy for correctness, and picking the wrong one silently corrupts your aggregates.
- 03
Streaming Pill 3: A Topic Is Not a Queue
A queue deletes a message the moment it is read. A Kafka topic keeps every event on disk and lets many independent consumers replay the same history. That difference is why log-based systems changed streaming, and it mirrors how a database's write-ahead log works.
- 04
Streaming Pill 4: Micro-Batching Is Not Streaming
A SQL script that runs every 10 minutes looks like streaming from a distance. Three things break that idea: which events belong to which run, the growing cost of recomputing state, and how long the business can wait.
- 05
Streaming Pill 5: State Is the Real Bottleneck
Lambda functions have no memory between invocations, so computing anything over time forces a round trip to an external database on every single event. Stateful engines like Flink solve this by keeping a key-value store next to the computation itself.
- 06
Streaming Pill 6: Windows Are a Design Decision, Not a Data Property
A stream has no start or end, so any window you compute over it, tumbling, sliding, or session-based, is a choice about memory, accuracy, and latency, not something you discover in the data.
- 07
Streaming Pill 7: What to Do With a Late Event
Networks fail and phones lose signal, so late events are guaranteed to happen, not a rare exception. Every streaming pipeline needs an explicit policy: drop the event and log it, or reopen the window and correct the result.
- 08
Streaming Pill 8: Streaming Joins Are a Memory Problem
In batch, a join is one SQL statement. In streaming, a join is a promise to hold data in memory for a period of time, because streams never end and you cannot join against everything.
- 09
Streaming Pill 9: Three Questions Before You Design a Streaming System
Before reaching for Flink or Kafka Streams, three questions decide whether you need a stateful streaming engine at all: is the pipeline stateful, what is your tolerance for late data, and can you trust the clocks on your source events.