messagingestablished · high operational burden

Stream Processing

Also known as: event-stream-processing, continuous-processing

Intent

Continuously compute over unbounded event streams using windows, state and event-time semantics.

Problem

Batch jobs deliver insights hours late, which is too slow for fraud detection, alerting or live personalization. Events arrive out of order and late, so naive counting gives wrong answers.

Forces

•Low latency versus completeness of late data
•Stateful computation versus fast recovery
•Throughput versus ordering guarantees
•Exactly-once effects versus sink capabilities

Solution

Overview

Stream processing treats data as an unbounded sequence and computes results incrementally as events arrive. It replaces periodic full recomputation with always-on operators that keep state per key and emit updates.

How it works

Events flow from a partitioned log into operators: filter, map, join, aggregate. Aggregations run over windows: tumbling (fixed, non-overlapping), sliding, or session windows based on inactivity gaps.

orders topic --> keyBy(customer) --> window(5 min, event time)
            --> sum(amount) --> sink (table / topic)
watermark = max_event_time - 30s   (window closes when watermark passes its end)

Event time is the timestamp inside the record, not arrival time. The watermark is the engine's estimate that no earlier events remain. Events later than allowed lateness go to a side output for repair. Periodic checkpoints snapshot operator state and input offsets, so recovery resumes without loss or duplication inside the engine. End-to-end exactly-once additionally needs transactional or idempotent sinks.

Pitfalls

Using processing time for business windows, which gives different answers on replay.
Unbounded state: joins and distinct counts without TTL exhaust memory or disk.
Hot partitions: a few popular keys serialize on one task; salt keys or pre-aggregate.
Changing the job graph without a state migration plan; savepoints and stable operator ids matter.
Ignoring backpressure signals until lag is hours.

Key takeaways

Choose event time and define lateness policy deliberately.
State plus checkpoints give fault tolerance, sinks decide end-to-end guarantees.
Monitor consumer lag, watermark delay and state size.
Use batch when freshness is not a requirement.

✓ When to Use

•Results needed in seconds or less
•Per-key running state such as sessions or counters
•Continuous enrichment or anomaly detection

✗ When Not to Use

•Daily reports where batch is simpler and cheaper
•Logic needs full dataset scans each time
•Team lacks capacity to operate stateful jobs

Pros

+Low-latency insight and reactions
+Handles continuous high volume
+Incremental computation avoids recompute

Cons

−State, watermarks and lateness are hard to reason about
−Debugging and testing are harder than batch
−Always-on cluster cost

Cost Profile

Infrastructure

Medium - always-running cluster and state storage

Operational

High - tuning, upgrades, state migration

Cognitive

High - time semantics

Failure Modes

Skewed keys overload one partition
Watermark too aggressive drops late events
State grows unbounded without TTL

Real-World Examples

Alternatives

Related Patterns

Competency Domains

data statedistribution communication