Stream Processing Design

Streaming pipelines process unbounded data continuously. Their central difficulty is that the world delivers events late, twice, and in the wrong order, so correct designs make time, state, and recovery explicit.

How it works

Events carry an event timestamp. Windows group events by event time, such as tumbling, sliding, or session windows. A watermark estimates how far event time has progressed, and a window closes when the watermark passes its end plus an allowed lateness. State, such as running aggregates and join buffers, is kept per key and snapshotted periodically with source offsets, so a restart resumes from a consistent point.

Implementation steps

1. Define the questions, freshness target, and tolerance for approximate or revised results.
2. Choose keys that spread load and keep related events together, and set partition counts with growth in mind.
3. Assign event timestamps at source and pick watermark strategy, based on measured out-of-order delay.
4. Decide late data policy: drop with counting, route to a side output, or emit corrections that update earlier results.
5. Bound state with windows, time-to-live, and compaction, and size storage for peak.
6. Configure checkpointing and delivery semantics, and make sinks idempotent via upserts keyed by window and key.
7. Keep the input topic retained long enough to replay, and rehearse a replay with a new job version.

Common pitfalls

• Using processing time for business logic, producing different answers on replay.
• Skewed keys that overload a single partition.
• Unbounded state from joins without time bounds.
• Watermarks stalled by one idle partition.
• Changing job logic without a state migration plan.

Metrics to watch

• Consumer lag and end-to-end latency.
• Watermark delay and late event rate.
• State size and checkpoint duration.
• Backpressure and partition skew.
• Sink error rate and duplicate write counts.

Example configuration

pipeline: orders-per-minute
source:
  topic: orders.placed
  partitions: 24
  eventTimeField: placedAt
watermark:
  maxOutOfOrderSeconds: 30
window:
  type: tumbling
  size: 60s
  allowedLateness: 5m
  lateData: side-output
state:
  ttl: 1h
checkpoint:
  intervalSeconds: 30
sink:
  mode: upsert
  key: [windowStart, region]

Practical rollout

Prove the design with a replay test before launch. Feed a recorded day of events, including a deliberate burst of late and duplicated records, through the pipeline twice and compare outputs. Differences point to dependence on processing time or non-idempotent sinks. Agree with consumers on how corrections are published, since downstream dashboards must handle updated window results rather than assuming each window is emitted once. Document the replay procedure so on-call engineers can run it.

Key takeaways

• Use event time and watermarks, not arrival time.
• Bound state and plan for skew.
• Make sinks idempotent and checkpoint with offsets.
• Retain input so pipelines can be replayed.