Stream Windowing

Overview

Windowing slices time into pieces that can be aggregated. The key discipline is using event time, the moment something happened, not the time it reached the processor.

A watermark is the engine belief that no more events older than a given time will arrive.

Pick the window type to match the question. Tumbling windows fit per-minute counts, sliding windows fit moving averages that update more often than the window length, and session windows fit user activity separated by quiet gaps. Each assigns events differently and carries different state costs, since sliding windows hold each event in several windows.

How it works

Each event is mapped to one or more windows by its timestamp. When the watermark passes a window end plus allowed lateness, the aggregate is emitted and state released.

window = floor(event.ts / 60s) * 60s     # tumbling, 1 min
counts[(key, window)] += 1

when watermark >= window + 60s + lateness:
    emit(key, window, counts.pop((key, window)))

Handling lateness is a product decision as much as a technical one. A longer allowed lateness improves accuracy but delays final results and holds more state. Many systems emit an early provisional result and then updates as late data arrives. Whatever you choose, count dropped late events as a metric so you know the real cost of your watermark setting.

Pitfalls

• Aggregating by processing time, so retries skew results.
• A single idle partition holding back the watermark for the whole job.
• Setting lateness too tight and discarding real data without measuring it.
• Choosing session gaps that never close for bots.

Key takeaways

• Use event time with watermarks.
• Pick the window type to fit the question asked.
• Route late data to a side output and measure it.
• Watch state size per open window.