Problem
Business users want results within seconds, but accurate answers require reprocessing the full history. A single engine historically could not deliver both freshness and correctness at scale.
data-statelegacy · high operational burden
Lambda Architecture
Also known as: lambda-data-architecture
Intent
Combine a batch layer for accurate recomputation with a speed layer for low-latency views over the same data.
Forces
- •Low-latency results versus exact, complete results
- •Immutable raw data enables full recomputation
- •Two code paths must agree on business logic
- •Query layer must merge views from different sources
Solution
Overview
Lambda architecture, described by Nathan Marz, splits analytics into a batch layer, a speed layer, and a serving layer. The batch layer is the source of truth: it recomputes views from an immutable event log. The speed layer fills the gap for data that arrived since the last batch run.
How it works
Every event is written once to the master dataset and also fanned out to the stream path.
events --> master dataset --> batch job (hourly/daily) --> batch view --+
| +--> serving/query merge
+-----> stream processor (incremental) -------------> realtime view -+
A query for total orders today reads the batch view up to the last completed run boundary and adds the real-time view for events after that boundary. When the next batch run finishes, the real-time view for the covered period is truncated. Because the batch layer always recomputes from raw data, bugs in the speed layer are self-healing within one batch cycle.
The cost is that aggregation logic exists twice, often in different frameworks such as Spark for batch and Flink or Storm for streaming. Teams reduce this with shared libraries or unified APIs such as Apache Beam.
Pitfalls
- Logic divergence: a rounding or dedup rule changes in one path only, so dashboards flicker when the batch view replaces the real-time one.
- Boundary errors: events that arrive late straddle the handoff and are counted twice or not at all. Define the cutoff on event time and keep it explicit.
- Operational load: two clusters, two sets of alerts, two deploy pipelines.
- Using it by default: most modern stream engines support stateful processing and replay, making the dual path unnecessary.
Key takeaways
- Immutable raw data plus recomputation is the core idea; the layers are an implementation detail.
- The main price is maintaining equivalent logic twice.
- Prefer a single streaming path with replay when your engine allows it.
- If you adopt lambda, test that batch and speed outputs reconcile automatically.
✓ When to Use
- •Streaming engine cannot yet match batch accuracy for your logic
- •Need both second-level freshness and audited historical totals
- •Regulatory need to recompute from raw data
✗ When Not to Use
- •Your stream processor supports replay and exactly-once (use kappa)
- •Small data where one batch job suffices
- •Team cannot maintain two implementations
Pros
- +Fresh and accurate results together
- +Batch layer repairs speed layer errors automatically
- +Raw data retained for reprocessing
Cons
- −Duplicate logic in two codebases
- −Merge logic in serving layer is subtle
- −Higher infrastructure cost
Cost Profile
Infrastructure
High - two processing stacks plus serving store
Operational
High - keep both paths consistent
Cognitive
High - dual mental model
Failure Modes
Batch and speed layers drift and return different numbers
Window handoff double counts or drops events
Batch run delays leave serving view stale
Real-World Examples
Alternatives
Related Patterns
Competency Domains
data stateeconomics evolution