Stackbook Logo
data-stateestablished · medium operational burden

Lakehouse Architecture

Also known as: data-lakehouse, open-table-format-architecture

Intent

Combine cheap object-storage data lakes with warehouse-style transactions and SQL using open table formats.

Problem

Teams keep a data lake for raw and ML data and a separate warehouse for BI, copying data between them. This duplicates storage, delays freshness and splits governance.

Forces

  • •Cheap scalable storage versus ACID guarantees
  • •Open formats versus engine-specific optimizations
  • •BI performance versus flexibility for ML workloads
  • •Single governance versus multiple engines

Solution

Overview

A lakehouse puts a transactional metadata layer over files in object storage so that one copy of data serves SQL analytics, streaming and machine learning. It addresses the weaknesses of plain lakes: no atomic writes, no schema enforcement, no safe updates.

How it works

Writers add Parquet files and then commit a new table version by atomically appending to a log or swapping a metadata pointer. Readers pick a snapshot and see a consistent set of files.

table/
  data/part-0001.parquet ...
  _log/000012.json   (adds: part-0007, removes: part-0003)
reader: snapshot = latest log version --> list files --> scan

This yields ACID commits, row-level MERGE, schema evolution, and time travel to earlier snapshots for audit or rollback. Statistics in metadata allow file skipping, and clustering or partitioning limits scans. Engines such as Spark, Trino and Flink access the same tables through a shared catalog.

Pitfalls

  • Small files: streaming writes create many tiny files. Schedule compaction and set target file sizes.
  • Over-partitioning: partitioning by high-cardinality columns explodes metadata. Prefer coarse partitions or clustering.
  • Snapshot retention: deleting old files too soon breaks time travel and slow readers; expire snapshots on a defined policy.
  • Optimistic concurrency: heavy concurrent updates to the same files cause commit conflicts; shard writers by key range.
  • Catalog sprawl: different engines pointing at different catalogs lose consistent governance.

Key takeaways

  • The table format, not the storage, supplies transactions.
  • Plan routine maintenance: compaction, snapshot expiry, statistics.
  • Open formats reduce lock-in but require compatible engine versions.
  • Keep one governed catalog as the entry point.

✓ When to Use

  • •Both BI and ML need the same data
  • •Want to avoid copying between lake and warehouse
  • •Need updates, deletes and time travel on object storage

✗ When Not to Use

  • •Small BI-only workloads fit a plain warehouse
  • •Sub-second point lookups are needed
  • •Team cannot manage compaction and metadata maintenance

Pros

  • +One copy of data for many workloads
  • +Open formats reduce lock-in
  • +ACID and time travel on cheap storage

Cons

  • −Small-file and metadata maintenance needed
  • −Concurrent writer conflicts
  • −Performance trails tuned warehouses on some queries

Cost Profile

Infrastructure

Medium - object storage plus compute engines

Operational

Medium - compaction, vacuum, catalog

Cognitive

Medium - table format concepts

Failure Modes

  • Too many small files degrade scan performance

  • Concurrent writers conflict and retry repeatedly

  • Vacuum removes files still needed by long readers

Real-World Examples

Alternatives

    Related Patterns

      Competency Domains

      data stateeconomics evolution