Stackbook Logo
← Back to System Architectures

Data Partitioning Strategies

When data grows beyond a single database's capacity, partitioning becomes necessary.

3 min read

When data grows beyond a single database's capacity, partitioning becomes necessary.

Partitioning Strategies

Strategy 1: Horizontal Partitioning (Sharding)

Split rows across multiple databases. Each database (shard) holds a subset of the data.

Key decisions:

  • Shard key: What field determines which shard a row goes to?
  • Number of shards: How many shards to start with? Plan for growth.
  • Shard migration: How do you add/remove shards later?

Common shard keys:

  • User ID range (user_id % num_shards)
  • Geographic region
  • Tenant ID
  • Application ID

Pros: Scales horizontally, can colocate with compute Cons: Complex joins across shards, uneven data distribution, rebalancing complexity

Strategy 2: Vertical Partitioning

Split columns across multiple databases. Different tables or column groups on different databases.

When to use: Different access patterns for different columns, some data much hotter than others.

Pros: Can scale hottier data on more powerful hardware, simpler than horizontal partitioning Cons: Joins across partitions, still limited by single database scale

Strategy 3: Caching

Cache frequently accessed data at the edge or in-memory.

Cache-aside pattern: Application checks cache first, misses, reads from database, writes to cache.

Write-through cache: Writes go to both database and cache simultaneously.

Write-behind cache: Writes go to cache first, asynchronously written to database.

Pros: Dramatic read performance improvement Cons: Cache coherence complexity, stale data risk, additional operational overhead

Strategy 4: Read/Write Splitting

Separate read and write workloads to different databases.

Primary-replica pattern: All writes go to primary, reads can go to any replica.

Pros: Read scaling without affecting write performance Cons: Replication lag can cause stale reads, write throughput limited by primary

Strategy 5: Materialized Views

Pre-computed views of data for specific access patterns.

Pros: Excellent read performance for specific queries Cons: Write overhead (updates must refresh views), storage overhead, data consistency considerations

Choosing a Strategy

When Horizontal Partitioning Makes Sense

  • Data volume exceeds single database capacity
  • Access patterns allow reasonable shard keys
  • Team has operational capability for shard management
  • You've outgrown read/replica scaling

When Caching Makes Sense

  • Read-heavy workloads
  • Frequently accessed data with relatively static content
  • Latency requirements exceed what database alone can provide
  • Team has caching operational capability

When Read/Write Splitting Makes Sense

  • Read-heavy workloads (common in business applications)
  • You can tolerate some staleness (replication lag)
  • You want to scale reads without sharding complexity
  • You're not ready for full horizontal partitioning

The Honest Truth

Data partitioning adds significant operational complexity. Start simple and add complexity only when needed. Most applications outgrow a single database's performance long before they outgrow its storage capacity. Profiling and optimization often solve perceived scaling problems without the complexity of partitioning.

Published June 1, 2026 · 12 min read