Stackbook Logo
scalingestablished · high operational burden

Partitioning Strategies (Sharding)

Also known as: sharding-strategies, horizontal-partitioning, data-partitioning, consistent-hashing

Intent

Choose and implement partitioning strategy: hash, range, directory, consistent hashing — trade-offs for distribution, resharding, and query routing.

Problem

Single database hits limits. How to split data across nodes? Wrong strategy = hotspots, impossible resharding, cross-shard queries.

Forces

  • •Uniform distribution: avoid hotspots
  • •Query locality: keep related data together
  • •Resharding: split/merge without downtime
  • •Query routing: client or proxy knows shard

Solution

✓ When to Use

  • •Data > single node capacity (storage, CPU, connections)
  • •Throughput > single node (100k+ QPS)
  • •Multi-tenant isolation required

✗ When Not to Use

  • •Single node sufficient with headroom
  • •Complex cross-shard queries (joins, transactions)
  • •Team not ready for operational complexity

Pros

  • +Horizontal scale: add shards, not bigger nodes
  • +Fault isolation: one shard down ≠ total outage
  • +Tenant isolation: noisy neighbor contained

Cons

  • −Cross-shard queries: fan-out, high latency, no transactions
  • −Resharding: complex, risky, long-running
  • −Hotspots: skew in key distribution
  • −Operational: more nodes, more failure domains

Cost Profile

Infrastructure

High — more nodes, coordination

Operational

High — resharding, monitoring per shard

Cognitive

High — strategy choice, query routing, resharding

Failure Modes

  • Hot shard: 80% load on one shard

  • Resharding stall: backfill falls behind, switch delayed

  • Cross-shard transaction: partial commit, inconsistency

  • Shard count too high: metadata overhead, slow planning

  • Routing layer SPOF: proxy down = all shards unreachable

Real-World Examples

Alternatives

  • read-replica
  • vertical-scaling
  • caching
  • archive-old-data
  • columnar-store

Related Patterns

  • consistent-hashing
  • partitioning
  • tenant-isolation
  • read-replica
  • cdc
  • database-migration

Competency Domains

scalingdata statereliability opseconomics evolutiondistribution communication