Designing for Scalability: A Guided Track

Scalability is the ability to handle growth in load by adding resources, without a redesign and without costs growing faster than usage. It is not a property you add at the end, and it is not the same as performance. A fast system on one machine may not scale; a slower design that scales horizontally may serve far more users. This track walks through the core techniques in order and warns about the most common mistake: building for scale you do not have.

Learning goals

• Quantify load and set scaling targets with back-of-envelope estimation.
• Find the bottleneck before changing anything.
• Apply stateless scaling, caching, queues and data partitioning appropriately.
• Recognise the trade-offs each technique introduces.

Week 1: Measure before you design

Start with numbers. Estimate requests per second, data volume, read-to-write ratio and growth rate.

10M daily active users x 20 requests/day = 200M requests/day
200M / 86,400 s = about 2,300 requests/s average
Peak factor 3x  = about 7,000 requests/s

Then profile the existing system. Find the saturated resource: CPU, memory, disk I/O, network, database connections, or a lock. Amdahl's law is a useful reminder: if 10 percent of the work is serial, no amount of parallel hardware gives more than a 10x speedup.

Exercise: for a system you know, produce a one-page capacity model with current load, 12-month projection and the first resource expected to saturate.

Week 2: Scale the stateless tier horizontally

The easiest win is making application servers stateless so you can add instances behind a load balancer.

• Move session state to a shared store or signed tokens.
• Keep instances identical and disposable.
• Use health checks and autoscaling driven by a meaningful metric such as queue depth or request concurrency, not only CPU.

Trade-off: statelessness pushes state elsewhere, usually to the database or cache, which becomes the next bottleneck.

Exercise: load test a service at 1, 2 and 4 instances and plot throughput. Where does it stop scaling linearly, and why?

Week 3: Caching and asynchronous work

Caching reduces load on slower resources but introduces staleness and invalidation complexity.

| Layer | Example | Main risk | |---|---|---| | Client / CDN | Static assets, public pages | Stale content | | Application cache | Computed results in a shared cache | Invalidation bugs | | Database cache | Query cache, buffer pool | Limited control |

Watch for cache stampedes, where many requests recompute an expired entry simultaneously. Mitigations include request coalescing, jittered expiry and serving stale data while refreshing.

Queues decouple slow or bursty work from the request path. Accept the request, enqueue the job, respond quickly, process in workers. Required disciplines: idempotent consumers, because delivery is typically at least once; dead-letter handling; and back-pressure so producers do not overwhelm consumers.

Exercise: move one slow operation, such as sending email or generating a report, to a queue. Define behaviour when the queue backs up.

Week 4: Scaling the data tier

Databases are the hardest part, since they hold state.

1. Optimise first. Indexes, query shape, connection pooling. Many systems that appear to need sharding need an index.
2. Read replicas. Offload reads, accepting replication lag. Route read-your-writes cases to the primary.
3. Partitioning (sharding). Split data by key across nodes.

shard = hash(customer_id) mod N

Partitioning choices matter: a poor key creates hot spots, such as one celebrity account. Simple modulo hashing makes adding nodes painful because most keys move; consistent hashing, used in Dynamo-style systems, limits data movement. Cross-shard queries and transactions become expensive or impossible, so model your access patterns first.

4. Choose the store for the access pattern. Wide-column and key-value stores scale writes horizontally but restrict queries.

Exercise: design a sharding key for an orders table, list the queries that become hard, and propose how to serve them.

Avoiding over-engineering

• Scale for roughly ten times current load, not a thousand times.
• Prefer the simplest technique that meets the target; a bigger machine is sometimes the right answer.
• Each new component adds failure modes and operational burden.
• Revisit estimates regularly with real data.

Key takeaways

• Quantify load and growth first, and find the actual bottleneck before changing the design.
• Make the application tier stateless to scale it horizontally.
• Use caching and queues deliberately, with plans for invalidation, idempotency and back-pressure.
• Optimise queries and use replicas before partitioning; choose partition keys by access pattern.
• Every scaling technique trades something, usually consistency or operational complexity.
• Design for the next order of magnitude, not an imagined one.