Stackbook Logo
← Back to System Architectures

Designing a URL Shortener

Design a URL shortener handling 100M new links a month and 10B redirects, covering ID generation, caching, storage and abuse handling.

4 min read

A URL shortener maps a short code to a long URL and redirects visitors. It looks trivial, but at consumer scale it is a read-heavy, latency-sensitive system where ID generation, caching and abuse control decide whether it holds up.

Requirements

Functional

  • Create a short link from a long URL, optionally with a custom alias and expiry.
  • Redirect a short link to the original URL.
  • Basic click analytics (count, referrer, country).

Non-functional

  • 100M new links per month, 10B redirects per month (100:1 read to write).
  • Redirect p99 under 50 ms at the edge, 99.99 percent availability.
  • Links must never collide, and must not be guessable in sequence.
  • Links live for years: 5 years of retention.

Capacity estimation

  • Writes: 100M / 2.6M seconds is roughly 40 writes per second, peak 200.
  • Reads: 10B / 2.6M is roughly 3,850 redirects per second, peak around 20,000.
  • Storage: 6B links over 5 years at about 500 bytes each is roughly 3 TB.
  • Code length: base62 with 7 characters gives 62^7, about 3.5 trillion codes, far above 6B.
  • Cache: the top 20 percent of links drive about 80 percent of traffic. Caching 20 percent of daily hot links needs tens of GB, which fits in a small Redis cluster.

High-level design

Client -> CDN/Edge -> Load Balancer -> Redirect Service -> Cache (Redis)
                                              |                |
                                              | miss           v
                                              +----------> Key-value store
Client -> API Gateway -> Create Service -> ID Generator -> Key-value store
                                |
                                +--> Kafka -> Analytics pipeline -> OLAP store

The redirect path and the create path are separate services so each scales independently. Analytics are emitted asynchronously and never block a redirect.

Data model

A single table keyed by the short code:

links(code PK, long_url, user_id, created_at, expires_at, is_blocked)

Lookups are always by primary key, so a partitioned key-value store (DynamoDB, Cassandra) fits well. Hash-partitioning by code spreads load evenly. A secondary index on user_id supports account dashboards.

Deep dives

ID generation. Three common options:

  1. Hash the URL (MD5, take 7 characters). Simple, but collisions need a check and retry, and identical URLs from different users share a code.
  2. Counter plus base62 encoding. Collision-free, but sequential codes are enumerable. Fix by passing the counter through a reversible permutation or a Feistel cipher.
  3. Pre-generated key service. A background job fills a pool of unused codes. Each create server leases a batch of, say, 1,000 codes into memory, so no coordination happens per write.

Option 3 is the usual production choice: no per-request contention and no collisions. If a server crashes it loses at most one unused batch, which is negligible against 3.5 trillion codes.

Redirect status code. A 301 is cached by browsers, which cuts load but hides later clicks from analytics and prevents link edits. A 302 keeps every click visible. Most products use 302 or a short-lived 301 with Cache-Control max-age.

Caching. Use cache-aside with an LRU policy and a TTL. Negative-cache missing codes for a short period so scanners probing random codes do not hammer the database. Put the CDN in front for the hottest links.

Expiry. Check expires_at on read, and delete lazily with a background sweeper. Do not reuse expired codes for a long grace period to avoid sending old holders to the wrong site.

Abuse. Check submitted URLs against a malware and phishing blocklist at creation and periodically afterwards. Rate limit creation per user and IP. Setting is_blocked must invalidate the cache immediately.

Failure modes and trade-offs

  • Cache outage: the database sees a surge. Mitigate with replicated caches, request coalescing for the same code, and database autoscaling.
  • Key service down: servers keep serving from leased batches, and a standby refills the pool.
  • Region failure: replicate the key-value store across regions and route by latency. Writes tolerate eventual consistency because a new link is rarely clicked in the first seconds, but read-your-write for the creator can be served from the local region.
  • Custom aliases need a conditional write (put if absent) to guarantee uniqueness, which makes them slightly slower than generated codes.
  • Hot keys: a viral link can overload one partition. Replicate hot entries across cache nodes or in process memory.

Key takeaways

  • The system is about 100:1 read-heavy, so optimise the redirect path with CDN and cache first.
  • Seven base62 characters give trillions of codes; the hard part is collision-free, non-guessable allocation.
  • A pre-generated key pool removes write coordination.
  • Choose 301 versus 302 deliberately: it trades load against analytics accuracy.
  • Treat abuse handling and cache invalidation on blocking as first-class requirements.