Stackbook Logo
← Back to System Architectures

Designing a Content Delivery Network

Design a global CDN with edge caches, anycast and DNS routing, tiered origin shielding, cache invalidation and DDoS-resilient delivery of web and video.

5 min read

A CDN places caches close to users so content is served in tens of milliseconds, while shielding the origin from load. This article designs a general-purpose CDN for web assets and video.

Requirements

Functional

  • Serve HTTP and HTTPS content from edge locations, pulling from customer origins on a miss.
  • Purge by URL, prefix or tag; per-path cache rules and TTLs.
  • TLS termination, range requests, compression.

Non-functional

  • 50 Tbps peak egress, 20 million requests per second.
  • Time to first byte under 50 ms for cached content for 95 percent of users.
  • Cache hit ratio above 95 percent for static assets.
  • Global purge propagation under 5 seconds.
  • 99.99 percent availability; survive loss of any edge site.

Capacity estimation

At 50 Tbps, assuming a 100 Gbps per server effective output of about 60 Gbps, we need roughly 850 servers at peak; provision 2000 across 300 points of presence for headroom and locality. If 95 percent of traffic hits cache, only 2.5 Tbps flows to origins, and with a regional shield tier absorbing most of that the origin sees well under 1 Tbps. Cache storage: a hot working set of 100 TB per metro is plausible, using NVMe for hot objects and HDD for the long tail.

High-level design

 User --> DNS / Anycast --> Nearest PoP (edge cache)
                               | miss
                               v
                        Regional shield cache (mid-tier)
                               | miss
                               v
                           Customer origin

 Control plane: config push, purge bus, health, logs, billing

Routing sends users to a nearby healthy PoP. A second tier shields the origin: edge nodes in a region fetch from a regional parent, so a viral object triggers one origin fetch, not hundreds.

Data model

Cache key: scheme, host, normalized path and selected query parameters plus vary headers such as encoding. Cache entry: body chunks on disk, response headers, TTL, validators (ETag, Last-Modified), surrogate tags and a stale-allowed window. Customer configuration (origins, rules, certificates) is versioned and pushed to every edge as an immutable snapshot.

Key deep-dives

Request routing. Two common methods. DNS-based: authoritative DNS returns the IP of the best PoP using resolver location and live load, but resolver location is an imperfect proxy for the user (EDNS client subnet helps) and TTLs slow failover. Anycast: all PoPs advertise the same IP via BGP, so the network delivers packets to the nearest one and a failed site withdraws its route. Many CDNs combine both.

Cache hierarchy and placement. Within a PoP, consistent hashing across servers makes each object live on one or two machines, multiplying effective cache size. Hot objects replicate to more servers automatically. Eviction uses a segmented LRU or admission policies such as TinyLFU so one-hit wonders do not flush valuable content.

Origin protection. Request collapsing coalesces simultaneous misses for one object into a single origin request. Stale-while-revalidate and stale-if-error keep serving when the origin is slow or down. Negative caching of 404s with short TTLs prevents repeated misses.

Invalidation. TTL expiry is simplest. For instant purge, the control plane publishes purge messages over a replicated message bus; each edge applies them and surfaces versioned tags (surrogate keys) so one call can purge all pages carrying a product tag. Versioned URLs (app.4f3a.js) avoid purge entirely and are the best practice for static assets.

Video delivery. Segmented formats (HLS, DASH) turn video into small cacheable files. Prefetch the next segments, and pre-position popular titles into edge storage during off-peak hours.

Security. Terminate TLS at the edge with session resumption and OCSP stapling. Absorb volumetric DDoS by spreading traffic across the whole anycast footprint; add rate limiting and a web application firewall at the edge.

Failure modes and trade-offs

  • PoP failure: BGP withdrawal or DNS health checks redirect users within seconds to tens of seconds; neighbors must have spare capacity, so run PoPs below 60 percent utilization.
  • Origin outage: serve stale content; shield tier prevents a thundering herd on recovery.
  • Purge lag: a missed purge serves outdated content. Make purges idempotent and have edges reconcile with a version number on reconnect.
  • Cache poisoning: an incorrect cache key (ignoring a header that changes the response) leaks one user's content to another. Keep keys strict and never cache personalized responses without explicit Vary.
  • Hit ratio versus freshness: longer TTLs raise hit ratio and cut origin cost but lengthen staleness.
  • Cost: many small PoPs lower latency but are expensive; fewer large ones are cheaper but farther away.

Key takeaways

  • Combine anycast or DNS steering with a tiered cache (edge, shield, origin) to maximize hits and protect origins.
  • Consistent hashing inside a PoP increases effective cache size and avoids duplicate storage.
  • Collapse concurrent misses and serve stale on error; these two features prevent most origin incidents.
  • Prefer versioned asset URLs over purging, and keep a fast tag-based purge for dynamic content.
  • Cache key correctness is a security property, not just a performance setting.