Stackbook Logo
← Back to System Architectures

Designing a Web Crawler

Design a distributed web crawler fetching 1B pages a month with a URL frontier, politeness, deduplication, robots.txt handling and trap avoidance.

4 min read

A web crawler discovers and downloads pages for a search engine or data pipeline. The challenge is not fetching one page but doing it politely, without duplicates and traps, across billions of URLs while keeping content fresh.

Requirements

Functional

  • Start from seed URLs, fetch pages, extract links, and repeat.
  • Respect robots.txt and per-site crawl delays.
  • Detect duplicate and near-duplicate content.
  • Re-crawl pages based on how often they change.

Non-functional

  • 1B pages fetched per month; the full corpus is about 10B URLs.
  • Polite: no more than about 1 request per second per host by default.
  • Fault tolerant and horizontally scalable; resumable after failure.
  • Extensible to new content types such as PDFs and images.

Capacity estimation

  • 1B pages per month is roughly 385 per second on average; plan for 1,000 per second to allow retries and re-crawls.
  • Average page 500 KB: 500 TB per month of raw content; compress to about 150 TB.
  • URL metadata for 10B URLs at 100 bytes is 1 TB.
  • Bandwidth: 1,000 pages per second at 500 KB is 500 MB per second, about 4 Gbps, spread across fetcher nodes.
  • DNS: each fetch may need a lookup, so cache DNS locally, otherwise resolver latency dominates.

High-level design

Seed URLs -> URL Frontier (priority + politeness queues)
                  |
                  v
           Fetcher workers --> DNS cache --> Internet (robots.txt cache)
                  |
                  v
        Raw content store (object storage)
                  |
        Parser/Extractor --> Content fingerprint check (dedupe)
                  |
        Link extractor --> URL normaliser --> URL seen filter --> back to Frontier

Data model

url_state(url_hash PK, url, host, last_fetch, next_fetch, status, content_hash, change_rate)
robots(host PK, rules, fetched_at, ttl)
content(content_hash PK, storage_key, size, fetched_at)

Partition url_state by host hash so that all URLs for a host live on the same node, which makes per-host politeness enforcement local.

Deep dives

URL frontier. The frontier has two layers. Front queues rank URLs by priority (page importance, change rate, depth). Back queues are one per host, each with a timer enforcing politeness delay. A selector picks the next ready host. Workers are assigned hosts consistently so a host is never hit by two workers at once. The bulk of the frontier lives on disk with in-memory buffers at the head.

Politeness and robots. Fetch and cache robots.txt per host with a TTL of about a day, and obey disallow rules and crawl delay. Adapt rate to server response time: back off on slow responses or 429 and 5xx codes. Identify the crawler with a clear user agent.

URL normalisation and seen filter. Lowercase host, remove fragments, sort query parameters, strip tracking parameters, and resolve relative paths. To check whether a URL has been seen among billions, use a Bloom filter in front of the authoritative store. At 10B URLs and a 1 percent false positive rate, a Bloom filter needs about 12 GB, a handful of shards. False positives mean a few new URLs are skipped, an acceptable loss.

Content deduplication. Compute a hash of normalised content for exact duplicates, and SimHash or MinHash for near duplicates. Index by fingerprint so a duplicate page is stored once and linked.

Spider traps. Infinite calendars, session IDs in URLs and generated links create unbounded URL spaces. Mitigate with maximum URL length and depth, per-host page budgets, and detection of repeated path patterns.

Freshness. Estimate each page's change rate from past fetches and schedule recrawl intervals accordingly: news pages every few minutes, static pages monthly. Use conditional requests (If-Modified-Since, ETag) to save bandwidth.

Rendering. JavaScript heavy pages need a headless browser, which is roughly 10 to 50 times more expensive. Use it only for domains where static fetch yields little content.

Failure modes and trade-offs

  • Fetcher crash: the URL lease expires and the URL returns to the frontier; fetches are idempotent.
  • Frontier loss: persist frontier state with checkpoints so the crawl resumes instead of restarting.
  • Slow or hostile hosts: strict connect and read timeouts, size caps, and content type checks protect workers.
  • Depth first versus breadth first: breadth first with priority gives better coverage of important pages early.
  • Coverage versus politeness: raising per-host rate speeds the crawl but risks blocks and harms sites. Politeness is a hard constraint, not a tunable.

Key takeaways

  • The URL frontier, combining priority with per-host politeness, is the core component.
  • Partition by host so politeness is enforced locally without coordination.
  • A Bloom filter plus authoritative store handles the seen-URL check at billions of URLs.
  • Content fingerprints, including SimHash, prevent storing duplicates.
  • Trap limits and adaptive recrawl scheduling keep the crawl bounded and fresh.