Stackbook Logo
← Back to System Architectures

Designing a News Feed and Timeline

Design a social news feed for 300M daily users, comparing fan-out on write and read, hybrid celebrity handling, ranking and cache sizing.

4 min read

A news feed shows each user a ranked list of recent posts from the accounts they follow. The central tension is that reads vastly outnumber writes, yet a single post can have to reach millions of followers.

Requirements

Functional

  • Publish posts (text, media references).
  • Follow and unfollow accounts.
  • Fetch a home timeline, paginated, newest or ranked first.

Non-functional

  • 300M daily active users, each opening the feed about 10 times per day.
  • Feed load p99 under 300 ms.
  • New posts visible to followers within 5 seconds for normal accounts.
  • Highly available; slightly stale feeds are acceptable.

Capacity estimation

  • Feed reads: 300M x 10 = 3B per day, roughly 35,000 per second, peak 100,000.
  • Posts: 100M per day, roughly 1,200 per second.
  • Average follower count 200, so fan-out on write produces about 20B feed insertions per day, roughly 230,000 per second.
  • Feed cache: keep the latest 800 post IDs per user at 8 bytes each, about 6.4 KB per user. For 300M users that is roughly 2 TB of RAM, spread across a Redis cluster.

High-level design

Client -> API Gateway -> Post Service -> Post store (sharded)
                              |
                              v
                           Kafka (post events)
                              |
                      Fan-out Workers --reads--> Social Graph Service
                              |
                              v
                    Feed Cache (user_id -> list of post_ids)

Client -> Feed Service -> Feed Cache -> Hydrate posts + users -> Ranker -> Response

Data model

posts(post_id PK, author_id, body, media_ref, created_at)
follows(follower_id, followee_id, created_at)      -- two indexes: by follower, by followee
feed:{user_id}  -> sorted list of (post_id, score)  -- in Redis

Post IDs are time-sortable (Snowflake style) so ordering needs no extra column and pagination uses a cursor of the last post ID, not an offset.

Deep dives

Fan-out on write (push). When a user posts, workers read the follower list and prepend the post ID to each follower's feed list. Reads become a single cache lookup, which is ideal for the 100:1 read ratio. The cost is write amplification.

Fan-out on read (pull). Store nothing per follower. At read time, fetch the recent posts of everyone the user follows and merge. Writes are cheap, but reads touch hundreds of partitions and are slow.

Hybrid. The practical answer: push for normal accounts, pull for celebrities. An account with more than, say, 1M followers is flagged. Its posts are not fanned out. At read time the feed service merges the precomputed list with the latest posts from the few celebrities the user follows. This bounds worst-case fan-out to something workers can finish in seconds.

Inactive users. Skip fan-out for users inactive for 30 days; rebuild their feed on next login by pull. This cuts writes substantially.

Ranking. A lightweight ranker scores a candidate set of a few hundred posts using recency, author affinity, engagement and media type. Candidate generation stays cheap; the model runs only on candidates, with a strict latency budget of about 50 ms and a fallback to reverse-chronological order on timeout.

Hydration. The cache holds only IDs. The feed service batch-fetches post bodies, author profiles and counters from separate caches. Counters (likes, comments) are eventually consistent and cached separately because they change often.

Pagination. Cursor-based. If the cursor falls off the cached 800 entries, fall back to pull from the post store.

Failure modes and trade-offs

  • Fan-out lag: a Kafka backlog delays posts. Partition by follower shard, autoscale workers, and prioritise active users first.
  • Cache loss: feeds can be rebuilt from the follows table and post store through pull, at a latency cost. Rebuild lazily on first read, with request coalescing.
  • Deletes and blocks: removing a post requires filtering at hydration time, since removing the ID from millions of lists is too expensive. Tombstone check on read is cheap.
  • Celebrity threshold: too low increases read cost, too high increases write cost. Tune it from measured metrics.
  • Consistency: a user may not see their own just-published post if fan-out is slow. Insert it into their own feed synchronously to give read-your-writes.

Key takeaways

  • The workload is read-dominated, so precompute feeds as lists of post IDs in memory.
  • Pure push breaks for celebrities and pure pull breaks reads; a hybrid bounds both.
  • Store IDs, hydrate on read, and filter deletes at hydration time.
  • Ranking runs on a small candidate set with a latency budget and a chronological fallback.
  • Skip fan-out for inactive users and rebuild their feeds on demand.