Design a Social Media Feed (Twitter/Instagram Timeline)

Clarifying questions I'd ask first

• Reverse-chronological or ranked feed?
• Replies, reposts, quote-posts — separate post types in the feed?
• Real-time updates while user is viewing?
• Multiple feed surfaces (home, profile, hashtag)?
• Acceptable consistency on unfollow — strict or eventual?

Assumptions for this design

500 million DAU, 2 billion total users, 400 million posts a day, peak 4× during major events. Median user has ~200 followers; 99.9th-percentile celebrity has 50 million plus. We pick a 10K-follower threshold for the hybrid boundary. Sub-200-millisecond p99 for the home timeline first page. Ranked, not strictly chronological.

Quick capacity check

• Posts: ~4,600/sec average, 18,000/sec peak
• Reads: ~58,000/sec average, 250,000/sec peak
• Pure fan-out-on-write would cost 4.6 million timeline writes/sec on average and tens of millions on a celebrity post — doesn't work
• With celebrity exclusion, fan-out drops to ~1.5 million writes/sec — manageable
• Storage: 400 GB/day of posts, 5 TB Redis for timelines

The central decision — hybrid fan-out

• Authors with under 10K followers: fan-out-on-write — each post written into every follower's timeline ZSET in Redis
• Authors with 10K+: fan-out-on-read — the post sits in a celebrity-recent cache; followers pull at read time
• The 10K number balances write cost (linear in follower count) against read merge cost (linear in follow count)
• A non-celebrity becoming a celebrity overnight is handled gracefully by eventual consistency — we just stop pushing for them; the read path picks up new entries via the celebrity cache

High-level shape

• Publish: Post Service writes to Cassandra, emits to Kafka; Fan-Out Service consumes and applies the hybrid policy
• Read: Feed Service reads pre-computed timeline ZSET, fetches celebrity-recent caches in parallel, dedupes, ranks, hydrates author and engagement, returns paginated
• Real-time: post publish notifies actively-connected followers via WebSocket/SSE

Data model

• Cassandra posts partitioned by author_id, clustered by post_id descending — profile timeline is one partition scan
• Cassandra followers and following tables for graph (denormalized; both written on follow/unfollow)
• Redis ZSETs timeline:{user_id} capped at 200 entries
• Redis lists celebrity:{user_id}:recent capped at 100 entries
• Engagement counters in Redis, periodically flushed to Cassandra

Read path — step by step

• Pull pushed posts from timeline:{user_id} ZSET (200 candidates)
• Look up celebrity followees, fetch their recent posts in parallel
• Merge and dedupe
• Rank with a lightweight scorer using cached ML features (offline-trained model produces per-user candidate-quality cache)
• Apply cursor pagination, take 20
• Hydrate: expand post_ids to full FeedItem with author, counts, viewer-relative state — aggressively cached

LLD — clean abstractions

• TimelineStore interface (RedisTimelineStore impl; can swap for purpose-built timeline DB)
• FanOutPolicy extracted — the threshold logic lives in one tested class
• Ranker as its own collaborator — swap models with config, A/B test, shadow traffic
• HydrationService parallelized with caching
• Feed Service uses parallel streams for celebrity fetch — serial would blow the latency budget

Caching layers

• L1 in-process Caffeine for hydrated FeedItems
• L2 Redis hydration cache keyed by (post_id, viewer_id)
• L3 the pre-computed timeline ZSET
• Origin Cassandra
• Hot-post problem (viral post in 50M timelines) handled by L1+L2 — underlying row read once, served millions of times

Mid-scroll consistency

• Unfollow: relaxed — stale entries OK for the current page; future entries excluded
• Block: strict — always filter at read time, even at extra cost

Failure handling

• Fan-Out crash: Kafka replay; Redis ZADD is idempotent
• Ranker slow: 30ms timeout, fallback to chronological with recency decay
• Redis shard down: degraded feed (celebrity pulls only) until failover (~30s)
• Cassandra node down: LOCAL_QUORUM tolerates 1 of 3
• Cold cache: pre-warm top 10% from snapshot; rebuild on demand for the rest

Production extras

• Block-list filtering at read time (strict)
• Shadow ban support for abuse handling
• Rate limit posts per user (300/hour)
• Content moderation async; tombstones propagate to caches
• Distributed tracing across the parallel read fan-out

At 10× scale

• Purpose-built timeline DB (Twitter Manhattan, Meta TAO) instead of Redis
• Per-region active-active
• Edge ranking (CDN/edge compute)
• Pre-fetched next page on the client

Recommendation

Hybrid fan-out at a 10K-follower threshold, pre-computed timelines in Redis ZSETs for the long tail, celebrity caches for pull-merge at read time, Cassandra for posts and graph, lightweight read-time ranking backed by an offline-trained candidate-quality model, and aggressive hydration caching — the same shape Twitter, Instagram, and LinkedIn run today.