Design a Chat/Messaging System (WhatsApp)

Clarifying questions I'd ask first

• One-on-one only, or also group chats — and what's the max group size?
• Media — images, video, file attachments?
• End-to-end encryption — in scope or can the server read messages?
• Read receipts and typing indicators?
• Multi-device support — one user across multiple devices in sync?
• Retention — keep messages forever or expire after N days?

Assumptions for this design

A billion users, 300 million daily active, around 200 million concurrent connections at peak. 40 billion messages a day, average size 100 bytes, group chats up to 256 members. Sub-100-millisecond p99 in-region. No end-to-end encryption — we'll discuss the trade-off but assume server-side plaintext.

Quick capacity check

• Sends: roughly 460,000 per second average, a million per second at peak
• Deliveries with average group size 5: about 3 million per second
• Storage: 10 terabytes per day of message rows — needs Cassandra or ScyllaDB sharded across hundreds of nodes
• 200 million concurrent WebSockets needs around 2,000 Connection Service instances at 100K each

High-level shape

• Edge layer: Connection Services terminate WebSockets, stateless about content but stateful about which user is on which server
• Bus: Kafka decouples Connection Services from the Message Service; absorbs bursts, gives durable replay
• Service layer: Message Service persists to Cassandra and orchestrates fan-out
• Storage: Cassandra for messages (partitioned by conversation_id), PostgreSQL for membership, Redis for presence

Send path — step by step

• Sender's WebSocket frame arrives at a Connection Service
• Connection Service publishes to Kafka, partitioned by conversation_id
• Message Service consumes, allocates a per-conversation sequence number from Redis (atomic INCR), writes to Cassandra
• ACKs sender with the assigned sequence
• Looks up each recipient's Connection Service in Redis presence; routes the delivery frame via Kafka to the right server
• Offline recipients get a push via APNS or FCM

Ordering — server-assigned sequence per conversation

• Every message has a strictly monotonic sequence number, allocated by Redis INCR
• Clients render by sequence, not by client timestamp — timestamps are display-only
• I rejected vector clocks because we already have a coordinator (the Message Service); they'd be unnecessary complexity
• I rejected server timestamp because two messages in the same millisecond on different instances would tie

Delivery guarantees

• At-least-once from server to client; the server resends if no DELIVERED receipt arrives in 5 seconds
• Client dedup by client_msg_id — effectively exactly-once at the UI
• Offline: message persists in Cassandra, push notification fires, client pulls catch-up on reconnect

Data model

• Cassandra messages table: partition key conversation_id, clustering key message_id descending — scrolling history is one partition scan
• Cassandra user_conversations table: partition by user_id, clustered by last_activity_at for the inbox view
• PostgreSQL conversation_members for relational membership data
• Redis presence:{user_id} with 90-second TTL refreshed by client heartbeat

Fan-out — hybrid approach

• Small groups (under 256): fan-out-on-write, message persisted into each recipient's inbox — simplifies offline catchup
• Large broadcast channels: fan-out-on-read, store one message per channel and clients scan the channel timeline
• Our design assumes ≤256 member groups, so fan-out-on-write across the board

LLD — components in the send pipeline

• ConnectionHandler — WebSocket I/O
• MessageConsumer — reads SendCommand from Kafka, allocates sequence, persists, fans out
• SequenceAllocator — wraps Redis INCR; can swap to batched allocation for hot conversations
• MessageRepository interface backed by Cassandra; in-memory impl for tests
• DeliveryRouter — per-recipient decision: online routing via Kafka, offline via push
• PresenceService — thin Redis wrapper

Failure modes

• Connection Service crash: clients reconnect, request catch-up by sequence; presence TTL cleans up stale entries
• Message Service crash mid-fanout: Kafka replay; Cassandra write is idempotent on PK; clients dedup
• Cassandra node down: LOCAL_QUORUM keeps reads and writes alive on 2 of 3 replicas
• Kafka down: small in-memory backlog at Connection Service, then 503 to client; client retries
• Redis presence cluster down: degrade to broadcast routing across all Connection Servers in the region

Production extras

• TLS 1.3 + JWT auth on connect, periodic token refresh
• Per-user rate limit (50 sends/minute, 1000 frames/minute)
• Distributed tracing across the full send pipeline
• E2EE if required — acknowledge that you lose server-side search, link previews, and abuse detection

At 10× scale

• Edge-region Connection Services for geo-proximity
• Per-region Cassandra with cross-region replication
• Gossip-based presence instead of Redis
• QUIC/HTTP3 for history pulls

Recommendation

Stateless Connection Services with WebSocket termination, Kafka as the durable bus, Message Service with server-assigned per-conversation sequence numbers from Redis, Cassandra partitioned by conversation_id for messages, and APNS/FCM for offline notifications — the same shape WhatsApp, Slack, and Discord run in production.