Designing a Chat and Messaging System

A chat system delivers messages between users in real time, in order, and without loss, even when recipients are offline. The hard parts are managing millions of persistent connections, ordering, and delivery guarantees.

Requirements

Functional

• One-to-one and group chat (up to 1,000 members).
• Delivery and read receipts.
• Offline delivery with push notification and message history sync across devices.
• Media attachments by reference.

Non-functional

• 500M users, 100M concurrently connected.
• 50B messages per day.
• End-to-end latency p99 under 200 ms for online recipients.
• At-least-once delivery with client-side dedupe, never lose an acknowledged message.
• Per-conversation ordering.

Capacity estimation

• 50B messages per day is roughly 580,000 per second, peak around 1.5M.
• Average message 200 bytes plus 100 bytes metadata: 15 TB per day, about 5.5 PB per year.
• Connections: a tuned gateway holds roughly 200,000 WebSocket connections, so 100M concurrent needs about 500 gateway hosts plus headroom, so plan for 800.
• Session map: 100M entries of user to gateway at about 100 bytes is 10 GB, easily held in Redis.

High-level design

Client <==WebSocket==> Gateway cluster <--> Session Registry (Redis)
                            |
                            v
                      Chat Service --> Message Queue (per-conversation partition)
                            |                   |
                            v                   v
                      Message Store        Delivery Workers --> target Gateway
                       (wide-column)               |
                                                   +--> Push Service (offline)

Data model

messages(conversation_id, message_id, sender_id, body, created_at)
   PK = (conversation_id), clustering key = message_id DESC
conversations(conversation_id, type, members)
inbox(user_id, conversation_id, last_read_id, last_message_id)

A wide-column store such as Cassandra suits this: writes are append-only and reads are range scans on one partition. For very active groups, bucket the partition by time (conversation_id, month) to stop partitions growing unbounded.

Deep dives

Connections. Clients hold a WebSocket to a gateway chosen by a load balancer. On connect, the gateway records user to gateway in the session registry with a TTL refreshed by heartbeats every 30 seconds. Gateways are stateless otherwise, so a crash just triggers client reconnect with exponential backoff and jitter.

Ordering. Message IDs are monotonically increasing per conversation, assigned by the partition owner (a sequence per conversation) or a Snowflake ID with the conversation partitioned to a single queue partition. Clients sort by ID and detect gaps, then request the missing range.

Delivery guarantee. The sender gets an ack only after the message is durably written. Delivery is at-least-once: workers retry until the recipient device acks, and clients drop duplicates by message ID. Exactly-once is not attempted; idempotent clients are cheaper.

Group fan-out. For small groups, write one copy to the message store and push to each member's gateway. For large groups, store once and let members pull from the shared conversation log, notifying them with a lightweight signal. Per-member state is only the read pointer.

Offline users. If the registry has no entry, the worker skips the socket and triggers a mobile push through APNs or FCM. On reconnect the client sends its last seen message ID per conversation and the server streams the rest.

Multi-device. Each device has its own connection and cursor. Messages are fanned out per device, and read receipts sync through the inbox record.

End-to-end encryption. If required, clients encrypt per device keys (a double ratchet scheme). The server stores only ciphertext, so server-side search and moderation of content are not possible, which is a real product trade-off.

Failure modes and trade-offs

• Gateway crash: thousands of clients reconnect at once. Jittered backoff and connection-rate limits prevent a thundering herd.
• Stale session entry: a message is sent to a dead gateway. The delivery worker gets no ack, retries via the registry, and falls back to push.
• Hot conversation: a huge group concentrates writes on one partition. Time-bucketing and a dedicated queue partition limit the damage.
• Network partition across regions: keep each conversation homed in one region for ordering, and replicate asynchronously. Failover accepts a short window of possible reordering.
• Retention cost: 5.5 PB per year pushes older data to cheaper tiers and compaction.

Key takeaways

• Gateways hold connections and stay stateless; a session registry maps users to gateways.
• Order per conversation, not globally, and let clients detect and repair gaps.
• Use at-least-once delivery with idempotent clients rather than chasing exactly-once.
• Small groups push per member; large groups store once and pull.
• Offline delivery combines push notifications with cursor-based sync on reconnect.