Design a chat system
- Lesson 40 of 40
- 14 min
- Advanced
- 12 steps
A chat app looks simple until millions of phones are connected at once. Then fast is not enough: every message has to arrive, in order, and never twice.
- Delivery time
- 100 ms
- Open connections
- 20M
- Can lose messages
- no
- Order guaranteed
- yes
- request
- response
- handshake
- event
- failed request
Step 1 of 12
Concept
The brief: a messaging app
Third full design: one-to-one chats and small groups. If the recipient is online the message reaches them right away; if not, it waits for them; and it shows up on every one of their devices.
Trust is what matters here. No message may be lost, all of them must show in the order they were sent, and none may appear twice. Speed counts too (under 200 ms when both are online), but never above those three guarantees.
Step 2 of 12
Concept
First, the numbers
Assume (example values) 60 million daily users sending 30 messages each: 1.8 billion a day, about 21,000 per second on average and several times that at peak. At around 200 bytes per message that is 360 GB a day, about 130 TB a year.
The number that shapes the design is a different one: connections. The server pushes messages, so every online user keeps a connection open the whole time. At peak that is 20 million at once.
What changed
- ≈ 21,000 messages/s · 360 GB/day
- 20 M connections open at peak
Step 3 of 12
Solution
First draft: one server, two sockets
As we saw in Real-time updates, asking every few seconds wastes requests and adds delay. So each phone opens a WebSocket: a connection that stays open and lets the server send data whenever it wants.
With a single server everything is easy: Alice and Bob are connected to the same machine. When Alice’s message arrives, the server looks up Bob’s socket in its own memory and writes to it.
What changed
- Chat server
- One WebSocket per device
Step 4 of 12
Problem
A hundred servers: where is Bob?
Twenty million sockets need about a hundred machines. Split the roles: gateways only hold connections open, and a stateless chat service behind them holds the logic.
Now Alice is connected to gateway 1 and Bob to gateway 2. Alice’s message reaches the chat service, which has no way to know which of the hundred gateways holds Bob’s socket.
What changed
- Gateways that only hold sockets
- Logic moves to a stateless service
- The chat service cannot find Bob
Step 5 of 12
Solution
A registry of who is connected where
When a device connects, its gateway writes bob → gateway 2 into an in-memory store, with a short TTL it renews with periodic heartbeats. The chat service checks that registry and passes the message to the right gateway, which writes it to Bob’s socket.
It is the idea from Service discovery applied to every user: a registry kept up to date by heartbeats.
What changed
- Session registry: user → gateway
- Heartbeats keep each entry fresh
- Each message goes to the right gateway
Step 6 of 12
Solution
Store the message before delivering it
So far a message only exists in memory while it travels: if a server crashes halfway, it is gone. The rule is store first, deliver after. The chat service writes each message to a durable store and only then confirms to Alice that it was sent.
While storing it, the service gives it the conversation’s next number (seq). That number, not each phone’s clock, decides the order in which everyone sees the messages.
What changed
- Durable message store
- Store, confirm, then deliver
- Sequence number per conversation
Step 7 of 12
Problem
Case: Bob drives into a tunnel
Bob’s phone loses signal. His registry entry expires, so the chat service does not even try to deliver. The message is already safe in the store, and a push notification (the phone operating system’s alert service) tells him he has new messages.
When he reconnects, Bob’s app says “for conversation 88 I have up to #41” and the server sends everything after it. This cursor sync works the same for a five-second drop, a week offline or a brand-new phone.
What changed
- Bob offline
- Push notification service
- Sync from the last number received
Step 8 of 12
Problem
Case: sent twice, seen once
Alice sends a message from an elevator. It arrives and is stored, but the confirmation is lost on the way back. Her app cannot tell, so it retries, and the server receives the same message twice.
The fix comes from Timeouts and retries: the app creates a unique id for each message before sending it and repeats that same id on every retry. The server recognises an id it has already stored and simply confirms again. Receiving apps drop duplicates the same way.
What changed
- Unique id created on the device
- A repeated id is only confirmed again
Step 9 of 12
Technique
Groups: store once, deliver to many
A group message is stored once, in the group’s conversation. Then the chat service looks up the members, finds each one’s gateway in the registry and sends a copy to every online device: fan-out on write, as in the news feed.
That works well up to a few hundred members. A channel with 100,000 members brings back the celebrity problem: there it is better for each app to fetch new messages when the channel is opened.
What changed
- Carla joins the group
- One copy per online member
Step 10 of 12
Trade-off
“Online” and “typing…” take another lane
Presence looks cheap and is not: every time someone connects or disconnects, the screens of all their contacts might need a change. With 20 million people coming and going, that is a shower of tiny messages.
Treat it as ephemeral. Derive “online” from the registry’s heartbeats, send changes only to people who have that chat or list open right now, and never store “typing…”. If an update is lost, the next heartbeat fixes it.
What changed
- Presence comes from heartbeats
- “Typing…” is never stored
Step 11 of 12
Problem
Case: a gateway crashes
Gateway 2 crashes and its 200,000 sockets drop at once. No message is lost, because everything was stored first, but 200,000 apps try to reconnect in the same second and can knock over whichever gateway they land on.
The apps reconnect with exponential backoff and jitter (Timeouts and retries), the load balancer spreads them across the healthy gateways, each one registers its new location and every app syncs from its cursor.
What changed
- Gateway 2 down
- Reconnect with exponential backoff and jitter
- bob → gateway 1
Step 12 of 12
Summary
The finished design
Each decision in the table answers one guarantee from the brief: deliver at once, never lose, never reorder and never repeat.
End-to-end encryption (Encryption and secrets) changes what the server can read, not this architecture: it stores and forwards ciphertext.
← → step · R replay · Space pause
You will learn
- Size the open connections, not just the messages (covered)
- Find each user among many stateful gateways (covered)
- Store before delivering so nothing is lost or reordered (covered)
- Deliver to offline users and drop duplicates (covered)
- Scale to groups, show presence and survive a gateway crash (covered)