Design: a chat system

Key points

• Clients hold a WebSocket to a gateway. A registry maps user to gateway, and fan-out goes through a broker rather than local memory.
• The server assigns the sequence number, never the client. Client clocks disagree; a per-conversation counter never does.
• Store messages keyed by (chat_id, seq), so one conversation lives on one shard and history is a single range scan.
• An offline device gets a push notification instead, and on reconnect asks for everything after the last seq it saw.
• Delivered and read receipts are two more small writes per message - in a group chat they easily outnumber the messages themselves.
• Group chat is fan-out on write into member inboxes up to some size; a huge channel pulls on read instead, exactly like a news feed.

Example

// The gateway owns the sockets. Ordering and fan-out happen behind it.
@MessageMapping("/chat.send")
void send(Incoming in, Principal user) {

    long seq = sequences.next(in.chatId());       // server-side, per conversation
    Message saved = messages.append(in.chatId(), seq, user.getName(), in.body());
    // INSERT INTO message (chat_id, seq, sender_id, body, created_at) VALUES (...)
    // shard key = chat_id, primary key = (chat_id, seq) -> history is a range scan

    for (String member : members.of(in.chatId())) {
        String gateway = registry.gatewayFor(member);        // Redis: user -> node
        if (gateway != null) {
            broker.publish("gateway." + gateway, saved);     // that node holds the socket
        } else {
            push.send(member, saved.preview());              // APNs or FCM instead
        }
    }
}

// Reconnect is never "give me the last 50". It is always "everything after my seq",
// which is what makes a dropped socket harmless.
@GetMapping("/chats/{chatId}/messages")
List<Message> since(@PathVariable String chatId, @RequestParam long afterSeq) {
    return messages.range(chatId, afterSeq, 200);
}

Server-assigned sequence numbers plus a resume-from-seq API keep delivery correct when sockets drop.