Designing a Real-Time Messaging System

When I built the cross-agent communication layer for workflow-engine-rs, the core problem was not sending a message — it was delivering that message to every subscriber who was listening, without one slow subscriber blocking the others.

This is the fan-out problem: when a message is sent to a channel, every connected subscriber must receive it. The naive implementation — the thing I'd write on a first pass, and you probably would too — is to iterate over subscribers and write to each one. It has a critical flaw: a slow subscriber (one whose write buffer is full, one on a poor network) blocks the iteration. While you are waiting for the slow client, no other subscriber receives the message. The more subscribers you have, the more catastrophic a single slow client becomes.

The fix is the pattern I use in workflow-engine-rs: per-connection channels. Each subscriber gets its own mpsc (multi-producer, single-consumer) channel. When a message needs to be delivered, the dispatcher sends to all channel senders simultaneously — a non-blocking operation. Each subscriber's receiver independently pulls from its own channel. A slow subscriber's channel fills up; the dispatcher is not blocked.

Think of it like a radio station and its listeners. The station broadcasts continuously. Each listener's radio has its own receive buffer. A listener who walks into a tunnel (slow connection) misses signals into their buffer, but the station keeps broadcasting for all other listeners. The station is never blocked by any single listener's reception quality.

What This Module Covers

This module walks through the fan-out design with per-connection channels and a connection registry, the protocol choice for real-time delivery, failure modes at scale, and the scale limit that forces a distributed design.

The fan-out pattern has two components: a connection registry that maps channel IDs to a set of sender handles, and a per-connection receive loop that reads from the connection's channel and writes to the WebSocket.

flowchart TD
    M[Incoming Message\nto channel: "room-42"] --> R[Connection Registry\nchannel_id -> set of senders]
    R -->|"send(msg)"| S1["Sender A\n(non-blocking)"]
    R -->|"send(msg)"| S2["Sender B\n(non-blocking)"]
    R -->|"send(msg)"| S3["Sender C\n(non-blocking)"]
    S1 --> Q1["Channel A\n(bounded queue)"]
    S2 --> Q2["Channel B\n(bounded queue)"]
    S3 --> Q3["Channel C\n(bounded queue)"]
    Q1 --> WS1["WebSocket\nClient A"]
    Q2 --> WS2["WebSocket\nClient B"]
    Q3 --> WS3["WebSocket\nClient C"]
    style R fill:#3b82f6,color:#fff
    style S1 fill:#d97706,color:#fff
    style S2 fill:#d97706,color:#fff
    style S3 fill:#d97706,color:#fff

The dispatcher sends to all sender handles for a channel. Each send is non-blocking — it either places the message in the channel's queue or fails immediately if the queue is full. The dispatcher never waits for any subscriber to receive.

Each subscriber's receive loop runs independently on its own task or thread. It reads from the channel and writes to the WebSocket at whatever pace the client can handle.

The try_send call is non-blocking: if the channel is full (the client is slow), the message is dropped for that subscriber without affecting others. This is the right behavior — you accept that slow clients may miss messages rather than block the entire fan-out.

Handling Disconnections

When a client disconnects, you must remove its sender from the registry. Otherwise, subsequent fan-out calls attempt to send to a closed channel, generating errors for every message.

This must run whether the client disconnects cleanly (WebSocket close frame) or abruptly (network drop). In practice: wrap each per-connection task in a cleanup handler that calls unsubscribe when the task exits, for any reason.

A single-server fan-out with per-connection channels scales to tens of thousands of concurrent WebSocket connections on a single server. Memory consumption is the practical limit — each channel queue and connection handle takes memory, and a message being fanned out to N subscribers creates N copies in the channel queues.

The scale wall is reached when your users are distributed across multiple servers. Subscriber A might be connected to Server 1 while Subscriber B is connected to Server 2. A message sent to a channel must reach both — but Server 1's registry has no knowledge of Subscriber B.

The standard solution: a message bus (Redis Pub/Sub, NATS, or Kafka) as a cross-server fan-out layer.

Each server subscribes to the Redis channels its local clients are subscribed to. When a message is published — from any server — Redis delivers it to every subscribed server. Each server then fans out locally to its connected clients. The result: a message sent from any server reaches all subscribers, regardless of which server they are connected to.

Mid-Module Checkpoint

Before the exercise, verify you can answer these from memory:

1. Why does try_send (non-blocking) rather than send (blocking) matter in the fan-out loop?
2. What happens to the connection registry if you forget to call unsubscribe on disconnect?
3. What is the scale limit of a single-server registry, and what does Redis Pub/Sub add?

Apply the fan-out design to a concrete system.

The task: Design a real-time collaborative document editing notification system. When a user saves a change to a document, all other users viewing that document should see a "document updated" notification within 1 second. The system must handle 10,000 active documents with an average of 5 simultaneous viewers each.

<CodeExample title="Design Prompt Starter — Collaborative Editing Notifications" language="markdown" code={`## Requirements

• Notify all viewers of a document when any viewer saves a change
• Delivery within 1 second
• 10,000 active documents, avg 5 viewers each = 50,000 concurrent connections
• At-most-once delivery is acceptable (missed notification = refresh button)