CONCEPT · 60 minutes
Design a streaming AI chat backend using an Actix actor model for per-connection isolation — grounded in the rag-engine-rs streaming architecture.
When I first added streaming to rag-engine-rs, I expected it to be a minor change — swap a blocking response for a streaming one and call it done. It was not. Within the first hour of testing I had three distinct failure modes that did not exist in the request-response version: a slow client that stopped reading mid-stream was holding a thread open indefinitely; a language model timeout mid-stream left the client with a partial response and no indication it had ended; and sending a second request from the same client while the first was still streaming caused a resource leak because I had no isolation between connections.
Every one of those failures had the same root cause: I was treating streaming like a slow request-response. It is not. A streaming backend is stateful and long-lived per connection. The architecture must reflect that.
The core difference: in a standard API, a request arrives, you compute a response, you close the connection. Total duration: milliseconds to low seconds. In a streaming chat backend, a connection opens when the user sends a message, tokens arrive from the language model one by one and are forwarded to the client as they are generated, and the connection closes when the model finishes — which can take 10–30 seconds for a long response. Multiple users create multiple simultaneous long-lived connections, all generating tokens at their own pace.
Think of it like a radio broadcast versus a fax machine. A fax (request-response) sends a complete document end-to-end and then stops. A radio broadcast (streaming) continuously emits content that arrives at each listener's receiver independently — a slow listener does not pause the broadcast.
The key architectural shift: Each connected client needs its own isolated execution context. One slow client must not block other clients from receiving their tokens. The isolation boundary must be enforced at the architecture level, not by hoping clients behave well.
This module walks through the streaming backend as I built it in rag-engine-rs: the Actix actor model for per-connection isolation, the SSE vs WebSocket protocol choice, the streaming pipeline from language model to client, backpressure handling, and the scale strategy for a stateful system.
The obvious move is a thread pool — one thread per connection. It works fine at low concurrency and breaks badly at high: threads are expensive, context-switching overhead compounds, and one thread blocked on a slow client starves the others. If you've tried this, you've probably hit the wall.
The actor model is the right abstraction for a streaming chat backend. An actor is a lightweight concurrent unit that has its own state, processes messages one at a time, and communicates with other actors only by sending messages — never by sharing memory.
In rag-engine-rs, each connected client maps to one Actix actor. That actor owns the language model request, receives each generated token, and sends it down the connection. When the client disconnects, the actor stops. One slow client's actor being blocked never touches another client's actor.
flowchart TD
A[Client A] <-->|SSE connection| AA["Actor A\n(owns LM request A\nqueues tokens for A)"]
B[Client B] <-->|SSE connection| BB["Actor B\n(owns LM request B\nqueues tokens for B)"]
C[Client C] <-->|SSE connection| CC["Actor C\n(owns LM request C\nqueues tokens for C)"]
AA --> LM[Language Model API]
BB --> LM
CC --> LM
style AA fill:#3b82f6,color:#fff
style BB fill:#3b82f6,color:#fff
style CC fill:#3b82f6,color:#fff
style LM fill:#d97706,color:#fffEach actor is independent. Actor A crashing does not affect Actor B or Actor C. The language model API is shared, but each actor's connection to it is separate — Actor A's timeout does not cancel Actor B's in-flight request.
Why not shared threads? A thread pool with one thread per connection works at very low concurrency. At 1,000 concurrent streaming connections, threads become expensive — each thread carries a stack of hundreds of kilobytes. Actors are far lighter: Actix can run hundreds of thousands of actors on a small number of OS threads using cooperative scheduling.
Each actor runs the same pipeline for its client:
The actor's job is simple: start the request, forward each token, signal when done. The complexity is all in what happens when things go wrong — which is the real design challenge for streaming systems.
Two protocols are commonly used for server-to-client streaming: Server-Sent Events (SSE) and WebSockets. For a chat backend, the choice matters.
Server-Sent Events (SSE): a simple HTTP-based protocol where the server sends a stream of data: lines over a long-lived HTTP connection. Clients subscribe with a standard EventSource API. SSE is unidirectional (server-to-client only) and reconnects automatically. It works over standard HTTP/1.1 — no protocol upgrade required.
WebSockets: a full-duplex protocol that upgrades an HTTP connection to a persistent bidirectional channel. Clients can send messages at any time; the server can push messages at any time. WebSockets are lower overhead per message once established, but require a protocol upgrade and are not automatically reconnecting.
For a chat backend, SSE is often the right default:
When WebSockets become necessary: If your design requires the client to send messages while a previous response is still streaming (interrupting a generation, for example), SSE cannot express that — you need a bidirectional channel. WebSockets also become preferable when the overhead of a new HTTP connection per message is measurable at scale.
Before the scale section, verify you can answer these from memory:
Reflective prompt: In rag-engine-rs, each actor owns a single streaming request. What happens if the language model API rate-limits the backend mid-stream — after tokens have already started arriving? How would your design handle this without silently truncating the client's response? Write two sentences before moving to the exercise.
A streaming backend is stateful: each client's connection is pinned to a specific server instance for the duration of the stream. This is the fundamental scaling challenge that does not exist in stateless request-response backends.
Single-server scaling: actor-based backends scale vertically well. A single Actix server can handle thousands of concurrent streaming connections on modest hardware. The bottleneck is language model API concurrency limits, not the server itself.
Horizontal scaling: when you add more servers, incoming connections must be routed to the same server for the duration of the stream. Two options:
In rag-engine-rs, I use sticky sessions — the system handles single-server scale, and the complexity of distributed fan-out is not justified by the usage pattern. If I were designing for thousands of concurrent users across multiple servers, I would add a Redis Pub/Sub layer.
The backpressure failure mode: If a client stops reading mid-stream (slow network, tab backgrounded), the server's send buffer fills. Without backpressure handling, the actor blocks waiting for the buffer to drain — which holds a language model connection open and consumes resources. Handle this by setting a bounded buffer on the SSE sender and disconnecting slow clients that have not read in a configurable timeout period.
Apply the streaming backend design to a concrete problem.
The task: Design a streaming AI tutoring backend. Students ask math questions in plain English; the backend streams a step-by-step explanation back. The system should handle 500 concurrent student sessions. Each session lasts an average of 45 seconds. The student can ask follow-up questions mid-session.
<CodeExample title="Design Prompt Starter — Streaming Tutoring Backend" language="markdown" code={`## Requirements Functional:
Non-functional:
Apply streaming backend design principles to realistic scenarios
1. A streaming backend uses a shared thread pool with one thread per client connection. What failure mode does the actor model prevent?
Correct Answer: B
With a shared thread pool, a client that stops reading mid-stream holds a thread open. Under high concurrency, threads are a scarce resource — slow clients reduce the pool available for fast clients. The actor model uses cooperative scheduling: each actor runs on shared OS threads but yields when waiting, so slow clients do not starve fast ones.
2. A student can send follow-up questions while the previous answer is still streaming. Should the backend use SSE or WebSocket?
Correct Answer: B
SSE is server-to-client only. If the student needs to send a message while a response is streaming, SSE cannot express that on the same connection. WebSocket is bidirectional: the client can send a follow-up question at any time, and the server can push tokens at any time. For interactive chat with mid-stream follow-ups, WebSocket is the right choice.
3. A client stops reading tokens mid-stream (phone goes to sleep). What is the correct behavior for the streaming actor?
Correct Answer: B
Buffering indefinitely holds a language model connection open and consumes memory for the backlog. Pausing the LM call is not possible mid-stream. Sending [DONE] silently truncates the response. The correct response is backpressure: when the write buffer fills, start a timer. If the client has not read within the timeout, close the connection cleanly. The client must reconnect and start a new session.
4. Why are sticky sessions necessary for a horizontally scaled streaming backend?
Correct Answer: B
A streaming connection is stateful: the actor holds the conversation history and the in-flight LM request. If a reconnect or follow-up request hits a different server, that server has no actor for that session. Sticky sessions ensure all requests for a session hit the same server. The alternative — Redis Pub/Sub fan-out — lets any server serve any client but adds significant complexity.
5. A language model API call takes 25 seconds. If you do not set a per-request timeout, what is the failure mode?
Correct Answer: B
Without an explicit timeout, a hung LM API call holds the actor — and its resources — open indefinitely. The client sees no tokens and no error; from their view, the server has frozen. The actor is effectively leaked until the server restarts. Always set an explicit timeout on the LM API call; when it fires, send an error event to the client and close the connection cleanly.
6. Where should conversation history be stored so a student can resume a session after a browser refresh?
Correct Answer: B
In-actor state is lost when the actor stops (disconnect, server restart). Browser localStorage cannot survive a session across different devices. Postgres works but is heavier than necessary for conversational data with a natural expiry. Redis with a TTL is the right fit: fast reads and writes, natural expiry matching the session timeout, and survives actor restarts while the TTL is active.