CONCEPT · 75 minutes
End-to-end design of a retrieval-augmented generation system — document ingestion pipeline, chunking strategy, embedding model choice, pgvector index tuning, hybrid BM25 + semantic retrieval, and streaming the grounded answer. Grounded in the rag-engine-rs architecture.
When I started building rag-engine-rs, I expected the hard part to be the language model integration. It was not. The language model call turned out to be the easiest component — a single HTTP request, a streaming response, some token counting.
The hard parts were everything else: how to ingest documents at scale without blocking the upload endpoint; how to choose chunk sizes that made retrieval actually useful; how to combine keyword search and vector search without writing a complex ranking algorithm; and how to stream a response that was grounded in retrieved context without the user waiting five seconds for the first word.
Those are distributed systems problems. That is why "design a RAG system" is a genuine system design problem — not a language model question.
Think of a RAG system like a research assistant who has read every document in a library. When you ask a question, the assistant does two things: finds the most relevant pages in the library, then composes an answer using those pages. Your design controls both steps — how the library is organized (ingestion and indexing) and how the assistant searches and answers (retrieval and generation).
The Two-Phase Architecture: Every RAG system has two separate phases — offline ingestion (chunking, embedding, indexing) and online query (retrieval, re-ranking, generation). Design them independently. Their failure modes, scaling bottlenecks, and latency budgets are completely different.
Here is the end-to-end pipeline this module walks through:
flowchart LR
subgraph Offline ["Offline — Ingestion"]
A[Document Upload] --> B[Parse and Chunk]
B --> C[Embed Chunks]
C --> D[(pgvector Index)]
end
subgraph Online ["Online — Query"]
E[User Question] --> F[Embed Query]
F --> G[BM25 + Semantic Retrieval]
D --> G
G --> H[Re-rank Top-N]
H --> I[LLM Generation]
I --> J[Streaming Response]
end
style Offline fill:#dbeafe,stroke:#3b82f6
style Online fill:#dcfce7,stroke:#16a34aThe key insight from building rag-engine-rs: these two phases have almost nothing in common. The ingestion phase is a batch pipeline that runs once per document — throughput matters, latency does not. The query phase runs on every user request — latency matters above all. Design each path for its own constraints.
This module walks through the full RAG pipeline as I implemented it in rag-engine-rs: document ingestion, chunking, embedding, pgvector indexing, two-stage hybrid retrieval, and streaming the generated answer. Each section describes the design choice, the trade-off it resolves, and what breaks at scale.
The ingestion pipeline converts raw documents (PDF, HTML, plain text) into vector chunks stored in Postgres with pgvector. It runs offline — decoupled from the online query path.
Each step is a distinct concern with its own failure modes. The critical design decision: run this pipeline asynchronously, not inline with the document upload.
The Synchronous Ingestion Mistake: A common first attempt is to embed documents synchronously in the upload handler. For a document with 200 chunks, that means 200 sequential embedding API calls inside a single HTTP request. One timeout fails the entire upload. The correct pattern: accept the upload, return 202 Accepted, queue the ingestion job asynchronously.
Chunking splits a document into segments small enough to retrieve meaningfully. The core trade-off:
In rag-engine-rs, I use 512-token chunks with 64-token overlaps. The overlap prevents retrieval from missing relevant content that spans a chunk boundary.
Each chunk is embedded into a dense vector representation using an embedding model. The vector captures the semantic meaning of the text — similar meanings produce similar vectors.
On Embedding Model Choice: Do not pin a design to a specific embedding model ID from memory — model availability and best practices change. Instead, describe the trade-off: larger embedding dimensions give richer representations at higher storage and query cost. For most production RAG deployments, a well-tuned mid-size embedding model outperforms a large model with poor chunking.
The embedding step is the ingestion bottleneck. For large corpora, batch embedding calls and parallelize across workers. Store the vector alongside the chunk text and source metadata in Postgres.
pgvector extends Postgres with a vector type and approximate nearest-neighbor (ANN) search. It runs inside your existing Postgres instance — no separate vector database required.
pgvector supports two index types:
HNSW (Hierarchical Navigable Small World): builds a navigable graph where each node is connected to its approximate nearest neighbors at multiple granularity levels. Queries traverse the graph to find nearest neighbors efficiently.
IVFFlat (Inverted File with Flat Compression): partitions vectors into clusters (lists), then searches only the nearest clusters at query time.
nlist (cluster count) at build time — too few clusters means searching too broadly, too many means missing relevant vectorsThe Index Tuning Reality: Index parameters (m, ef_construction, ef_search for HNSW; nlist, nprobe for IVFFlat) interact with your corpus size and embedding dimensions. The right values require benchmarking on your actual data. When you present the design, name the parameters and the trade-off they control — do not claim a specific value is always correct.
A vector similarity query retrieves the top-K chunks nearest to the query embedding:
Pure semantic search has a well-known failure mode: exact-match queries. Product names, error codes, version numbers, and proper nouns do not survive the embedding averaging process intact. A query for "error code E4021" will return conceptually similar chunks, but may miss the chunk that actually contains "E4021."
BM25 (Best Match 25) is a classic keyword-relevance ranking algorithm that excels at exactly these cases. Combining BM25 and semantic search — hybrid retrieval — gives you the best of both signals.
Stage 1: Retrieve candidate chunks from both BM25 and semantic search independently, taking the union. Stage 2: Re-rank the union using a cross-encoder model that scores each chunk against the query independently, then return the top-N.
A lightweight alternative to a cross-encoder re-ranker is Reciprocal Rank Fusion (RRF). RRF combines rankings from multiple retrieval systems without requiring a separate model:
When to Use RRF vs Cross-Encoder: RRF adds negligible latency and no additional model calls — good for low-latency requirements. A cross-encoder re-ranker (a model that scores each chunk-query pair) improves recall quality further but adds latency proportional to the number of candidates. In rag-engine-rs, I use RRF for the initial fusion and reserve cross-encoder re-ranking for the cases where answer quality is critical.
Before the streaming section, verify you can answer:
After retrieval and re-ranking, the top-N chunks become the context injected into the generation prompt. The language model generates an answer grounded in this context.
A grounded generation prompt has three parts: system instructions, the retrieved context, and the user's question.
A generation over retrieved context can produce 400–800 tokens. Without streaming, the user waits for the full generation before seeing anything — often several seconds. With streaming, the first token appears within a fraction of a second of the model starting, and the user reads while generation continues.
In rag-engine-rs, I use Actix's streaming actor model to push each token to the HTTP response as it arrives from the language model API. Each connection has its own isolated actor, so a slow client does not block other connections.
The pattern on the client side is a Server-Sent Events (SSE) stream or WebSocket connection that receives chunks of the response text as the model generates them.
The Streaming Latency Model: Total generation time is unchanged by streaming. What changes is perceived latency — users begin reading immediately. For a response that takes 4 seconds to generate fully, streaming typically delivers the first token in under 500ms. That difference, between a blank screen for 4 seconds versus text appearing immediately, is the primary UX motivation for streaming in any generation system.
Think about the RAG pipeline as a whole — from document upload through streaming the answer.
Identify the two components in this pipeline where a failure would be silent — where the system continues operating but answer quality degrades invisibly. How would you detect each failure in production? What metric or log event would tell you quality had degraded before a user complained?
Write down two components and two detection strategies before checking the quiz answers. Silent quality degradation is what separates a demo RAG system from a production one.
Apply the RAG design principles to a concrete system. The goal is a design you can defend, not a perfect one.
The task: Design a RAG system for a software company's internal engineering wiki — thousands of Markdown documents covering architecture decisions, runbooks, and onboarding guides. Engineers query it in plain English: "How do we deploy to staging?" or "What are the retry limits on the task queue?"
Work through the six-step framework for this system, then check your design against the reference.
<CodeExample title="Design Sketch Starter — Engineering Wiki RAG" language="markdown" code={`## Requirements
Apply RAG design principles to realistic design scenarios
1. Why choose 512-token chunks over 2,048-token chunks? What is the core trade-off?
Correct Answer: B
Chunking is a precision-recall trade-off. Smaller chunks produce embeddings that represent narrower concepts — they rank well when the query matches the chunk closely. Larger chunks include surrounding context but the embedding averages over more content, diluting the match signal. The right size depends on your document structure and query patterns.
2. Someone proposes pure semantic search for a RAG system over a support knowledge base. What failure mode should you flag?
Correct Answer: B
Pure semantic search struggles with exact-match queries — error codes, product names, version strings, and proper nouns do not survive the embedding averaging process intact. BM25 excels at these exact matches. Hybrid retrieval combines both signals: BM25 for exact-match recall, semantic search for conceptual similarity.
3. You are designing a pgvector deployment for a 10-million-chunk knowledge base. What is the trade-off between HNSW and IVFFlat when choosing an index type?
Correct Answer: A
HNSW (Hierarchical Navigable Small World) builds a graph-based index that achieves high recall at low query latency, but the graph structure requires more memory. IVFFlat partitions vectors into clusters (inverted file) and searches only the nearest clusters — lower memory but recall depends on a well-chosen nlist value at build time. For a 10M-chunk corpus with latency requirements, HNSW is typically preferred if memory budget allows.
4. Why is streaming the generated answer important in a RAG system's user experience?
Correct Answer: B
Long generations (500+ tokens) can take several seconds to complete. Without streaming, users see a blank screen until generation finishes. Streaming emits tokens as they are generated — the time to first token is typically under a second, which dramatically improves perceived latency even though total generation time is unchanged.
5. Someone designs a RAG ingestion pipeline that embeds each chunk synchronously in the same HTTP request that uploads the document. What is the problem?
Correct Answer: B
Embedding a document with hundreds of chunks requires hundreds of embedding API calls. Doing this synchronously inside the upload HTTP handler creates a slow, fragile endpoint — a single embedding API timeout fails the entire upload. The correct pattern is to accept the upload immediately, return a 202 Accepted, and process chunking and embedding asynchronously in a background job queue.
6. After running hybrid retrieval, you have 20 candidate chunks. Why add a re-ranking step before passing context to the language model?
Correct Answer: B
Bi-encoder retrieval (BM25 + semantic) is fast but approximate — it ranks chunks relative to each other using embedding distance. A cross-encoder re-ranker scores each chunk against the query independently with a richer attention mechanism, producing more accurate relevance scores. Re-ranking the top 20 to select the top 5 improves answer quality at the cost of added latency.