Building a retrieval-augmented generation (RAG) app: the moving parts

Retrieval-augmented generation (RAG) lets a language model answer using your data instead of only what it learned during training. The idea is simple; doing it well takes care.

The pipeline

1. Ingest — load documents and split them into chunks.
2. Embed — turn each chunk into a vector with an embedding model.
3. Store — keep vectors and metadata in a vector index.
4. Retrieve — find the chunks most relevant to the user question.
5. Generate — pass the question and retrieved context to the model.

def answer(question: str) -> str:
    chunks = index.search(embed(question), top_k=5)
    context = "\n\n".join(c.text for c in chunks)
    prompt = f"Answer using only this context:\n{context}\n\nQuestion: {question}"
    return llm.complete(prompt)

Where demos break in production

• Chunking — chunks that are too large or too small hurt retrieval quality.
• No evaluation — build a small test set of questions and check retrieval and answers on every change.
• Access control — filter retrieved content by what the current user is allowed to see.
• Stale data — plan how documents are re-ingested when sources change.

Start with a simple pipeline, measure it, and add complexity (hybrid search, re-ranking) only where the measurements justify it.