RAG fundamentals: giving a model the knowledge it wasn't trained on

A model only knows what it was trained on — knowledge that stops at a cutoff date and never included your internal documents. RAG (Retrieval-Augmented Generation) is the standard fix: retrieve the relevant text at question time and hand it to the model as context, so it answers from your knowledge instead of its memory.

Two phases: index, then query

RAG has an offline half and an online half. Offline, you prepare your knowledge: split documents into chunks, turn each chunk into an embedding (a vector that captures its meaning), and store them in a vector database. This is indexing — you do it once, and refresh it as your data changes.

At query time

• Embed the question — turn the user's query into a vector the same way the chunks were embedded.
• Search — find the chunks whose vectors are most similar to the question's.
• Rerank (optional) — reorder the top candidates so the most relevant context comes first.
• Generate — hand those chunks to the model as context and ask it to answer using them, ideally with citations.

Why RAG instead of fine-tuning or a giant prompt

RAG updates knowledge by updating data, not by retraining — cheaper, faster, and auditable. It grounds answers in sources you can cite, which cuts hallucination. And it sidesteps the context window: you don't stuff everything in, you retrieve only what's relevant to each question.

Where RAG goes wrong

The model can only be as good as the chunks it's handed, so most RAG failures are really retrieval failures — poor chunking, weak embeddings, or no reranking. I go deeper on production-grade retrieval in a separate post; the short version is that getting retrieval right is most of the battle, and generation largely takes care of itself.

RAG doesn't make the model smarter — it makes it informed. The hard part isn't the generation; it's retrieving the right context to generate from.