Making RAG better: a menu of techniques that actually move the needle

Basic RAG — chunk, embed, retrieve, generate (see the fundamentals post) — gets you a demo that works about 70% of the time. Closing the rest is a stack of techniques, and because the model can only be as good as the context it's handed, most of them target retrieval. Here's the toolkit, organised by where in the pipeline it acts.

Index-time: get the corpus right

• Document preprocessing — clean the source, strip boilerplate and navigation, and preserve structure (headings, tables); noise in, noise out.
• Smarter chunking — respect document structure, tune chunk size, and use overlap so ideas aren't split mid-thought; chunking is a product decision, not a default.
• A better encoder — a stronger or domain-tuned embedding model lifts every downstream retrieval for free.
• Hierarchical indexing — index summaries over chunks so you can retrieve at the right level of granularity for the question.

Query-time: retrieve better

• Query rewriting — reformulate a vague or messy user query into something retrievable before you search.
• Query expansion — search several phrasings (or a hypothetical answer, HyDE) to catch relevant chunks the literal query would miss.
• Hybrid search — combine keyword (BM25) and dense retrieval to get both exact-match precision and semantic recall.
• Re-ranking — a cross-encoder reorders the top candidates so the best context comes first; often the single highest-ROI change you can make.

Advanced architectures

• Graph RAG — retrieve over a knowledge graph, so questions that need multiple connected facts (multi-hop, relationship-heavy) can actually be answered.
• Agentic RAG — let an agent decide whether and what to retrieve, run multiple searches, and verify its own answer before responding.

How to actually apply this

Don't bolt on everything at once — that's how you get a slow, undebuggable pipeline. Measure where retrieval is failing, then add the one technique that targets that failure and measure again. In practice, better chunking and a reranker are the usual first wins; reach for Graph or Agentic RAG only when simpler methods hit a real ceiling.

Improving RAG isn't about adding every trick — it's about finding where retrieval breaks, and fixing exactly that.