RAG Explained — Retrieval-Augmented Generation Guide for 2026

How RAG works

RAG combines two systems: a retriever that finds relevant information and a generator (the LLM) that produces answers using that information. The standard pipeline:

1. Indexing (offline) — Split your documents into chunks, convert each chunk into a vector embedding, store in a vector database
2. Retrieval (per query) — Convert the user's question into a vector, search for the most similar document chunks (top-k results)
3. Augmentation — Insert the retrieved chunks into the LLM prompt as context: "Based on the following documents: [chunks]. Answer the user's question: [query]"
4. Generation — The LLM generates an answer grounded in the provided context, ideally citing specific sources

This is why it's called Retrieval-Augmented Generation — you're augmenting the LLM's knowledge with retrieved documents at generation time. The model doesn't need to have memorised your data; it reads it fresh each query.

Chunking: the most underrated step

How you split documents into chunks has an outsized impact on RAG quality. Get this wrong and retrieval fails regardless of everything else:

• Chunk size — Too small (50 tokens) loses context. Too large (2000 tokens) dilutes relevance and wastes context window. Start with 200-500 tokens and adjust based on your content type.
• Overlap — Include 10-20% overlap between adjacent chunks so information at chunk boundaries isn't lost.
• Semantic chunking — Split at natural boundaries (paragraphs, sections, headers) rather than fixed character counts. A chunk that cuts a sentence in half is worse than one that respects document structure.
• Metadata enrichment — Attach metadata to each chunk: source document, section title, page number, date. This enables filtered retrieval ("only search documents from 2024") and better citation.
• Parent document retrieval — Embed small chunks for precise matching but return the larger parent document for full context. This combines retrieval precision with generation context.

Improving retrieval quality

The most impactful optimisations for RAG quality, in order of bang-for-buck:

1. Better embedding model — Switching from a basic embedding model to a state-of-the-art one can improve retrieval accuracy by 10-30%. See our vector databases guide for embedding model recommendations.
2. Hybrid search — Combine vector similarity search with traditional keyword search (BM25). Some queries are better served by exact keyword matching; others by semantic similarity. Hybrid captures both.
3. Reranking — After initial retrieval (top-20), use a cross-encoder reranking model to re-score results. This dramatically improves the precision of your final top-5 results.
4. Query transformation — Rewrite the user's query before searching. Techniques: HyDE (generate a hypothetical answer and search for it), query expansion (add synonyms), multi-query (search with several paraphrases and merge results).
5. Metadata filtering — Don't just search semantically — filter by document type, date range, author, or category first. This eliminates irrelevant results that might otherwise score high on semantic similarity.

When to use RAG vs. fine-tuning

RAG and fine-tuning solve different problems:

• Use RAG when — Your data changes frequently. You need source attribution. You want to control exactly what information the model accesses. You have a knowledge base, document collection, or FAQ that the model should reference.
• Use fine-tuning when — You need to change the model's style or behaviour. You want to teach it domain-specific terminology or reasoning patterns. You need consistent formatting that prompt engineering can't achieve. See our fine-tuning guide.
• Use both when — You want domain-specific behaviour AND access to up-to-date information. Fine-tune for style/format, RAG for knowledge.

In practice, RAG is the right choice for 80%+ of "make the AI know about my stuff" use cases. It's cheaper, doesn't require ML expertise, and your data stays up-to-date without retraining.

Common failure modes and fixes

When RAG isn't working well, the problem is almost always in retrieval, not generation:

• "The answer is in my documents but the model says it doesn't know" — The retriever isn't finding the right chunks. Check: is your chunking splitting the relevant passage? Is the query semantically similar to how the information is phrased in the document? Try hybrid search or query transformation.
• "The model makes up information instead of admitting it doesn't know" — Add explicit instructions: "Only answer based on the provided context. If the context doesn't contain the answer, say 'I don't have information about that.'" Also consider reducing temperature to 0.
• "Answers are vague or incomplete" — Your chunks may be too small, losing critical context. Try larger chunks or parent-document retrieval. Also check that you're retrieving enough chunks (try top-10 instead of top-3).
• "Retrieval is slow" — Index size, embedding dimension, and query volume all affect speed. Consider approximate nearest neighbour (ANN) indexes, reducing embedding dimensions, or caching frequent queries.