What is RAG? How to Stop Your AI From Making Things Up

ByMatt Ortiz

Large language models hallucinate. Studies from Stanford and other leading research institutions put the hallucination rate of production LLMs anywhere between 3% and 27% depending on the task — a range that makes any enterprise deployment genuinely risky. Retrieval-Augmented Generation (RAG) is the engineering pattern that tames this problem by grounding model outputs in documents you control.

In this guide you will learn exactly how RAG works under the hood, why it has become the default architecture for knowledge-intensive AI applications, how to choose the right vector database, and how to build a working pipeline in Python. Whether you are evaluating RAG for the first time or looking to sharpen an existing implementation, every section delivers something concrete.

Let’s start with the simplest possible explanation and then work inward toward the details that actually matter at build time.

What Is Retrieval-Augmented Generation?

The Core Idea in One Paragraph

RAG combines two subsystems: a retrieval engine that fetches relevant passages from a document store, and a generation engine (the LLM) that drafts an answer using those passages as context. Without RAG, the model answers from memory alone — whatever patterns it absorbed during training. With RAG, it answers from memory plus live evidence. The difference is enormous when your knowledge base changes faster than you can afford to retrain a model.

Why This Matters Now

The research paper that coined the term — Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (Facebook AI, 2020) — demonstrated that RAG models outperformed pure parametric models on open-domain question answering without any fine-tuning. Since then, the pattern has moved from research curiosity to production standard. A 2024 survey by Databricks found that RAG was the most commonly deployed LLM architecture among enterprise teams, ahead of fine-tuning and prompt engineering alone.

For Python developers in particular, the tooling has matured rapidly. Libraries like LangChain, LlamaIndex, and — as we will explore — MongoEngine with MongoDB Atlas Vector Search make it possible to stand up a production-grade RAG pipeline in an afternoon.

The Problem RAG Solves

Every LLM has a knowledge cutoff. GPT-4’s training data ends in early 2024; your company’s internal documentation does not exist in it at all. Fine-tuning can inject some domain knowledge, but it is expensive, slow, and the updated knowledge bakes in statically — it goes stale again the moment documents change. RAG sidesteps all of this. Your retrieval index is the knowledge store; the model is the reasoning engine. Update the index, and the model automatically reasons over fresh information.

How RAG Works: A Step-by-Step Breakdown

Step 1 — Indexing Your Documents

Before a user ever sends a query, you preprocess your document corpus offline. The pipeline looks like this:


  1. Load raw documents (PDFs, Markdown files, database records, web pages).

  2. Chunk them into manageable pieces — typically 256 to 512 tokens, with some overlap to preserve context across chunk boundaries.

  3. Pass each chunk through an embedding model (e.g., OpenAI’s text-embedding-3-small, or an open-source alternative like BGE or E5) to produce a dense vector representation.

  4. Store each chunk’s text alongside its vector in a vector database.

The embedding model is doing something remarkable: it is compressing the semantic meaning of a paragraph into a list of ~1,500 floating-point numbers. Chunks that discuss similar ideas end up numerically close to each other in this high-dimensional space — and that proximity is what makes fast semantic search possible.

Step 2 — Retrieval at Query Time

When a user submits a question, your application:


  1. Embeds the query using the same embedding model used during indexing.

  2. Runs an approximate nearest-neighbor (ANN) search against the vector index to find the k most similar chunks (typically k = 3–10).

  3. Optionally re-ranks results with a cross-encoder model for higher precision.

This step typically completes in under 100 milliseconds for indexes of several million vectors — fast enough to be invisible to end users. For more on how vector databases handle this efficiently, see Vector Databases for AI Apps.

Step 3 — Augmented Generation

The retrieved chunks are injected into the LLM’s prompt as context, usually between a system instruction and the user’s question. The model is instructed to answer only from the provided context, cite sources when possible, and say “I don’t know” if the context does not cover the question. This prompt engineering is often called the RAG prompt template, and getting it right is one of the highest-leverage optimisations available.

RAG vs. Fine-Tuning: Choosing the Right Approach

This is the question developers ask most. The honest answer is that they solve different problems and are often used together. The table below captures the key trade-offs:

Factor

RAG

Fine-Tuning

Knowledge update

Swap or add documents anytime

Requires retraining the model

Cost

Lower — no GPU training runs

High — compute-intensive

Hallucination risk

Reduced — answers grounded in docs

Still present; model may confabulate

Best for

Dynamic, up-to-date knowledge bases

Specific tone, style, or domain jargon

Transparency

High — sources are citable

Low — knowledge is baked in

When to Use RAG

RAG is the right default whenever:

  • Knowledge changes frequently — news feeds, product catalogues, internal wikis, regulatory documents.
  • Source attribution matters — legal, medical, and financial applications often require citations.
  • Budget is constrained — running a vector search costs a fraction of a GPU training run.
  • Time-to-production is critical — a RAG pipeline can be prototype-ready in hours, not weeks.

When Fine-Tuning Adds Value

Fine-tuning shines when you need the model to adopt a specific style, tone, or output format — for example, generating code in a proprietary DSL, writing in a company’s exact voice, or reliably producing structured JSON without extensive prompt engineering. Many mature AI products combine both: fine-tune for style, RAG for factual grounding.

If you are exploring AI tooling more broadly, the guide on AI Coding Assistants for Python Developers covers how modern coding tools use similar retrieval patterns to surface relevant code context.

Building a RAG Pipeline in Python

Choosing Your Stack

A minimal production RAG stack has four components:

  • Embedding model — OpenAI text-embedding-3-small (cost-effective), Cohere Embed v3 (multilingual), or a self-hosted BGE model (privacy-sensitive data).
  • Vector store — MongoDB Atlas Vector Search, Pinecone, Weaviate, or pgvector. MongoDB’s advantage is that you can store documents, metadata, and vectors in one place — no data duplication.
  • LLM — GPT-4o, Claude 3.5 Sonnet, Llama 3, or any model accessible via an API.
  • Orchestration library — LangChain, LlamaIndex, or a thin custom wrapper if you want full control.

A Minimal Working Example

The following pseudocode outlines the core logic. A complete, runnable version is available in the MongoEngine documentation.

# 1. Chunk and embed documents

chunks = split_into_chunks(raw_docs, size=512, overlap=50)

vectors = embedding_model.encode(chunks)

collection.insert_many([{‘text’: c, ’embedding’: v} for c, v in zip(chunks, vectors)])

# 2. At query time

query_vec = embedding_model.encode([user_query])[0]

results = collection.aggregate([vector_search_stage(query_vec, k=5)])

# 3. Augment and generate

context = ‘\n\n’.join([r[‘text’] for r in results])

prompt = RAG_TEMPLATE.format(context=context, question=user_query)

answer = llm.complete(prompt)

Key Parameters to Tune

Three parameters have the biggest impact on RAG quality:

  • Chunk size — smaller chunks (128–256 tokens) improve retrieval precision; larger chunks (512–1024 tokens) preserve more local context for the LLM. Test both against your evaluation set.
  • k (number of retrieved chunks) — more chunks give the LLM more information but increase latency and can dilute the signal. Start at k=5 and tune.
  • Retrieval strategy — pure vector search, hybrid search (vector + BM25 keyword), or a two-stage approach with re-ranking. Hybrid search typically beats pure vector search on short, keyword-rich queries.

Advanced RAG Patterns and Optimisations

HyDE — Hypothetical Document Embeddings

A clever trick: instead of embedding the raw user query, you ask the LLM to generate a hypothetical answer first, then embed that hypothetical answer. Because the hypothetical answer is written in the same style as your indexed documents, it often retrieves more relevant chunks than the terse original query. Benchmarks from the original HyDE paper (Gao et al., 2022) showed retrieval accuracy improvements of up to 15% on open-domain QA tasks.

Contextual Compression

Raw retrieved chunks often contain irrelevant sentences. Contextual compression passes each retrieved chunk through a small LLM call that strips out the irrelevant parts before inserting the chunk into the final prompt. This reduces prompt length, lowers cost, and often improves answer quality because the LLM has less noise to reason through. LangChain ships a

ContextualCompressionRetriever that implements this pattern out of the box. Research published by Anthropic and others suggests that reducing context noise can improve factual accuracy by 8–12% on benchmark tasks.

For a deeper dive into embedding models and vector indexing algorithms, the Pinecone Learning Center offers one of the clearest technical explanations available.

Agentic RAG

Standard RAG retrieves once per query. Agentic RAG gives the LLM the ability to decide when to retrieve, what to retrieve, and whether to retrieve again after inspecting an initial answer. This is implemented via tool calling: the model calls a “search” function, inspects the results, and either generates a final answer or calls the search function again with a refined query. Agentic RAG handles multi-hop questions — questions that require synthesising information from several non-adjacent documents — far better than single-shot retrieval.

Evaluating Your RAG Pipeline

The RAGAS Framework

You cannot improve what you do not measure. RAGAS (Retrieval-Augmented Generation Assessment) is an open-source framework purpose-built for RAG evaluation. It measures four dimensions:

  • Faithfulness — does the answer stay within the retrieved context, or does it hallucinate?
  • Answer relevancy — does the answer actually address the question?
  • Context precision — are the retrieved chunks truly relevant to the question?
  • Context recall — did the retriever surface all the chunks needed to answer the question?

Running RAGAS on a golden dataset of 100–200 hand-curated QA pairs before and after each system change gives you a reliable signal about whether your changes helped. Teams that measure RAG performance systematically tend to converge on production-quality systems in weeks rather than months.

RAGAS scores can also be integrated directly into CI/CD pipelines. If a change causes faithfulness to drop below your threshold, the deployment is blocked — the same way a failing unit test blocks a code merge. For authoritative benchmarking methodology, BEIR (Benchmarking IR) provides the standard benchmark suite used in academic RAG research.

Common Failure Modes

Understanding where RAG pipelines typically break saves significant debugging time:

  • Retrieval miss — the correct chunks are not returned. Root cause: poor embedding model, overly large chunks, or a mismatch between query language and document language. Fix: switch to a better embedding model or add hybrid search.
  • Context overflow — too many chunks exceed the LLM’s context window or distract it from the relevant information. Fix: reduce k or add contextual compression.
  • Prompt leakage — the LLM ignores the retrieved context and answers from its parametric memory. Fix: strengthen the system prompt instruction to answer only from the provided context.

Frequently Asked Questions

What is RAG in simple terms?

RAG is a technique where an AI model searches a document library for relevant passages before generating an answer, rather than relying solely on what it learned during training. Think of it like an open-book exam: the model can look things up rather than recite from memory. This makes answers more accurate and up to date.

How is RAG different from a standard chatbot?

A standard chatbot answers from the patterns it learned during training and has no access to external information. A RAG-powered chatbot retrieves relevant passages from a specified document corpus at query time and grounds its answer in that evidence. The result is far fewer hallucinations and the ability to keep knowledge current without retraining the model.

Do I need a vector database for RAG?

For any non-trivial document corpus (hundreds of documents or more), yes. A vector database stores dense embedding vectors and executes approximate nearest-neighbor search efficiently — something a relational database or plain file system cannot do at speed. For small prototypes, an in-memory library like FAISS works. For production, a managed service like MongoDB Atlas Vector Search is the practical choice.

What embedding model should I use for RAG?

For general English text, OpenAI’s text-embedding-3-small offers an excellent cost-performance balance. For multilingual use cases, Cohere Embed v3 or the open-source E5-multilingual models are strong choices. The most important rule: use the same embedding model at indexing time and at query time. Mixing models breaks semantic alignment and degrades retrieval quality dramatically.

Can RAG completely eliminate hallucinations?

No, but it reduces them substantially. The LLM can still hallucinate if the retrieved context is ambiguous, if the model drifts from the context prompt, or if the question has no good answer in the document corpus. Robust RAG systems pair retrieval with strict prompt instructions, a confidence threshold, and a fallback response when context coverage is insufficient.

Wrapping Up

Three things are worth carrying away from this guide. First, Retrieval-Augmented Generation solves the two biggest production LLM problems simultaneously — knowledge staleness and hallucination — without the cost and complexity of retraining. Second, the architecture is modular: you can improve retrieval quality, generation quality, and evaluation independently, which means iterative improvement is tractable. Third, the Python ecosystem has matured to the point where a working RAG pipeline is an afternoon’s work, not a research project.

If you are ready to build, the logical next step is to spin up a MongoDB Atlas cluster, enable Vector Search, and wire it to your embedding model of choice. The MongoEngine documentation walks through every step in detail.

Similar Posts