What Is Retrieval-Augmented Generation (RAG)?

Retrieval-Augmented Generation (RAG) is the pattern that turns a large language model from a closed system trained on a fixed dataset into a system that can ground its answers in current, organization-specific, or otherwise external information. Instead of relying solely on what the model learned during training, a RAG system retrieves relevant documents (or passages, or database records) at query time and provides them as context for the model to reason over. The model then generates a response informed by both its general training and the specific retrieved material. RAG has become the dominant architecture for enterprise AI applications precisely because it solves the two biggest problems pure LLMs have: their knowledge has a cutoff date, and they don’t know anything specific to your organization.

This post walks through what RAG actually is, why it became the default enterprise AI pattern, the components of a RAG system, common implementation patterns, the limitations and failure modes, and how to think about whether RAG is the right architecture for a given AI use case.

What RAG actually does

A pure LLM (GPT, Claude, Gemini, others) generates responses based entirely on the patterns it learned during training. Ask it a question about your company’s internal policies and it has no idea. Ask it about an event that happened after its training cutoff and it doesn’t know. Ask it about a product you launched last week and it’s guessing.

RAG fixes this by inserting a retrieval step before generation. The flow:

• The user asks a question.
• The system searches an external knowledge source (your documentation, your database, your customer support tickets, the public web, whatever’s relevant) for information related to the question.
• The system retrieves the most relevant chunks of information.
• The system constructs a prompt that includes the original question plus the retrieved chunks as context.
• The LLM generates a response that uses the retrieved context to inform its answer.

The user gets a response that’s grounded in actual relevant information rather than the model’s potentially-outdated or generic knowledge. The retrieved chunks can be cited in the response, which gives users a way to verify where the information came from.

Why RAG became the default enterprise pattern

By the mid-2020s, RAG had become the architecture for the majority of enterprise AI deployments that need to access organization-specific information. The reasons compound.

LLM training is expensive and slow. Fine-tuning a frontier-tier model on your organization’s data costs significant compute and produces a model that’s still frozen at the moment of training. New information added next week requires retraining or accepting that the model doesn’t know about it. RAG sidesteps this entirely: update the underlying knowledge source and the model sees the new information at the next retrieval, no retraining required.

RAG works with the best models without modification. You can change which LLM your RAG system uses (GPT-5.5 today, Claude 4.6 tomorrow, Gemini next quarter) without changing your knowledge base or your retrieval pipeline. The flexibility is valuable as the model landscape evolves.

RAG handles citations and grounding naturally. Because the system knows which chunks it retrieved, it can attribute specific claims to specific sources. The user can verify. For compliance-sensitive or fact-critical use cases (medical, legal, financial), citation matters a lot.

RAG separates the "what" from the "how." Your knowledge base contains the authoritative information; the LLM provides the language interface. You can update the knowledge without changing the model, and you can swap models without rebuilding the knowledge.

RAG reduces hallucination on facts it can retrieve. A model asked to answer a question with relevant retrieved context is much less likely to make up incorrect facts than a model relying purely on training data. The grounding isn’t perfect, but it’s substantially better than no grounding.

The components of a RAG system

A working RAG system has several pieces. Building one requires assembling them; using one (through an off-the-shelf product) requires understanding what’s underneath.

The knowledge source. Documents, articles, database records, support tickets, transcripts, whatever the system is supposed to know about. The knowledge source can be a curated set of PDFs, a CMS content database, a Confluence wiki, customer support history, internal documentation, or any combination.

The chunking strategy. Large documents need to be broken into smaller chunks for retrieval. A 200-page PDF can’t be inserted into the model’s context window in one piece, and even if it could, the retrieval would be too coarse. Common patterns: chunks of 500-1,500 tokens, with some overlap between adjacent chunks, broken at natural boundaries (paragraphs, sections) where possible.

The embeddings model. Converts text chunks (and queries) into vector representations that capture semantic meaning. The embeddings model is typically a smaller specialized model (OpenAI’s text-embedding-3, Cohere’s embeddings, open-source sentence-transformers, others). Same model needed for indexing and querying to keep the vector space consistent.

The vector database. Stores the embedded chunks and provides fast similarity search at query time. Pinecone, Weaviate, Chroma, Qdrant, pgvector (Postgres extension), Elasticsearch, and many others. See our vector databases piece for more detail on this layer.

The retrieval logic. When a query comes in, the system embeds the query, searches the vector database for the most-similar chunks, and returns the top N chunks. Modern systems often add re-ranking (a second-pass scoring of the retrieved chunks for better relevance), hybrid search (combining vector search with traditional keyword search), and filtering (applying access controls or metadata constraints to the search).

The prompt construction. Takes the original query plus the retrieved chunks and assembles a prompt for the LLM. Good prompt design tells the model what role to play, how to cite the retrieved information, and what to do if the retrieved information doesn’t actually contain the answer.

The LLM. Generates the response. Any of the major LLMs work; the choice affects cost, latency, and response quality but not the underlying RAG pattern.

The evaluation pipeline. RAG systems need to be measured. Does the retrieval bring back relevant chunks? Does the generation correctly use them? Are there hallucinations on retrieved content? Are there hallucinations beyond what was retrieved? Evaluation is the part most teams under-invest in, and it shows in their system quality.

Common RAG implementation patterns

Several patterns recur across real-world RAG deployments.

Document Q&A. The simplest pattern. Users ask questions about a specific document corpus (a product manual, a policy library, an internal wiki) and the system retrieves relevant passages and answers. This is the canonical RAG use case.

Customer support augmentation. RAG over the company’s support history, knowledge base articles, and product documentation. The system either answers customer questions directly or augments human support agents with retrieved context.

Internal knowledge base. RAG over the company’s internal documentation, meeting notes, project records, and communications. Employees ask questions and get answers grounded in actual organizational knowledge.

Search reinvented. RAG over the company’s content (or the public web, in the case of products like Perplexity) where the LLM rewrites the query, retrieves results, and synthesizes an answer rather than returning a list of links.

Conversational agents with persistent memory. RAG over a user’s conversation history plus other context, letting an agent remember and use prior interactions in future responses.

Specialized vertical applications. Legal-research RAG over case law, medical-literature RAG over journal articles, financial-research RAG over SEC filings and analyst reports.

The patterns aren’t mutually exclusive; many real systems combine several.

The limitations and failure modes

RAG isn’t a complete solution. The limitations matter for any team building or evaluating a RAG system.

Retrieval quality is a hard problem. The retrieval step depends on the query being similar enough to the relevant chunks in vector space. Queries that are phrased differently from the source material, or that depend on synthesizing information across multiple documents, often retrieve the wrong context. Hybrid search, re-ranking, and query rewriting help but don’t fully solve this.

Chunking matters more than people expect. A chunk that splits a key concept across boundaries can prevent retrieval. Chunks that are too large dilute the relevance signal. Chunks that are too small lose context. Getting chunking right for a specific corpus is empirical work.

The model can still hallucinate on top of retrieved content. RAG reduces hallucination on facts that are retrievable; it doesn’t eliminate hallucination. The model may interpret retrieved chunks incorrectly, fill in gaps with plausible-but-wrong information, or assert things the chunks don’t actually say. Evaluation and prompt-design discipline reduce but don’t eliminate this.

Access controls are tricky. In enterprise contexts, different users have access to different documents. A RAG system needs to filter the retrieval based on the requesting user’s permissions, which is operationally non-trivial.

Stale data is invisible. RAG retrieves whatever’s in the knowledge source. If the knowledge source is stale, the responses are stale, and the user has no way to know. Discipline around updating the underlying data is essential.

Cost scales with use. Each query embeds the query, retrieves chunks, constructs a prompt, and generates a response. The cost per query is small but adds up at scale. High-volume RAG systems require cost engineering.

When RAG fits (and when it doesn’t)

RAG is the right answer when:

• The application needs to answer questions about organization-specific or constantly-updated information that the model doesn’t have in training.
• The answers need to be grounded in source material that users can verify.
• The knowledge changes faster than retraining cycles can keep up with.
• Citations matter for trust, compliance, or fact-verification.

RAG is less likely to fit when:

• The application needs reasoning over the entire content rather than retrieved passages. Some questions require synthesizing across many documents that won’t all be retrieved. Larger context windows partially address this but cost more.
• The application primarily needs general knowledge that the model already has. Adding retrieval for questions the model could answer well from training adds latency and cost without quality benefit.
• The required behavior depends on style or capability the model needs to be fine-tuned for. Sometimes fine-tuning is the right answer; sometimes both fine-tuning and RAG together.
• The use case is conversational with stable knowledge. A chatbot answering general product questions about a fixed product line may do better with fine-tuning or careful prompting than with RAG.

The mental model: RAG handles the knowledge-injection problem. If your application has a knowledge-injection problem (organization-specific data, time-sensitive information, citation requirements), RAG is the appropriate architecture. If it doesn’t, RAG adds complexity without commensurate benefit.

How to start building a RAG system

The realistic path for a team new to RAG:

• Pick a narrow use case with a defined knowledge corpus (typically a few hundred to a few thousand documents). Customer support Q&A or internal-documentation search are common starting points.
• Use a managed platform for the first build. LangChain, LlamaIndex, OpenAI’s built-in tools, Anthropic’s Claude integrations, and many SaaS RAG platforms (Vectara, Glean, others) let you skip building the infrastructure yourself.
• Get the retrieval working first. Build the simplest possible end-to-end system (load documents, chunk them, embed them, store in a vector database, retrieve at query time) and measure retrieval quality before worrying about prompt engineering.
• Add an evaluation pipeline early. A set of test queries with expected answers, run regularly against the system. Without evaluation, improvements are guesses.
• Iterate on the failures. Bad retrievals point at chunking, embedding-model, or query-rewriting issues. Bad generations on good retrievals point at prompt engineering. Hallucinations beyond retrieved content point at guardrails.
• Scale carefully. The system that works for 100 documents may not work the same way for 100,000. Retrieval quality, latency, and cost all change with scale.

For broader context on the AI stack RAG sits inside, our AI Agents practitioner’s guide covers the agent layer that often uses RAG as a tool, and our Model Context Protocol piece covers the integration standard that increasingly governs how AI systems access RAG-style data sources.

Frequently Asked Questions