How RAG works: from your documents to a grounded answer.

A language model can write a fluent answer without having access to your current handbook, product catalog, research archive, or internal policy. Retrieval-augmented generation adds an information-retrieval step before generation. The application searches an external corpus, selects relevant material, and gives that material to the model as context for the response.

Microsoft describes RAG as a pattern that extends a model with proprietary content. That framing is useful because it keeps the boundaries clear. RAG does not pour a database into the model's memory, and it does not make a model permanently learn every document. It fetches evidence for a particular request. When a source changes, the index can be updated without retraining the foundation model.

The best reason to use RAG is that the answer depends on information that is private, changing, too recent, or too specific to expect a general model to know reliably. A company policy assistant needs the current policy. A technical support assistant needs the relevant product version. A research assistant needs the papers it is allowed to cite. RAG creates a path from the source to the answer.

The limitation is equally important: if the right evidence is not retrieved, the model is still working without it. A beautiful answer based on the wrong passage is not a successful RAG result. Treat retrieval as a first-class system to design and measure, not as a hidden database call before the interesting model work.

RAG quality begins with the documents. Remove duplicate versions, identify authoritative owners, extract text accurately, preserve headings and tables where possible, and attach metadata such as title, URL, date, product, department, region, access group, and version. A search system cannot recover meaning that was destroyed during extraction.

Chunking breaks a long document into smaller passages that can be retrieved independently. A chunk should contain a coherent idea with enough surrounding context to be understood. Splitting every document into identical character counts can separate a policy rule from its exception or a table heading from its values. Microsoft recommends treating content type and structure as design inputs rather than assuming one universal chunk size.

Store the relationship between each chunk and its source. Keep the document title, stable URL or identifier, heading path, page or section, version, and last-updated information. This supports citations and makes it possible to remove or replace all chunks belonging to an old document. It also lets a reviewer inspect what the model was shown.

Define an update strategy. Some collections need a scheduled refresh; others need event-driven indexing when a policy or product document changes. Stale content can be more dangerous than missing content because it looks authoritative. Make the effective date visible and decide how conflicting versions are handled before users rely on the system.

At query time, the system turns a user's question into a search request. Keyword search is useful when exact names, identifiers, or legal phrases matter. Vector search helps match related meaning even when the wording differs. Hybrid search combines keyword and vector signals, and semantic ranking can reorder candidates using broader language understanding. The right choice depends on the corpus and questions, so measure it rather than selecting by label.

Conversational questions need query understanding. A user may ask 'What is the refund window?' after discussing a specific product and region. The retrieval layer may need the relevant conversation context, a safe query rewrite, filters, or a clarification question. A rewrite should add context without changing the user's intent. Never let a query optimizer silently broaden access to documents the user is not authorized to see.

Retrieve enough evidence to cover the answer, but do not send an indiscriminate document dump to the model. Microsoft notes the practical tension among relevance, token limits, and response time. More context can introduce contradictions and distract attention. Use top-k retrieval, filters, reranking, or a second targeted search when the question genuinely has multiple parts.

Relevance is not permission. Apply tenant, user, department, document, and record-level access controls at retrieval time and again before generation if needed. The model should never receive a private passage merely because its embedding was close to the query. In a multi-tenant system, the authorization context must travel with the request through the orchestrator and search layer.

The generation step usually combines the user's question, retrieved passages, system instructions, output requirements, and citation metadata. Tell the model what the passages are, what authority they have, and what to do when they do not answer the question. A useful rule is: answer from the supplied evidence, identify uncertainty, and say what is missing instead of filling the gap with general knowledge.

Separate instructions from retrieved content. A document may contain text that looks like an instruction, but it is evidence about the domain, not permission to change the agent's behavior. Mark source content clearly and keep the higher-authority application rules outside it. This is one layer of defense against prompt injection in webpages, emails, uploaded files, and other untrusted material.

Citations should point to the evidence actually used. Preserve source identifiers through retrieval and generation, and validate that a cited passage supports the claim. A citation-shaped link that leads to a general homepage is not grounded evidence. For high-value answers, show the document title, section, date, and a link or location a reviewer can inspect.

Make refusal a valid result. If the retrieved context is weak, conflicting, stale, or outside the user's permissions, the system should ask a clarifying question, provide a limited answer, or escalate. The goal is not to answer every question. The goal is to answer the supported questions accurately and make unsupported ones visible.

A classic RAG flow often sends one query to one search index and then passes the results to a model. It is simple, fast, and appropriate for many straightforward questions. Problems appear when the user asks a multi-part question, needs comparisons across sources, requires a current calculation, or needs the system to choose among several knowledge bases.

Query decomposition can split a complex question into focused searches. A reranker can improve the ordering of retrieved passages. Metadata filters can narrow by product, date, region, or permission. A second model call can compress or synthesize evidence before the final answer. Each extra step adds latency, cost, and another place to fail, so introduce it because an evaluation shows a need.

Agentic retrieval is an evolution in which an agent treats retrieval as a tool and decides when to search, how to break down a question, or which source to use. Microsoft's current guidance distinguishes this from a fixed classic pipeline. Agentic retrieval can help with complex conversational questions, but it also increases the importance of tool permissions, stop conditions, trajectory evaluation, and monitoring.

Long documents and long conversations create context pressure even when the model supports a large window. Anthropic's context-engineering work argues for curating high-signal context, using just-in-time retrieval, compaction, structured notes, or specialist sub-agents for long-horizon tasks. Retrieval is not a license to send everything; it is a way to select what matters now.

A final answer score can hide whether the problem was search or generation. Create a representative question set with known supporting sources. Measure whether the correct evidence appears in the retrieved set, whether irrelevant or unauthorized evidence appears, and whether the final answer is supported by the evidence it cites.

For retrieval, inspect recall, ranking, filters, freshness, and source coverage. For generation, inspect groundedness, completeness, relevance, citation correctness, refusal behavior, and clarity. The metric names vary by stack, but the principle is stable: test the steps independently and then test the end-to-end experience.

Include unanswerable, ambiguous, adversarial, and permission-sensitive questions. An RAG system should not confidently quote a near-match when the requested document is absent. Test conflicting policies and superseded versions. Test a user who is allowed to see the title but not the contents. Test a malicious passage that tries to override the system instructions.

Keep the corpus snapshot and retrieval configuration with the result. Changes to chunking, embeddings, filters, ranking, source documents, model, prompt, or temperature can change behavior. Your [AI evaluation guide](/ai-evaluation) explains how to turn failures into regression tasks and how to combine automated checks with human calibration.

Start with a narrow corpus and a narrow question family. Build a small golden set of real questions and mark the supporting passage for each. This makes it possible to compare extraction, chunking, search, and answer changes without needing a huge platform first.

Then build ingestion with source metadata and an explicit update path. Test whether the indexed text matches the source. Add keyword, vector, or hybrid retrieval and inspect the top results manually. Do not tune a final answer prompt while the search results are clearly wrong; fix the stage that is failing.

Add grounding instructions, citations, access filters, and refusal behavior. Evaluate the full pipeline. Measure latency and cost alongside quality because a technically strong answer that takes too long or costs more than the business value will not be adopted.

Pilot with a small group and review traces. Watch for questions outside the corpus, stale sources, repeated retrieval failures, overlong answers, and users who misinterpret a citation as a guarantee. Expand the corpus and autonomy only when the current question family has evidence of reliable performance.