A general-purpose language model knows a great deal about the world and nothing specific about your business. It has never read your returns policy, your onboarding checklist or last quarter’s pricing update. Ask about them and it will either decline or, worse, produce a confident answer that sounds right and isn’t.
Retrieval-augmented generation (RAG) is the most common way to close that gap without retraining a model. The term comes from a 2020 research paper by Lewis et al., which paired a language model with a searchable index of documents so that answers could draw on information outside the model’s training data.
How it works, in plain English
RAG adds a search step before the model writes anything:
- Prepare the knowledge. Documents are split into passages (“chunks”) and stored in a search index, often as vector embeddings that capture meaning, sometimes alongside traditional keyword search.
- Retrieve. When someone asks a question, the system searches the index for the passages most relevant to it.
- Generate. Those passages are placed into the prompt with instructions to answer from them, and the model writes a response, ideally citing where each part came from.
Anthropic’s engineering write-up on contextual retrieval describes this standard pipeline and one of its weak points: when documents are cut into small chunks, a passage can lose the context that made it findable.
A hypothetical example
A wholesale distributor keeps its returns policy, warranty terms and supplier agreements in shared folders, and customer-service staff spend a lot of time searching them.
A RAG assistant inside the support tool could take the question “Can a customer return opened stock after 30 days?”, retrieve the relevant policy clause and the exception for damaged goods, and draft an answer that links to both. The staff member reads the cited clauses before replying.
The value is not that the model “knows” the policy. It is that the right paragraph is put in front of the model, and in front of the person checking the answer.
What matters in implementation
- Retrieval quality comes first. If the right passage isn’t retrieved, the model can’t use it. Chunking, document structure, combined keyword and semantic search, metadata filters (product, region, effective date) and reranking all affect this. Measure retrieval separately from answer quality.
- Enforce permissions at retrieval time. An assistant must not surface a document the user couldn’t open themselves. OWASP’s guidance on vector and embedding weaknesses recommends permission-aware stores and strict partitioning of data with different access levels. In practice, carry access rules into the index and filter before passages ever reach the prompt. Telling the model “don’t reveal this” is not access control.
- Treat retrieved text as untrusted input. Documents can contain instructions, accidental or malicious, that a model might follow. OWASP describes this as indirect prompt injection. Keep the assistant’s permissions narrow, especially if it can take actions.
- Keep the index current. Policies change. Plan how new, updated and deleted documents reach the index, and record which version an answer was based on.
- Show sources and allow “I don’t know”. Instructing the model to answer only from the provided material, cite it, and admit uncertainty when the material doesn’t cover the question are documented ways to reduce hallucinations. Citations also let people verify answers quickly.
- Evaluate with real questions. Build a set of representative questions with known good answers and sources, and rerun it whenever chunking, models or prompts change.
Limitations and trade-offs
RAG improves grounding. It does not guarantee correct answers or eliminate hallucinations. The model can still misread a passage, combine two passages incorrectly, or answer when it should decline. Retrieval can miss the relevant document entirely. And poorly maintained source content produces poorly grounded answers, however good the pipeline is.
There are running costs too: embedding and storing documents, longer prompts on every request, and a little extra latency for the search step. For a small, stable knowledge set, placing the whole document in the prompt may be simpler. For live data such as order status or account balances, a direct, permission-checked API call is usually better than searching documents.
RAG also doesn’t change how a model behaves. If the problem is tone or output format, better prompting or other techniques are the right tools.
Where to start
Pick one well-bounded body of knowledge with clear owners, a group of people who ask repeated questions about it, and a way to check answers. Confirm that retrieval finds the right sources before polishing the generated responses.
If you’re considering an assistant over your own documentation, we’re happy to talk through whether RAG fits and what a first version should cover.
